Hi Bradley, Bruce,
On Wed, Mar 21, 2012 at 8:44 AM, Bruce Ravel
Hi Bradley,
Athena cannot use multiple CPUs. No threads, no message passing. Sorry.
As for why Athena uses only 25% of the CPU in your VM is a mystery to me. Thatis not a restriction I put on the code and it is not something that normally happens with programs written using the tools Athena uses. My suspicion is that it is a VM configuration issue.
I think Bradley said 'using at most 25% of the available CPU', which probably means 'at most 1 CPU core at a time', which is correct. The OS may be able to switch which core is used for Athena/Ifeffit, but only one can be used at any time.
As for the memory issue, the dominant use of memory by a long stretch in Athena is the memory that is statically allocated for the Ifeffit library. While it is possible to compile up Ifeffit to use more memory, that isn't likely to have an impact on how fast you LCF runs.
I doubt memory (22% of 8Gb) is an issue here. Ifeffit is 32-bit, and will not ever use more than ~2Gb on Windows, though the GUI could add to that, and might be able to go above 2Gb. Anyway, memory is not being swapped from RAM to disk. Low-level cache memory will definitely be swapped to RAM, which is likely to actually limit run time, not exactly the number of cores. That's all to say that the mathematical problem probably could be solved much better by making use of multiple cores. But Athena/Ifeffit won't be doing that anytime soon.
As for the underlying problem -- the fact that it takes an hour and 40 minutes to run a large combinatorial set, the best advice I can offer is to use some prior knowledge to restrict the scope of the problem. If you know that a standard is present in a sample, you can restrict the combinatorial set to always include that one. Similarly, if you can reduce the size of you set of standards by excluding ones that you know to be unlikely or absent, that too will help. Combinatorial fitting is a blunt instrument and you seem to be using it in the most blunt way.
Right, throwing more spectra at Athena's LCF is going to be expensive. The combinations are tried one at a time, and that won't ever be changed. OTOH, recasting the basic problem in terms of linear algebra might help, and might be able to make use of multiple cores, but that is not something Athena is going to do with Ifeffit 1.2, or probably ever.
PS: One thing that occurs to me is that you could run 2 or three instances of Athena. Each one will use its own CPU and you can be a human load leveler.
Yes, running 4 instances of Athena would help, as each could presumably use a different core. Even if they're sharing physical cores, it might actually go faster, depending on caching. Splitting the problem in half 4 different ways and ensuring overlap is likely to take less real time. --Matt