[Ifeffit] Decrease LCF Fitting Computational Time in Athena

[Bruce Ravel]

Hi Bradley,

Athena cannot use multiple CPUs.  No threads, no message passing.
Sorry.

As for why Athena uses only 25% of the CPU in your VM is a mystery to
me.  Thatis not a restriction I put on the code and it is not
something that normally happens with programs written using the tools
Athena uses.  My suspicion is that it is a VM configuration issue.

As for the memory issue, the dominant use of memory by a long stretch
in Athena is the memory that is statically allocated for the Ifeffit
library.  While it is possible to compile up Ifeffit to use more
memory, that isn't likely to have an impact on how fast you LCF runs.

As for the underlying problem -- the fact that it takes an hour and 40
minutes to run a large combinatorial set, the best advice I can offer
is to use some prior knowledge to restrict the scope of the problem.
If you know that a standard is present in a sample, you can restrict
the combinatorial set to always include that one.  Similarly, if you
can reduce the size of you set of standards by excluding ones that you
know to be unlikely or absent, that too will help.  Combinatorial
fitting is a blunt instrument and you seem to be using it in the most
blunt way.

Sorry I didn't have happier new for you.

B

PS:  One thing that occurs to me is that you could run 2 or three
instances of Athena.  Each one will use its own CPU and you can be a
human load leveler.

If you are up for a bit of programming, I can suggest another solution
which uses my new Demeter code to do the LCF.  On the plus side, it
would be much easier to automate your large amount of work.  On the
minuis side, you would have to do some programming.


On Tuesday, March 20, 2012 04:22:24 pm BradleyW Miller wrote:
> Hi,
> 
> I have been given access to a Virtual Machine (VM) with 8 GB of RAM and 4
> CPU's with the hopes of decreasing the time required to do LCF fittings. I
> have large set of environmental samples from different waste streams
> and/or contaminate sources. Currently I have 14 standards and I wanted to
> run a LCF model fitting up to 4 standards. This results in 1456 fits if I
> selected "fit all combinations". This takes about 100 mins. to run 1
> samples (I have 60+ and more coming).
> 
> It appears that Athena is constrained to using at most 25% of the
> available CPU and the physical memory is limited to 22% .
> 
> 1) Is there a way I could change Athena's settings to take advantage of
> this VM's multiple CPU's and larger RAM?
> 
> 2) Or if Athena can't handle multiple CPU's is there a way to increase the
> maximum percentage of the CPU being used? It appears that Athena is
> limited to accessing up to 25% of the CPU's. One CPU of the four appears
> to be processing a little more data than the other 3 CPU's  (when viewed
> on the Task Manager, Performance tab). When more CPU's were added to the
> VM,  computational time remained similar... if not worse.
> 
> 3) Similarly the physical memory is limited to 22% of available memory
> (when viewed on the Task Manager, Performance tab) when Athena is running
> the LCF models as well. Could Athena's setting be changed to increase the
> % of RAM available for LCF fitting?
> 
> 
> 4) Any other suggestions?
> 
> 
> I spoke with my IT guy who said he believes the software is limiting my
> hopes of decreasing the computational time. I'd love to call it all
> contaminated dirt, rent some big equipment like D-9 dozers, big trucks, a
> couple barges and dump it into the ocean... but we don't have that much
> $$$ for remediation and my kids eat fish. That's is a joke of course...
> my kids don't eat fish  :-)
> 
> 
> 
> 
> btw Thanks for your time and for use of the software!
> 
> Respectfully,
> Bradley W. Miller, Ph.D.
> Post Doctoral Fellow
> Oak Ridge Institute for Science and Education
> U. S. Environmental Protection Agency
> National Risk Management Research Laboratory
> Land Remediation and Pollution Control Division
> 5995 Center Hill Avenue, Cincinnati, OH  45224-1702
> 
> www.tinyurl.com/bwmiller
> 
> The great tragedy of Science—the slaying of beautiful hypothesis by an
> ugly fact.— Thomas H. Huxley
> 
> 
> Views or Opinions expressed in this email is solely representative of the
> sender and does not represents those of the EPA or any other agency.

-- 

 Bruce Ravel  ------------------------------------ bravel at bnl.gov

 National Institute of Standards and Technology
 Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2
 Building 535A
 Upton NY, 11973

 My homepage:    http://xafs.org/BruceRavel
 EXAFS software: http://cars9.uchicago.edu/ifeffit/Demeter