Hi Bruce, The explanation is eminently reasonable, and so is the workaround (use more than one project). If it's not difficult to code, though, I'd suggest modifying Athena so that it won't allow you to try to import more than, say, 50 groups. That would transform this from a bug into a limitation of the program. All programs have limitations, of course, but those don't lead to analysis errors. As it stands, an unwary user could publish results based in part on screwy data. --Scott Calvin Sarah Lawrence College At 07:02 PM 5/8/2007, you wrote:
Hi,
This week I received a bug report from an Athena user here at the APS. The problem is a bit complicated to explain, but the executive summary is that two different ways of moving chi(k) data from an Athena project file into Artemis resulted in two different spectra even though they should have been identical.
I spent a bit of time trying to track down the exact nature of the bug, but it has proven a bit elusive. I can say this, the Athena project file which demonstrates the problem has 148 groups in the group list. This problem cannot be replicated in a project file with a few (or even a couple dozen) groups in the group list.
Many of you will recognize the nature of the problem. 148 is more groups than Athena can comfortably handle. I'll explain this issue in some detail below. Before I do, I'll give a good rule of thumb that will help keep Athena happy with you and you happy with her:
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
Don't import too much data into your Athena project!
About 50 or 60 data groups in the group list is a good upper limit. With fewer groups, Athena should work reliably. With more, Athena will eventually become unstable.
If you need more than 60 data groups, you need to split your ensemble of data into two or more subsets and process each subset individually.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
Why is that? Well, to answer, I need to explain a bit about how Athena works. All of the hard work (Fourier transforms, background removal, normalization, and so on) is done by a library of numerical and exafs-related functions called Ifeffit, which is the thing written by Matt. Athena is really just a glorified bookkeeper. She keeps track of your data and of the values of all your parameters. When it comes time to do any of the actual numerical work, she organizes the problem, then asks the Ifeffit library to take care of it. When it comes time to make a plot, Athena organizes how the plot should be made, then asks Ifeffit to take care of it. In the case of plotting, Ifeffit, in turn, asks another library called Pgplot to actually make the plot on the screen.
This stack of software tools is actually quite more complicated than that. Ifeffit and Pgplot are written in a programming language called Fortran. Athena is written in a different language called Perl. There is a small piece of code written in C which allows the Perl program to talk to the Fortran libraries. Additionally, Athena uses a graphical toolkit called Tk, which is written in C, to draw buttons and sliders and text boxes and such on the screen.
The point here is that Athena is something of a house of cards. Lots of different software components need to be working together correctly in order for you to do your data analysis. (One might be astonished that Athena works in the first place. Ssshhhh!!! If you lose faith, everything will come tumbling down!) Because Athena requires so many concurrently functioning pieces, it has some weak spots. One of these weak spots is quite serious.
The Fortran programming language (the one used to write Ifeffit) is quite old -- early versions of Fortran date back to the mid 60s, a time when RAM was measured in kilobytes rather than gigabytes. One feature of modern programming languages that Fortran lacks is the ability to claim and release memory. Consequently, a Fortran program has to allocate enough memory to run at the moment that it starts running. If, during the course of operation, the Fortran program moves to a state in which it needs more memory -- it's out of luck. What's worse, Fortran does not even provide particularly good tools for failing gracefully when it attempts to exceed its memory limit.
Ifeffit, then, allocates some large amount of memory up front and then uses some clever heap management techniques written by Matt himself to make the most effective use of the available memory. It works well, up to a point. If you attempt to import too much data into Athena, Ifeffit will run into its memory limit and begin to behave erratically. Obviously, when Ifeffit behaves erratically, so does Athena. It would seem that having 148 data groups triggers a bug in the heap management scheme. Drat!
And that is why you should not have too many data groups in your Athena project file.
So, should Ifeffit be rewritten in some language that does dynamic memory allocation? Probably, but that would be a lot of work and Matt has both a daughter and a day job.
B
-- Bruce Ravel ---------------------------------------------- bravel@anl.gov
Molecular Environmental Science Group, Building 203, Room E-165 MRCAT, Sector 10, Advanced Photon Source, Building 433, Room B007
Argonne National Laboratory phone and voice mail: (1) 630 252 5033 Argonne IL 60439, USA fax: (1) 630 252 9793
My homepage: http://cars9.uchicago.edu/~ravel EXAFS software: http://cars9.uchicago.edu/~ravel/software/
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit