Hi, This week I received a bug report from an Athena user here at the APS. The problem is a bit complicated to explain, but the executive summary is that two different ways of moving chi(k) data from an Athena project file into Artemis resulted in two different spectra even though they should have been identical. I spent a bit of time trying to track down the exact nature of the bug, but it has proven a bit elusive. I can say this, the Athena project file which demonstrates the problem has 148 groups in the group list. This problem cannot be replicated in a project file with a few (or even a couple dozen) groups in the group list. Many of you will recognize the nature of the problem. 148 is more groups than Athena can comfortably handle. I'll explain this issue in some detail below. Before I do, I'll give a good rule of thumb that will help keep Athena happy with you and you happy with her: =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*= Don't import too much data into your Athena project! About 50 or 60 data groups in the group list is a good upper limit. With fewer groups, Athena should work reliably. With more, Athena will eventually become unstable. If you need more than 60 data groups, you need to split your ensemble of data into two or more subsets and process each subset individually. =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*= Why is that? Well, to answer, I need to explain a bit about how Athena works. All of the hard work (Fourier transforms, background removal, normalization, and so on) is done by a library of numerical and exafs-related functions called Ifeffit, which is the thing written by Matt. Athena is really just a glorified bookkeeper. She keeps track of your data and of the values of all your parameters. When it comes time to do any of the actual numerical work, she organizes the problem, then asks the Ifeffit library to take care of it. When it comes time to make a plot, Athena organizes how the plot should be made, then asks Ifeffit to take care of it. In the case of plotting, Ifeffit, in turn, asks another library called Pgplot to actually make the plot on the screen. This stack of software tools is actually quite more complicated than that. Ifeffit and Pgplot are written in a programming language called Fortran. Athena is written in a different language called Perl. There is a small piece of code written in C which allows the Perl program to talk to the Fortran libraries. Additionally, Athena uses a graphical toolkit called Tk, which is written in C, to draw buttons and sliders and text boxes and such on the screen. The point here is that Athena is something of a house of cards. Lots of different software components need to be working together correctly in order for you to do your data analysis. (One might be astonished that Athena works in the first place. Ssshhhh!!! If you lose faith, everything will come tumbling down!) Because Athena requires so many concurrently functioning pieces, it has some weak spots. One of these weak spots is quite serious. The Fortran programming language (the one used to write Ifeffit) is quite old -- early versions of Fortran date back to the mid 60s, a time when RAM was measured in kilobytes rather than gigabytes. One feature of modern programming languages that Fortran lacks is the ability to claim and release memory. Consequently, a Fortran program has to allocate enough memory to run at the moment that it starts running. If, during the course of operation, the Fortran program moves to a state in which it needs more memory -- it's out of luck. What's worse, Fortran does not even provide particularly good tools for failing gracefully when it attempts to exceed its memory limit. Ifeffit, then, allocates some large amount of memory up front and then uses some clever heap management techniques written by Matt himself to make the most effective use of the available memory. It works well, up to a point. If you attempt to import too much data into Athena, Ifeffit will run into its memory limit and begin to behave erratically. Obviously, when Ifeffit behaves erratically, so does Athena. It would seem that having 148 data groups triggers a bug in the heap management scheme. Drat! And that is why you should not have too many data groups in your Athena project file. So, should Ifeffit be rewritten in some language that does dynamic memory allocation? Probably, but that would be a lot of work and Matt has both a daughter and a day job. B -- Bruce Ravel ---------------------------------------------- bravel@anl.gov Molecular Environmental Science Group, Building 203, Room E-165 MRCAT, Sector 10, Advanced Photon Source, Building 433, Room B007 Argonne National Laboratory phone and voice mail: (1) 630 252 5033 Argonne IL 60439, USA fax: (1) 630 252 9793 My homepage: http://cars9.uchicago.edu/~ravel EXAFS software: http://cars9.uchicago.edu/~ravel/software/
Hi Bruce, The explanation is eminently reasonable, and so is the workaround (use more than one project). If it's not difficult to code, though, I'd suggest modifying Athena so that it won't allow you to try to import more than, say, 50 groups. That would transform this from a bug into a limitation of the program. All programs have limitations, of course, but those don't lead to analysis errors. As it stands, an unwary user could publish results based in part on screwy data. --Scott Calvin Sarah Lawrence College At 07:02 PM 5/8/2007, you wrote:
Hi,
This week I received a bug report from an Athena user here at the APS. The problem is a bit complicated to explain, but the executive summary is that two different ways of moving chi(k) data from an Athena project file into Artemis resulted in two different spectra even though they should have been identical.
I spent a bit of time trying to track down the exact nature of the bug, but it has proven a bit elusive. I can say this, the Athena project file which demonstrates the problem has 148 groups in the group list. This problem cannot be replicated in a project file with a few (or even a couple dozen) groups in the group list.
Many of you will recognize the nature of the problem. 148 is more groups than Athena can comfortably handle. I'll explain this issue in some detail below. Before I do, I'll give a good rule of thumb that will help keep Athena happy with you and you happy with her:
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
Don't import too much data into your Athena project!
About 50 or 60 data groups in the group list is a good upper limit. With fewer groups, Athena should work reliably. With more, Athena will eventually become unstable.
If you need more than 60 data groups, you need to split your ensemble of data into two or more subsets and process each subset individually.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
Why is that? Well, to answer, I need to explain a bit about how Athena works. All of the hard work (Fourier transforms, background removal, normalization, and so on) is done by a library of numerical and exafs-related functions called Ifeffit, which is the thing written by Matt. Athena is really just a glorified bookkeeper. She keeps track of your data and of the values of all your parameters. When it comes time to do any of the actual numerical work, she organizes the problem, then asks the Ifeffit library to take care of it. When it comes time to make a plot, Athena organizes how the plot should be made, then asks Ifeffit to take care of it. In the case of plotting, Ifeffit, in turn, asks another library called Pgplot to actually make the plot on the screen.
This stack of software tools is actually quite more complicated than that. Ifeffit and Pgplot are written in a programming language called Fortran. Athena is written in a different language called Perl. There is a small piece of code written in C which allows the Perl program to talk to the Fortran libraries. Additionally, Athena uses a graphical toolkit called Tk, which is written in C, to draw buttons and sliders and text boxes and such on the screen.
The point here is that Athena is something of a house of cards. Lots of different software components need to be working together correctly in order for you to do your data analysis. (One might be astonished that Athena works in the first place. Ssshhhh!!! If you lose faith, everything will come tumbling down!) Because Athena requires so many concurrently functioning pieces, it has some weak spots. One of these weak spots is quite serious.
The Fortran programming language (the one used to write Ifeffit) is quite old -- early versions of Fortran date back to the mid 60s, a time when RAM was measured in kilobytes rather than gigabytes. One feature of modern programming languages that Fortran lacks is the ability to claim and release memory. Consequently, a Fortran program has to allocate enough memory to run at the moment that it starts running. If, during the course of operation, the Fortran program moves to a state in which it needs more memory -- it's out of luck. What's worse, Fortran does not even provide particularly good tools for failing gracefully when it attempts to exceed its memory limit.
Ifeffit, then, allocates some large amount of memory up front and then uses some clever heap management techniques written by Matt himself to make the most effective use of the available memory. It works well, up to a point. If you attempt to import too much data into Athena, Ifeffit will run into its memory limit and begin to behave erratically. Obviously, when Ifeffit behaves erratically, so does Athena. It would seem that having 148 data groups triggers a bug in the heap management scheme. Drat!
And that is why you should not have too many data groups in your Athena project file.
So, should Ifeffit be rewritten in some language that does dynamic memory allocation? Probably, but that would be a lot of work and Matt has both a daughter and a day job.
B
-- Bruce Ravel ---------------------------------------------- bravel@anl.gov
Molecular Environmental Science Group, Building 203, Room E-165 MRCAT, Sector 10, Advanced Photon Source, Building 433, Room B007
Argonne National Laboratory phone and voice mail: (1) 630 252 5033 Argonne IL 60439, USA fax: (1) 630 252 9793
My homepage: http://cars9.uchicago.edu/~ravel EXAFS software: http://cars9.uchicago.edu/~ravel/software/
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
On Wednesday 09 May 2007, Scott Calvin wrote:
The explanation is eminently reasonable, and so is the workaround (use more than one project). If it's not difficult to code, though, I'd suggest modifying Athena so that it won't allow you to try to import more than, say, 50 groups. That would transform this from a bug into a limitation of the program. All programs have limitations, of course, but those don't lead to analysis errors. As it stands, an unwary user could publish results based in part on screwy data.
Scott, you are the third person to suggest this, either on the list or in personal email. I am not completely sold on the correct way to proceed, so I'd like to solicit opinions from more people on the list. Given that I have never been able to reliably predict when the problem under discussion rears its ugly head, I have been hesitant to do the thing you suggest. For instance, I could belive that you could run into a version of this problem by importing and removing 45 data groups repeatedly. So what do you all think? Is it better to impose the somewhat arbitrary limit that Scott suggests in hopes that it will work as intended most of the time? Or is it better to leave things as they are, allowing you to import more data but running the risk of corrupting Ifeffit's memory management? Thanks, B -- Bruce Ravel ---------------------------------------------- bravel@anl.gov Molecular Environmental Science Group, Building 203, Room E-165 MRCAT, Sector 10, Advanced Photon Source, Building 433, Room B007 Argonne National Laboratory phone and voice mail: (1) 630 252 5033 Argonne IL 60439, USA fax: (1) 630 252 9793 My homepage: http://cars9.uchicago.edu/~ravel EXAFS software: http://cars9.uchicago.edu/~ravel/software/
So what do you all think? Is it better to impose the somewhat arbitrary limit that Scott suggests in hopes that it will work as intended most of the time? Or is it better to leave things as they are, allowing you to import more data but running the risk of corrupting Ifeffit's memory management?
Hi Bruce, I think that there should be a limit that is by default turned on and set to a value of something like 50 groups. This feature should be configurable so that the user, under a warning, can turn it off or change the group limit. Easy and configurable = plus 2cents. It occurs to me that a better way would be for Ifeffit (or Athena) to keep track of how much memory swapping is going on. Once it reaches some large but safe value, a warning is sent to the user. In this scenario the deleting and reading in a bunch of new groups would also trigger the warning. This seems like it would be hard to implement = negative 2cents. I'm even. Shelly
Hi Bruce, At this rate, you are going to get a lot of tuppence. :-) Given 1) that the issue occurs when the number of projects reaches some unclear number, and 2) that the effect can be non-obviously changed data, I think that there should be an arbitrary, but safe number that can be bypassed as Shelly suggests. If the effect caused bells and whistles and the program to crash, then it would not be as much of an issue, but changed data without some obvious symptom of the change is dangerous. Mark -----Original Message----- From: ifeffit-bounces@millenia.cars.aps.anl.gov [mailto:ifeffit-bounces@millenia.cars.aps.anl.gov] On Behalf Of Kelly, Shelly D. Sent: Wednesday, May 09, 2007 12:11 PM To: Ravel, Bruce D.; XAFS Analysis using Ifeffit Subject: Re: [Ifeffit] care and feeding of Athena
So what do you all think? Is it better to impose the somewhat arbitrary limit that Scott suggests in hopes that it will work as intended most of the time? Or is it better to leave things as they are, allowing you to import more data but running the risk of corrupting Ifeffit's memory management?
Hi Bruce, I think that there should be a limit that is by default turned on and set to a value of something like 50 groups. This feature should be configurable so that the user, under a warning, can turn it off or change the group limit. Easy and configurable = plus 2cents. It occurs to me that a better way would be for Ifeffit (or Athena) to keep track of how much memory swapping is going on. Once it reaches some large but safe value, a warning is sent to the user. In this scenario the deleting and reading in a bunch of new groups would also trigger the warning. This seems like it would be hard to implement = negative 2cents. I'm even. Shelly _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
From seeing the project file that was causing problems, I think the
Hi, problem was not an Ifeffit memory issue, but a problem with Artemis reading chi(k) from Athena projects that uses a standard in background removal. That's not to say that memory limitations aren't real -- I've definitely seen corrupted data with large sets of data. But I don't think the project in question is at that limit.
It occurs to me that a better way would be for Ifeffit (or Athena) to keep track of how much memory swapping is going on. Once it reaches some large but safe value, a warning is sent to the user. In this scenario the deleting and reading in a bunch of new groups would also trigger the warning. This seems like it would be hard to implement = negative 2cents.
Ifeffit does keep track of (and Athena / Artemis use) how much of its internal memory for array data is used (with &max_heap and &heap_free). The project in question, with ~150 groups of data, uses 5% of the heap. We often run Athena using considerably more internal data than this example (with multi-element detectors), and often swap out projects without trouble. These actions work the heap space much more than reading in one project with ~150 groups of mu(E) data. My "torture tests" for Ifeffit has non-corrupted data (though it definitely slows down!) with more than 90% of the heap space being used (with ~1000 arrays of data). There were definitely problems in the past with erasing and moving lots of groups around, and I think these may not all be resolved.... Personally, I'd like to study problem cases (still trying to produce a simple test case) before deciding the details. I don't see a real benefit to having the max number of groups being a user setting, especially as the number of groups is not going to be the actual problem. --Matt
Hi Bruce, All, These issues are almost certainly due to some bug in Ifeffit that gets triggered with large data sets. As Bruce mentioned, Ifeffit internally keeps track of arrays and groups using a built-in memory cache. By trying to write portable Fortran, this was more-or-less a requirement (at least, it was several years ago when Ifeffit was first being written). I have a few tests for moving arrays around that pass for me without any trouble, but there are definitely occasional problems with "large sets of data" -- I've seen these myself. Unfortunately, I don't have a good simple test case that always causes a problem. If you have a project file that is "acting weird" and data is clearly being corrupted, please send it to me. --Matt
participants (5)
-
Bruce Ravel -
Kelly, Shelly D. -
Mark Davidson -
Matt Newville -
Scott Calvin