Hello Bruce, thanks a lot for your fast reply. I did not want to attach an example file immediately when submiting a question to the mailing list. But there is no problem to send you some files. In the attachments you will now find the original data file for trying to reproduce my problems and hopefully to find a solution. I attach also the files reduced to 1711 and 1627 data points (just by trial and error. I checked 1711 (didn't work), and 1627 (worked). The limit might be somewhere inbetween. Attached is their read in in Athena, too. The counters are the following: 1 Energy Nominator 62 Absorption (Transmission), sample Reference Nominator 63 Reference (Fe foil) Best regards Joerg -----Ursprüngliche Nachricht----- Von: ifeffit-bounces@millenia.cars.aps.anl.gov [mailto:ifeffit-bounces@millenia.cars.aps.anl.gov] Im Auftrag von Ravel, Bruce Gesendet: Sonntag, 4. August 2013 18:40 An: XAFS Analysis using Ifeffit Betreff: Re: [Ifeffit] data point limits in Athena? Jeorg, I thought the default value for compiled-in array length limit in Ifeffit was 8192. I am not at work today, so I cannot look at the machine where I compile Ifeffit for the Demeter package. Perhaps it is smaller. I can look tomorrow. Of course, there isn't much more I can say or do with your problem. Your email is another one of those that makes me sound like a scratched record saying the same damn thing over and over again. If you don't provide an example that allows me to reproduce your problem on my computer, you cannot expect me to actually do anything about it. Here's a formula simple enough to commit to memory: example + clear recipe = actionable bug report no example + no recipe = whiny user B Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973 Homepage: http://xafs.org/BruceRavel Software: https://github.com/bruceravel ________________________________________ From: ifeffit-bounces@millenia.cars.aps.anl.gov [ifeffit-bounces@millenia.cars.aps.anl.gov] on behalf of Göttlicher, Jörg [joerg.goettlicher@kit.edu] Sent: Sunday, August 04, 2013 11:44 AM To: ifeffit@millenia.cars.aps.anl.gov Subject: [Ifeffit] data point limits in Athena? Hello, does someone now if there is a limit of data points for Athena/Athena-Demeter? I have realized that a XAFS data file containing about 2500 data points is not read in properly in Athena (v. 0.8.061) and is started at zero energy and cutted of before the end of the data rows in Athena-Demeter v. 0.9.9, whereas the same file reduced to about 1600 data points was read in without any problems in both Athena versions. I hope that someone has an idea about the reason(s) and how to manage the read in of such data without prior treatment? Using the binning mode or truncation (to cut off the zero energy point, that occured when reading in with Demeter 0.9.9), did not succeed because the data were not treated correctly during the import. Best regards Joerg _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit

Hi Jorg, Matt is on to the problem...Too many numbers... I deleted columns 2 - 58. Working with columns labelled Ioni1,2 and 3 was ok, with little problem reading it in to Athena 0.9.17 even with the 2460 lines. I say "little" because it does take several seconds on my system for Athena to read in the file so I can select columns. If I reduce it to just the 4 columns that are of most value to an end user (E, and detectors), it opens in about 1s. Notepad takes about that long to open the original file. So the problem seems to be not so much one of length, but girth - an obese datafile trying to squeeze into a hefty array-size/memory limit. I would not consider this an Athena issue. There are numerous columns in the datafile that are constants, many zero. The number of lines could also be improved. The data was collected at 0.3 eV steps from 6932 to 7700 eV and that certainly oversamples for most of that range. ..and this is the second instance I have seen of obese datafiles...starting to wonder how prevalent this issue is or will be, particularly when beamlines use large multi-element detectors. regards, Robert On Sun, Aug 4, 2013 at 3:00 PM, Matt Newville wrote:

Jorg,

...

Hello Bruce,

thanks a lot for your fast reply. I did not want to attach an example file immediately when submiting a question to the mailing list. But there is no problem to send you some files. In the attachments you will now find the original data file for trying to reproduce my problems and hopefully to find a solution.

I attach also the files reduced to 1711 and 1627 data points (just by

On Sun, Aug 4, 2013 at 12:49 PM, Göttlicher, Jörg wrote: trial and error. I checked 1711 (didn't work), and 1627 (worked). The limit might be somewhere inbetween.

...

Attached is their read in in Athena, too.

The counters are the following: 1 Energy Nominator 62 Absorption (Transmission), sample

Reference Nominator 63 Reference (Fe foil)

If you're having difficulty reading these data files, I think you may want to convert them into a more sensible format with say two or three columns before trying to read it into athena. That might help isolate where any problem might be.

--Matt

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit

Hello Bruce, Matt and Robert, thanks to you all for your helpfull comments. I didn't expect that the number of columns could have been the reason for the troubles to read in the data. There was not such a problem with importing data before because I usually measure XAS spectra with optimium energy/k step widths. Here, it was the first time when I decided to shift the energy dependent binning to a later step (because of some troubles with a mesurement macro during the night) which resulted in a higher number of data points, and together with the huge number of data columns exceeded Athena's limit. Nevertheless, for the future when more and more quick scans will be performed, it is good to have learned that data files should be cleared out before proceeding with evaluation. Next time I'll directly attach an example file to enable a more effective tracking of the problem. Sorry, for not doing so here. Finally, I would like to say that I very much appreciate the Iffefit program suite. You are doing a great job offering such comprised data evaluation possibilities to the XAS community worldwide. Best regards Joerg -----Ursprüngliche Nachricht----- Von: ifeffit-bounces@millenia.cars.aps.anl.gov [mailto:ifeffit-bounces@millenia.cars.aps.anl.gov] Im Auftrag von Bruce Ravel Gesendet: Montag, 5. August 2013 16:29 An: XAFS Analysis using Ifeffit Betreff: Re: [Ifeffit] data point limits in Athena? On 08/04/2013 01:49 PM, Göttlicher, Jörg wrote:

thanks a lot for your fast reply. I did not want to attach an example file immediately when submiting a question to the mailing list. But there is no problem to send you some files. In the attachments you will now find the original data file for trying to reproduce my problems and hopefully to find a solution.

Jörg, I know that it seems like you are being picked on just for asking an innocent question and I really do apologize for any bad feelings. However, your initial email is an example of a particularly unproductive kind of question to ask on this (or any other) mailing list. While it may seem like I am being a jerk for no good reason, it really would behoove you and *everyone* *else* reading this to think harder about how you ask questions. In this case, you made an incorrect assumption about the nature of the problem. You assumed the problem was the length of the file, when in fact the problem was its width. More specifically, Ifeffit and Athena have no problems with data a few thousand data points long. You presumed otherwise and did not attach an example of the data that was giving you trouble. To reach a resolution required a snotty response from me, a follow-up email from you, then -- finally -- useful responses from Robert and Matt. Had you simply attached an example of the file giving you trouble, your problem would have been addressed straight away and without all the attitude. As has been explained many, many times on this mailing list, Ifeffit (the library upon which Athena relies) is a rather old piece of software. It was written in an older dialect of Fortran which did not have dynamic memory allocation. This means that Ifeffit requests a fixed amount of memory from the computer at start-up which cannot be expanded upon at run-time. The reason that the width of your file is a problem is that huge amounts of that fixed memory are used to contain arrays that serve no purpose to you. Import a few of these files and **poof!** all of Ifeffit's memory is gone. It's either a really good thing or a really bad thing that the success of Athena among its users is such that people feel that they can get by without ever examining or even considering the content of their data files. Often that's OK. Often it's not. In this case, your beamline scientist isn't really doing you any favors sending you home with such a needlessly complex data file. Sigh.... Your best bet would be to somehow pre-process all of your data, removing the unnecessary columns (as both Robert and Matt suggested). This can either be done with a small, stand-along program or with a plugin for Athena (http://bruceravel.github.io/demeter/aug/other/plugin.html) B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973 Homepage: http://xafs.org/BruceRavel Software: https://github.com/bruceravel _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit