Handling large data sets in Athena (and energy dispersive calibration)
Dear All, Here's an issue I recently encountered while trying to handle some very large in situ datasets in Athena; it may be by design/an inherent limitation, rather than a bug, though: (I am running Demeter 0.9.21 pre and Demeter 0.9.20 , both x64 on Windows 8.1 x64 on separate machines). Each of my datafiles contains over 300 columns of u(E) data as space delimited plain text, however when I try and import them to Athena the import window will only display checkboxes up to column 87. I've attached an example datafile and a logfile to this email. (The data is very poor, but that shouldn't be an issue for debugging...) I've also attached a copy of the datafile as it came from the beamline - this also displays the issue, so it's nothing I've done to the file. On a related note, is it possible for the energy dispersive data calibration to be added back to Athena? To get the calibration parameters for this data I had to use the (very) old version! Thanks, Ian ---- Ian Godfrey PhD Student, UCL/JAIST Programme Industrial Doctorate Centre in Molecular Modelling and Materials Science, Department of Chemistry, University College London and School of Materials Science, Japan Advanced Institute of Science and Technology i.godfrey@ucl.ac.ukmailto:i.godfrey@ucl.ac.uk i.godfrey@jaist.ac.jpmailto:i.godfrey@jaist.ac.jp
Ian, This would be a perfect example for why Larch exists With it, you can read this file, and even make sense of it, even if Ifeffit (and so by extension, current versions of Athena) cannot. With Larch (essentially, Python), it is as simple as: larch> data = loadtxt(open('citrate_003_log_0_calibrated')) larch> plot(data[:, 0], data[:, 1:].sum(axis=1)) That summing of all the columns past the first was a guess, and it seems to look like noisy Au L3 edge XAFS. (aside: I had to guess because your columns are not labeled, This is not an issue for reading the data in the program, but is an issue when communicating scientific data. Label your axis!). So one way of putting this might be: the issue has been fixed, you're just not using that solution. For energy calibration, it looks like you used a linear calibration: larch> raw= loadtxt(open('citrate_003_log_0'), skip_rows=1) larch> slope, offset = polyfit(raw[:, 0], data[:, 0], 1) larch> print slope, offset 0.819999635553 11383.8402969 which means you could have calibrated the raw data with larch> energy = 11383.84 + 0.82 * raw[:, 0] My point here is: Don't expect Athena to have a canned solution for every analysis operation and every kind of data you might want to use. Instead, expect these tools that help you do the tasks you need. Hope that helps, --Matt
Hey Ian, Basically, ditto on what Matt said. 300 columns is not something that Athena is designed to do. My opinion is that you (or the beamline you visit) should have some kind of tool to turn a file that into XAS. As for the energy dispersive thing -- it's long been on the to do list. From time to time, I have poked at it, but I have never developed an algorithm that I actually like. Even the algorithm in the old Athena was unstable and required very good initial guesses. The best parts of A&A are the parts that I have needed in my career. I've never actually done my own experiment at an ED beamline, so my software doesn't really support it. B On 12/18/2014 04:19 AM, Godfrey, Ian wrote:
Dear All,
Here’s an issue I recently encountered while trying to handle some very large /in situ/ datasets in Athena; it may be by design/an inherent limitation, rather than a bug, though:
(I am running Demeter 0.9.21 pre and Demeter 0.9.20 , both x64 on Windows 8.1 x64 on separate machines).
Each of my datafiles contains over 300 columns of u(E) data as space delimited plain text, however when I try and import them to Athena the import window will only display checkboxes up to column 87.
I’ve attached an example datafile and a logfile to this email. (The data is very poor, but that shouldn’t be an issue for debugging…) I’ve also attached a copy of the datafile as it came from the beamline – this also displays the issue, so it’s nothing I’ve done to the file.
On a related note, is it possible for the energy dispersive data calibration to be added back to Athena? To get the calibration parameters for this data I had to use the (very) old version!
Thanks,
Ian
----
Ian Godfrey
PhD Student,
UCL/JAIST Programme
Industrial Doctorate Centre in Molecular Modelling and Materials Science,
Department of Chemistry,
University College London
and
School of Materials Science,
Japan Advanced Institute of Science and Technology
i.godfrey@ucl.ac.uk mailto:i.godfrey@ucl.ac.uk i.godfrey@jaist.ac.jp mailto:i.godfrey@jaist.ac.jp
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
-- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/
Dear Matt and Bruce,
Thanks for your comments and hints. I will give Larch a spin analysing these data and see how I get on, if I have any feedback I'll send it here.
Matt, you've got the calibration parameters I used completely right! (I calibrated the foil references in the old Athena and then wrote my own programme to apply the calibration parameters to the large data sets... it looks like Larch could have saved me a bit of time there!)
Just an explanation about the structure of the files: The reason that they're 300 column files is that the beamline we recorded them at (ID24, ESRF) is capable of high time resolution in situ studies; obviously in the case of this data we can't get anything useful at that time resolution - but in many cases it makes sense to provide them as 300 individual spectra, at least initially.
Regarding the energy dispersive calibration: I fully understand that things you don't use won't be high on your priority list.
All the best,
Ian
----
Ian Godfrey
PhD Student,
UCL/JAIST Programme
Industrial Doctorate Centre in Molecular Modelling and Materials Science,
Department of Chemistry,
University College London
and
School of Materials Science,
Japan Advanced Institute of Science and Technology
i.godfrey@ucl.ac.uk i.godfrey@jaist.ac.jp
________________________________________
From: ifeffit-bounces@millenia.cars.aps.anl.gov
Dear All,
Here’s an issue I recently encountered while trying to handle some very large /in situ/ datasets in Athena; it may be by design/an inherent limitation, rather than a bug, though:
(I am running Demeter 0.9.21 pre and Demeter 0.9.20 , both x64 on Windows 8.1 x64 on separate machines).
Each of my datafiles contains over 300 columns of u(E) data as space delimited plain text, however when I try and import them to Athena the import window will only display checkboxes up to column 87.
I’ve attached an example datafile and a logfile to this email. (The data is very poor, but that shouldn’t be an issue for debugging…) I’ve also attached a copy of the datafile as it came from the beamline – this also displays the issue, so it’s nothing I’ve done to the file.
On a related note, is it possible for the energy dispersive data calibration to be added back to Athena? To get the calibration parameters for this data I had to use the (very) old version!
Thanks,
Ian
----
Ian Godfrey
PhD Student,
UCL/JAIST Programme
Industrial Doctorate Centre in Molecular Modelling and Materials Science,
Department of Chemistry,
University College London
and
School of Materials Science,
Japan Advanced Institute of Science and Technology
i.godfrey@ucl.ac.uk mailto:i.godfrey@ucl.ac.uk i.godfrey@jaist.ac.jp mailto:i.godfrey@jaist.ac.jp
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
-- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/ _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
participants (3)
-
Bruce Ravel -
Godfrey, Ian -
Matt Newville