request for comments on XDI and NeXuS data formatting
Hi Folks, As some of you are aware, there is a Q2XAFS meeting in a few weeks. I will be presenting some work on using NeXuS and HDF5 to store and distribute XAS data. This builds on and closely follows the XDI format from Bruce Ravel. While XDI is a great way to represent a single XAS spectrum, it is not able to contain multiple spectra. It also is not very specific about how to deal with data files with a large number (say, > 20) columns as might be found coming from many beamlines with multi-element detectors. I think that many of us are starting to see the need to regularly provide "raw" or "nearly raw" data as supplementary data for published articles or as part of datasets made available under FAIR data principles. While XDI could be one way to do that, I think it is also reasonable to reconsider formats other than plaintext files holding one spectrum. As I am preparing to present this, I've put together something like a blog post on a proof-of-concept for a NeXuS file format based on XDI at https://millenia.cars.aps.anl.gov/nxxas/. There are a few example data files using the proposed layout at https://millenia.cars.aps.anl.gov/nxxas/nexus_xas.html. There is also a Pull Request in with the NeXuS developers for this. If you are interested and have time over the next week or two, please read the pages above and let me know your thoughts on this. I know that not everyone is interested in this, but I also know that some who might be interested are not able to attend Q2XAFS. I think the discussion on this topic should be as wide as possible, which is why I am posting this here and before the meeting. Thanks, --Matt Newville <newville at cars.uchicago.edu>
Hi Matt et al.
The improved NeXuS format for XAS data looks fine to me. Good idea. But
it makes me wonder whether
a similar formatting for theoretical XAS data would be useful, especially
for analysis purposes.
Cheers,
John
Cheers,
John
On Sun, Aug 6, 2023 at 11:45 PM Matt Newville
Hi Folks,
As some of you are aware, there is a Q2XAFS meeting in a few weeks. I will be presenting some work on using NeXuS and HDF5 to store and distribute XAS data. This builds on and closely follows the XDI format from Bruce Ravel.
While XDI is a great way to represent a single XAS spectrum, it is not able to contain multiple spectra. It also is not very specific about how to deal with data files with a large number (say, > 20) columns as might be found coming from many beamlines with multi-element detectors. I think that many of us are starting to see the need to regularly provide "raw" or "nearly raw" data as supplementary data for published articles or as part of datasets made available under FAIR data principles. While XDI could be one way to do that, I think it is also reasonable to reconsider formats other than plaintext files holding one spectrum.
As I am preparing to present this, I've put together something like a blog post on a proof-of-concept for a NeXuS file format based on XDI at https://millenia.cars.aps.anl.gov/nxxas/ https://urldefense.com/v3/__https://millenia.cars.aps.anl.gov/nxxas/__;!!K-H.... There are a few example data files using the proposed layout at https://millenia.cars.aps.anl.gov/nxxas/nexus_xas.html https://urldefense.com/v3/__https://millenia.cars.aps.anl.gov/nxxas/nexus_xa.... There is also a Pull Request in with the NeXuS developers for this.
If you are interested and have time over the next week or two, please read the pages above and let me know your thoughts on this. I know that not everyone is interested in this, but I also know that some who might be interested are not able to attend Q2XAFS. I think the discussion on this topic should be as wide as possible, which is why I am posting this here and before the meeting.
Thanks,
--Matt Newville
https://urldefense.com/v3/__http://cars.uchicago.edu__;!!K-Hz7m0Vt54!kMFKt35... _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov
https://urldefense.com/v3/__http://millenia.cars.aps.anl.gov/mailman/listinf... Unsubscribe: https://urldefense.com/v3/__http://millenia.cars.aps.anl.gov/mailman/options...
Hi John,
The improved NeXuS format for XAS data looks fine to me. Good idea. But it makes me wonder whether a similar formatting for theoretical XAS data would be useful, especially for analysis purposes.
Thanks. That is an interesting question. I don't have an immediate answer.
I think it is not too difficult to read the plain text output files from
FEFF and FDMNES. I don't have much experience withther codes. That is, I
expect that plain text files with a few labeled columns is probably good
enough for "energy" and "calculated mu". I am not at all sure if there
could be common ways to label the other parts of the calculation like
broadening terms or partial DOS, etc. And, it would be great to be able
to extend any conventions to resonant emission/absorption data like RIXS
plane or q-dependent X-ray Raman.
But it would be worth discussing this, maybe with other theorists, people
doing multiple XANES+DFT++ calculations, and maybe people from the
Materials Project.
The situation for experimental data is sort of worse, even for beamlines
that mostly collect XAFS and at well-run facilities. Beamline "raw data
files" might have 100 columns, and will often vary beamline-by-beamline and
sometimes month-to-month. For transmission data, it is often easy to guess
how to build mu(E), but for fluorescence data, it is often very hard to
know what "the data" is. It's OK if the people using the data know what
to do and then publish the reduced data. But as we move (some faster than
others) toward being required to make all data available, this is becoming
a problem.
But, yes, we want to keep in mind that "experimental plain XAFS data"
should not be the only goal.
On Mon, Aug 7, 2023 at 12:44 PM John J Rehr
Hi Matt et al.
The improved NeXuS format for XAS data looks fine to me. Good idea. But it makes me wonder whether a similar formatting for theoretical XAS data would be useful, especially for analysis purposes.
Cheers, John
Cheers, John
On Sun, Aug 6, 2023 at 11:45 PM Matt Newville
wrote: Hi Folks,
As some of you are aware, there is a Q2XAFS meeting in a few weeks. I will be presenting some work on using NeXuS and HDF5 to store and distribute XAS data. This builds on and closely follows the XDI format from Bruce Ravel.
While XDI is a great way to represent a single XAS spectrum, it is not able to contain multiple spectra. It also is not very specific about how to deal with data files with a large number (say, > 20) columns as might be found coming from many beamlines with multi-element detectors. I think that many of us are starting to see the need to regularly provide "raw" or "nearly raw" data as supplementary data for published articles or as part of datasets made available under FAIR data principles. While XDI could be one way to do that, I think it is also reasonable to reconsider formats other than plaintext files holding one spectrum.
As I am preparing to present this, I've put together something like a blog post on a proof-of-concept for a NeXuS file format based on XDI at https://millenia.cars.aps.anl.gov/nxxas/ https://urldefense.com/v3/__https://millenia.cars.aps.anl.gov/nxxas/__;!!K-H.... There are a few example data files using the proposed layout at https://millenia.cars.aps.anl.gov/nxxas/nexus_xas.html https://urldefense.com/v3/__https://millenia.cars.aps.anl.gov/nxxas/nexus_xa.... There is also a Pull Request in with the NeXuS developers for this.
If you are interested and have time over the next week or two, please read the pages above and let me know your thoughts on this. I know that not everyone is interested in this, but I also know that some who might be interested are not able to attend Q2XAFS. I think the discussion on this topic should be as wide as possible, which is why I am posting this here and before the meeting.
Thanks,
--Matt Newville
https://urldefense.com/v3/__http://cars.uchicago.edu__;!!K-Hz7m0Vt54!kMFKt35... _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov
https://urldefense.com/v3/__http://millenia.cars.aps.anl.gov/mailman/listinf... Unsubscribe: https://urldefense.com/v3/__http://millenia.cars.aps.anl.gov/mailman/options...
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
-- --Matt Newville <newville at cars.uchicago.edu> 630-327-7411
participants (2)
-
John J Rehr
-
Matt Newville