Hi Siebe, Just to agree with Edmund and Mauro: selecting "energy" and changing data type from "raw" to "XAS" should help. But, you would have to do this for every file, which is not ideal. The file reader is trying to guess whether data is in energy (and if so, what units are used) and whether it is really XAS data or some other kind of data. I suspect that it is defaulting to "raw" because the energy column is not labeled "energy", but "col1" -- it sort of looks like the reader is not doing a good job of guessing what the columns mean. Posting a file would help diagnose why it is guessing wrong. The file reading does try to guess what beamline a data file came from, so if you send a file, maybe we can add enough hints for files from that beamline to work more reliably. On Tue, Feb 28, 2023 at 6:37 AM Welter, Edmund wrote:

Dear Siebe,

did you try to change the data type from "raw" to XAS. If you click on the small arrow on the right you should have the choice between xas and raw. On my computer that works and after that I can choose the unit for the X-axis. For some reason "raw" seems to be the preset choice. Maybe Matt could change that, I fell into this trap several times.

Cheers, Edmund

------------------------------ *From: *"van der Veer, Siebe" *To: *"XAFS Analysis using Ifeffit" *Sent: *Tuesday, 28 February, 2023 13:16:18 *Subject: *[Ifeffit] XAS viewer unit detection during data reading

Dear all, I am not sure if this mailing list is the correct medium for this question. Apologies, if not.

I am looking to import files in XAS viewer, which worked fine in ATHENA, but in XAS viewer the correct units of the data are not detected and I cannot manually change them. I can get a plot of the data, but the data is not recognized by the software as XAS data, i.e. a lot of plotting/conversion options do not show.

I hope the following screenshots provide sufficient information to illustrate my issue.

Reading in a .dat file of fluorescence data. I can't change the units in the marked area.

[image: image.png]

The data can be read into the XAS viewer, but I don't have the regular plotting options in these dropdown menus.

[image: image.png]

Any advice would be much appreciated.

Kind regards,

-- *Siebe van der Veer* | *PhD student* Nanostructures of Functional Oxides | Zernike Institute for Advanced Materials University of Groningen | Nijenborgh 4, 9747 AG Groningen | The Netherlands Email: s.van.der.veer@rug.nl |

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

-- --Matt Newville <newville at cars.uchicago.edu> 630-327-7411

Dear Edmund, That's great that you have find why the ASCII data parser in Larch was not detecting automatically the energy column from the labels in the header. The current choice is to look for "energy" or "ene" in the header to identify a data column as energy array. Please, if you are willing to publicly share and example data file from P65, we could include it in the xafsdata examples here: https://github.com/xraypy/xraylarch/tree/master/examples/xafsdata/beamlines Those files are used for running the automatic tests and permit us improving the quality of Larch's code. This is valid for any beamline who wants to contribute, even if I strongly encourage distributing data in HDF5/NeXus format instead of ASCII files. Best, Mauro On 01/03/2023 14:31, Edmund Welter wrote:

Dear Maurizio, Matt,

I just checked why some older files from my beamline work correctly while the newer ones are not identified as xas data. After your explanation the reason is quite simple. In the old format the last line of the header was

#enc_energy mono_energy mono_bragg Step_Pos mono_X1vert....

After rewriting major parts of the beamline software among other things the last line of the header changed to:

# E_enc E_step Bragg_enc Bragg_step...

Once I replace E_enc with something like energy_enc or enc_energy the data is correctly identified as xas data.

So, the easiest solution for data taken at P65 is that I will change the header in the data file so that the first column has a name that contains "energy". I think i remember that Athena was looking for the first column were the value of every point n+1 is larger than the value of point n. it then usually correctly identified that column to be the energy axis. However, I think labeling the columns correctly, that means following common accepted conventions, is much easier than programming a bunch of complicated fail safe energy axis recognising procedures.

Cheers,

Edmund

On 28.02.23 18:16, Matt Newville wrote:

...

Hi Siebe,

Just to agree with Edmund and Mauro:  selecting "energy" and changing data type from "raw" to "XAS" should help.  But, you would have to do this for every file, which is not ideal.

The file reader is trying to guess whether data is in energy (and if so, what units are used) and whether it is really XAS data or some other kind of data. I suspect that it is defaulting to "raw" because the energy column is not labeled "energy", but "col1" -- it sort of looks like the reader is not doing a good job of guessing what the columns mean.

Posting a file would help diagnose why it is guessing wrong.  The file reading does try to guess what beamline a data file came from, so if you send a file, maybe we can add enough hints for files from that beamline to work more reliably.

On Tue, Feb 28, 2023 at 6:37 AM Welter, Edmund wrote:

Dear Siebe,

did you try to change the data type from "raw" to XAS. If you click on the small arrow on the right you should have the choice between xas and raw. On my computer that works and after that I can choose the unit for the X-axis. For some reason "raw" seems to be the preset choice. Maybe Matt could change that, I fell into this trap several times.

Cheers, Edmund

------------------------------------------------------------------------ *From: *"van der Veer, Siebe" *To: *"XAFS Analysis using Ifeffit" *Sent: *Tuesday, 28 February, 2023 13:16:18 *Subject: *[Ifeffit] XAS viewer unit detection during data reading

Dear all, I am not sure if this mailing list is the correct medium for this question. Apologies, if not.

I am looking to import files in XAS viewer, which worked fine in ATHENA, but in XAS viewer the correct units of the data are not detected and I cannot manually change them. I can get a plot of the data, but the data is not recognized by the software as XAS data, i.e. a lot of plotting/conversion options do not show.

I hope the following screenshots provide sufficient information to illustrate my issue.

Reading in a .dat file of fluorescence data. I can't change the units in the marked area.

image.png

The data can be read into the XAS viewer, but I don't have the regular plotting options in these dropdown menus.

image.png

Any advice would be much appreciated.

Kind regards,

-- *Siebe van der Veer* | *PhD student* Nanostructures of Functional Oxides | Zernike Institute for Advanced Materials University of Groningen | Nijenborgh 4, 9747 AG Groningen | The Netherlands Email: s.van.der.veer@rug.nl|

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

-- --Matt Newville http://cars.uchicago.edu> 630-327-7411

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe:http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

Chiming in from left field here. Strong bias to keep the data in ASCII. ASCII will be robust long after all other formats have passed into electronic oblivion. I insist that my students keep all original data in ASKII form, so am a bit biased. Thanks Mary From: Ifeffit On Behalf Of Matt Newville Sent: Wednesday, March 1, 2023 1:24 PM To: XAFS Analysis using Ifeffit Subject: [External] Re: [Ifeffit] XAS viewer unit detection during data reading Hi Edmund, Mauro, I think that maybe relaxing the test to be "if it starts with the letter E or e" and then also apply the current tests of the numerical data (sort of like: are the numbers increasing? is the minimum value between about 100 and 150,000? is the average step size somewhere between 0.01 and 10?). But also, if you or anyone else has a "typical beamline file" -- and especially if there is a positive identifier for a beamline say the name of the beamline in the first couple of lines of the file -- we can add that to the files identified as XAFS data files. OK, slight diversion: I "almost agree" with Mauro's recommendation to distribute data in HDF5/NeXus format instead of ASCII files. ;). I am probably biased, but I think XDI (Ravel, et al J Synch Rad. 2012 https://doi.org/10.1107/S0909049512036886 and also Ravel and Newville, J. Phys Conference, 2015 https://doi.org/10.1088/1742-6596/712/1/012148) would be the preferred way to exchange XAFS data. The 2012 paper actually discusses HDF5 as well. For HDF5/NeXus, I have a few slightly incompatible opinions that I think we should resolve: a) for many facilities, HDF5/NeXus is the preferred method for making all data available especially to comply with F.A.I.R. guidelines (or mandates). b) I use HDF5 every day. It is the right tool for some jobs, but it has some limitations. c) The current NeXus definition for XAS data is actually not very good and does not really comply with XDI or the discussion summarized in the J Synch Rad paper. I am not aware of anyone actually using this. I am willing to work on trying to get the NeXus XAS schema improved. d) We in the XAFS community should have a defined schema for NeXus that better complies with the 2012 paper and XDI. If anyone *is* collecting data into HDF5 or NeXus-like files, please send examples -- we want to support them in Larch, but also we want to get to a standard that is feasible. If anyone is interested in participating in discussions about "NeXus and/or HDF5 and/or similar for XAS", please contact me. --Matt On Wed, Mar 1, 2023 at 7:33 AM Edmund Welter mailto:edmund.welter@desy.de> wrote: Dear Maurizio, Matt, I just checked why some older files from my beamline work correctly while the newer ones are not identified as xas data. After your explanation the reason is quite simple. In the old format the last line of the header was #enc_energy mono_energy mono_bragg Step_Pos mono_X1vert.... After rewriting major parts of the beamline software among other things the last line of the header changed to: # E_enc E_step Bragg_enc Bragg_step... Once I replace E_enc with something like energy_enc or enc_energy the data is correctly identified as xas data. So, the easiest solution for data taken at P65 is that I will change the header in the data file so that the first column has a name that contains "energy". I think i remember that Athena was looking for the first column were the value of every point n+1 is larger than the value of point n. it then usually correctly identified that column to be the energy axis. However, I think labeling the columns correctly, that means following common accepted conventions, is much easier than programming a bunch of complicated fail safe energy axis recognising procedures. Cheers, Edmund On 28.02.23 18:16, Matt Newville wrote: Hi Siebe, Just to agree with Edmund and Mauro: selecting "energy" and changing data type from "raw" to "XAS" should help. But, you would have to do this for every file, which is not ideal. The file reader is trying to guess whether data is in energy (and if so, what units are used) and whether it is really XAS data or some other kind of data. I suspect that it is defaulting to "raw" because the energy column is not labeled "energy", but "col1" -- it sort of looks like the reader is not doing a good job of guessing what the columns mean. Posting a file would help diagnose why it is guessing wrong. The file reading does try to guess what beamline a data file came from, so if you send a file, maybe we can add enough hints for files from that beamline to work more reliably. On Tue, Feb 28, 2023 at 6:37 AM Welter, Edmund mailto:edmund.welter@desy.de> wrote: Dear Siebe, did you try to change the data type from "raw" to XAS. If you click on the small arrow on the right you should have the choice between xas and raw. On my computer that works and after that I can choose the unit for the X-axis. For some reason "raw" seems to be the preset choice. Maybe Matt could change that, I fell into this trap several times. Cheers, Edmund ________________________________ From: "van der Veer, Siebe" mailto:s.van.der.veer@rug.nl> To: "XAFS Analysis using Ifeffit" mailto:ifeffit@millenia.cars.aps.anl.gov> Sent: Tuesday, 28 February, 2023 13:16:18 Subject: [Ifeffit] XAS viewer unit detection during data reading Dear all, I am not sure if this mailing list is the correct medium for this question. Apologies, if not. I am looking to import files in XAS viewer, which worked fine in ATHENA, but in XAS viewer the correct units of the data are not detected and I cannot manually change them. I can get a plot of the data, but the data is not recognized by the software as XAS data, i.e. a lot of plotting/conversion options do not show. I hope the following screenshots provide sufficient information to illustrate my issue. Reading in a .dat file of fluorescence data. I can't change the units in the marked area. [cid:image001.png@01D94C52.02A690C0] The data can be read into the XAS viewer, but I don't have the regular plotting options in these dropdown menus. [cid:image002.png@01D94C52.02A690C0] Any advice would be much appreciated. Kind regards, -- Siebe van der Veer | PhD student Nanostructures of Functional Oxides | Zernike Institute for Advanced Materials University of Groningen | Nijenborgh 4, 9747 AG Groningen | The Netherlands Email: s.van.der.veer@rug.nlmailto:s.van.der.veer@rug.nl | _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.govmailto:Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.govmailto:Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit -- --Matt Newville http://cars.uchicago.edu> 630-327-7411 _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.govmailto:Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.govmailto:Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit -- --Matt Newville http://cars.uchicago.edu> 630-327-7411 Caution: This message originated from outside of the Tufts University organization. Please exercise caution when clicking links or opening attachments. When in doubt, email the TTS Service Desk at it@tufts.edumailto:it@tufts.edu or call them directly at 617-627-3376.

Hi Mary Jane, Edmund, Thanks -- I agree with the sentiment that plain text is more "durable", especially in contrast to HDF5. That was also part of the original argument for XDI - we are very sure that those files will be usable in fifty years. I also completely agree that supporting NeXuS/HDF5 is valuable. Thanks for the data from P65 -- we'll make sure that is identified. And, yes the Q2XAFS meeting will include a discussion of this -- I am hoping to have something useful to say, so again if anyone has HDF5/NeXus files of XAS data, or is at a facility that is currently generating these or planning to generate these, please let me know! On Thu, Mar 2, 2023 at 7:43 AM Edmund Welter wrote:

Dear Mary,

fortunately HDF5/NEXUS and ASCII would not be mutually exclusive. If we start to use HDF5/NEXUS at the beamline, and I think we have to soon, we can always store an ASCII file with the most important data (energy axis, I0, I1, FluoData etc.) and meta data in parallel. Alternatively, any user could write a short Python (or whatever) script that extracts the important data into an ASCII file.

There will be a whole session dedicated to the topic of "*Sharing and re-using XAS data"*, where data formats but also meta data (which data, which label etc.) at the Q2XAFS workshop in Melbourne in August this year. Info can be found here:

https://xrayabsorption.org/q2xafs-2023/

So, if anyone is interested in this or related topics concerning the broad field of "Quality Assurance" for XAFS spectroscopy, the Q2XAFS will be the place to present and discuss ideas!

Best regards,

Edmund

On 01.03.23 21:25, Shultz, Mary Jane wrote:

Chiming in from left field here.

Strong bias to keep the data in ASCII. ASCII will be robust long after all other formats have passed into electronic oblivion. I insist that my students keep all original data in ASKII form, so am a bit biased.

Thanks

Mary

*From:* Ifeffit *On Behalf Of *Matt Newville *Sent:* Wednesday, March 1, 2023 1:24 PM *To:* XAFS Analysis using Ifeffit *Subject:* [External] Re: [Ifeffit] XAS viewer unit detection during data reading

Hi Edmund, Mauro,

I think that maybe relaxing the test to be "if it starts with the letter E or e" and then also apply the current tests of the numerical data (sort of like: are the numbers increasing? is the minimum value between about 100 and 150,000? is the average step size somewhere between 0.01 and 10?).

But also, if you or anyone else has a "typical beamline file" -- and especially if there is a positive identifier for a beamline say the name of the beamline in the first couple of lines of the file -- we can add that to the files identified as XAFS data files.

OK, slight diversion:

I "almost agree" with Mauro's recommendation to distribute data in HDF5/NeXus format instead of ASCII files. ;).

I am probably biased, but I think XDI (Ravel, et al J Synch Rad. 2012 https://doi.org/10.1107/S0909049512036886 and also Ravel and Newville, J. Phys Conference, 2015 https://doi.org/10.1088/1742-6596/712/1/012148) would be the preferred way to exchange XAFS data. The 2012 paper actually discusses HDF5 as well.

For HDF5/NeXus, I have a few slightly incompatible opinions that I think we should resolve:

a) for many facilities, HDF5/NeXus is the preferred method for making all data available especially to comply with F.A.I.R. guidelines (or mandates).

b) I use HDF5 every day. It is the right tool for some jobs, but it has some limitations.

c) The current NeXus definition for XAS data is actually not very good and does not really comply with XDI or the discussion summarized in the J Synch Rad paper. I am not aware of anyone actually using this. I am willing to work on trying to get the NeXus XAS schema improved.

d) We in the XAFS community should have a defined schema for NeXus that better complies with the 2012 paper and XDI.

If anyone *is* collecting data into HDF5 or NeXus-like files, please send examples -- we want to support them in Larch, but also we want to get to a standard that is feasible. If anyone is interested in participating in discussions about "NeXus and/or HDF5 and/or similar for XAS", please contact me.

--Matt

On Wed, Mar 1, 2023 at 7:33 AM Edmund Welter wrote:

Dear Maurizio, Matt,

I just checked why some older files from my beamline work correctly while the newer ones are not identified as xas data. After your explanation the reason is quite simple. In the old format the last line of the header was

#enc_energy mono_energy mono_bragg Step_Pos mono_X1vert....

After rewriting major parts of the beamline software among other things the last line of the header changed to:

# E_enc E_step Bragg_enc Bragg_step...

Once I replace E_enc with something like energy_enc or enc_energy the data is correctly identified as xas data.

So, the easiest solution for data taken at P65 is that I will change the header in the data file so that the first column has a name that contains "energy". I think i remember that Athena was looking for the first column were the value of every point n+1 is larger than the value of point n. it then usually correctly identified that column to be the energy axis. However, I think labeling the columns correctly, that means following common accepted conventions, is much easier than programming a bunch of complicated fail safe energy axis recognising procedures.

Cheers,

Edmund

On 28.02.23 18:16, Matt Newville wrote:

Hi Siebe,

Just to agree with Edmund and Mauro: selecting "energy" and changing data type from "raw" to "XAS" should help. But, you would have to do this for every file, which is not ideal.

The file reader is trying to guess whether data is in energy (and if so, what units are used) and whether it is really XAS data or some other kind of data. I suspect that it is defaulting to "raw" because the energy column is not labeled "energy", but "col1" -- it sort of looks like the reader is not doing a good job of guessing what the columns mean.

Posting a file would help diagnose why it is guessing wrong. The file reading does try to guess what beamline a data file came from, so if you send a file, maybe we can add enough hints for files from that beamline to work more reliably.

On Tue, Feb 28, 2023 at 6:37 AM Welter, Edmund wrote:

Dear Siebe,

did you try to change the data type from "raw" to XAS. If you click on the small arrow on the right you should have the choice between xas and raw. On my computer that works and after that I can choose the unit for the X-axis. For some reason "raw" seems to be the preset choice. Maybe Matt could change that, I fell into this trap several times.

Cheers,

Edmund

------------------------------

*From: *"van der Veer, Siebe" *To: *"XAFS Analysis using Ifeffit" *Sent: *Tuesday, 28 February, 2023 13:16:18 *Subject: *[Ifeffit] XAS viewer unit detection during data reading

Dear all,

I am not sure if this mailing list is the correct medium for this question. Apologies, if not.

I am looking to import files in XAS viewer, which worked fine in ATHENA, but in XAS viewer the correct units of the data are not detected and I cannot manually change them. I can get a plot of the data, but the data is not recognized by the software as XAS data, i.e. a lot of plotting/conversion options do not show.

I hope the following screenshots provide sufficient information to illustrate my issue.

Reading in a .dat file of fluorescence data. I can't change the units in the marked area.

The data can be read into the XAS viewer, but I don't have the regular plotting options in these dropdown menus.

Any advice would be much appreciated.

Kind regards,

--

*Siebe van der Veer* | *PhD student*

Nanostructures of Functional Oxides | Zernike Institute for Advanced Materials

University of Groningen | Nijenborgh 4, 9747 AG Groningen | The Netherlands

Email: s.van.der.veer@rug.nl |

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

--

--Matt Newville <newville at cars.uchicago.edu> 630-327-7411

_______________________________________________

Ifeffit mailing list

Ifeffit@millenia.cars.aps.anl.gov

http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit

Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

--

--Matt Newville <newville at cars.uchicago.edu> 630-327-7411

*Caution:* This message originated from outside of the Tufts University organization. Please exercise caution when clicking links or opening attachments. When in doubt, email the TTS Service Desk at it@tufts.edu or call them directly at 617-627-3376.

_______________________________________________ Ifeffit mailing listIfeffit@millenia.cars.aps.anl.govhttp://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

-- --Matt Newville <newville at cars.uchicago.edu> 630-327-7411