[Ifeffit] [External] Re: XAS viewer unit detection during data reading
Matt Newville
newville at cars.uchicago.edu
Thu Mar 2 11:11:34 CST 2023
Hi Mary Jane, Edmund,
Thanks -- I agree with the sentiment that plain text is more "durable",
especially in contrast to HDF5. That was also part of the original argument
for XDI - we are very sure that those files will be usable in fifty years.
I also completely agree that supporting NeXuS/HDF5 is valuable.
Thanks for the data from P65 -- we'll make sure that is identified.
And, yes the Q2XAFS meeting will include a discussion of this -- I am
hoping to have something useful to say, so again if anyone has HDF5/NeXus
files of XAS data, or is at a facility that is currently generating these
or planning to generate these, please let me know!
On Thu, Mar 2, 2023 at 7:43 AM Edmund Welter <edmund.welter at desy.de> wrote:
> Dear Mary,
>
> fortunately HDF5/NEXUS and ASCII would not be mutually exclusive. If we
> start to use HDF5/NEXUS at the beamline, and I think we have to soon, we
> can always store an ASCII file with the most important data (energy axis,
> I0, I1, FluoData etc.) and meta data in parallel. Alternatively, any user
> could write a short Python (or whatever) script that extracts the important
> data into an ASCII file.
>
> There will be a whole session dedicated to the topic of "*Sharing and
> re-using XAS data"*, where data formats but also meta data (which data,
> which label etc.) at the Q2XAFS workshop in Melbourne in August this year.
> Info can be found here:
>
> https://xrayabsorption.org/q2xafs-2023/
>
> So, if anyone is interested in this or related topics concerning the broad
> field of "Quality Assurance" for XAFS spectroscopy, the Q2XAFS will be the
> place to present and discuss ideas!
>
> Best regards,
>
> Edmund
>
>
>
> On 01.03.23 21:25, Shultz, Mary Jane wrote:
>
> Chiming in from left field here.
>
> Strong bias to keep the data in ASCII. ASCII will be robust long after all
> other formats have passed into electronic oblivion. I insist that my
> students keep all original data in ASKII form, so am a bit biased.
>
> Thanks
>
> Mary
>
>
>
> *From:* Ifeffit <ifeffit-bounces at millenia.cars.aps.anl.gov>
> <ifeffit-bounces at millenia.cars.aps.anl.gov> *On Behalf Of *Matt Newville
> *Sent:* Wednesday, March 1, 2023 1:24 PM
> *To:* XAFS Analysis using Ifeffit <ifeffit at millenia.cars.aps.anl.gov>
> <ifeffit at millenia.cars.aps.anl.gov>
> *Subject:* [External] Re: [Ifeffit] XAS viewer unit detection during data
> reading
>
>
>
> Hi Edmund, Mauro,
>
>
>
> I think that maybe relaxing the test to be "if it starts with the letter E
> or e" and then also apply the current tests of the numerical data (sort of
> like: are the numbers increasing? is the minimum value between about 100
> and 150,000? is the average step size somewhere between 0.01 and 10?).
>
>
>
> But also, if you or anyone else has a "typical beamline file" -- and
> especially if there is a positive identifier for a beamline say the name of
> the beamline in the first couple of lines of the file -- we can add that
> to the files identified as XAFS data files.
>
>
>
> OK, slight diversion:
>
>
>
> I "almost agree" with Mauro's recommendation to distribute data in
> HDF5/NeXus format instead of ASCII files. ;).
>
>
>
> I am probably biased, but I think XDI (Ravel, et al J Synch Rad. 2012
> https://doi.org/10.1107/S0909049512036886 and also Ravel and Newville, J.
> Phys Conference, 2015 https://doi.org/10.1088/1742-6596/712/1/012148)
> would be the preferred way to exchange XAFS data. The 2012 paper actually
> discusses HDF5 as well.
>
>
>
> For HDF5/NeXus, I have a few slightly incompatible opinions that I think
> we should resolve:
>
> a) for many facilities, HDF5/NeXus is the preferred method for making
> all data available especially to comply with F.A.I.R. guidelines (or
> mandates).
>
> b) I use HDF5 every day. It is the right tool for some jobs, but it
> has some limitations.
>
> c) The current NeXus definition for XAS data is actually not very good
> and does not really comply with XDI or the discussion summarized in the J
> Synch Rad paper. I am not aware of anyone actually using this. I am
> willing to work on trying to get the NeXus XAS schema improved.
>
> d) We in the XAFS community should have a defined schema for NeXus
> that better complies with the 2012 paper and XDI.
>
>
>
> If anyone *is* collecting data into HDF5 or NeXus-like files, please send
> examples -- we want to support them in Larch, but also we want to get to a
> standard that is feasible. If anyone is interested in participating in
> discussions about "NeXus and/or HDF5 and/or similar for XAS", please
> contact me.
>
>
>
> --Matt
>
>
>
> On Wed, Mar 1, 2023 at 7:33 AM Edmund Welter <edmund.welter at desy.de>
> wrote:
>
> Dear Maurizio, Matt,
>
> I just checked why some older files from my beamline work correctly while
> the newer ones are not identified as xas data. After your explanation the
> reason is quite simple. In the old format the last line of the header was
>
> #enc_energy mono_energy mono_bragg Step_Pos mono_X1vert....
>
> After rewriting major parts of the beamline software among other things
> the last line of the header changed to:
>
> # E_enc E_step Bragg_enc Bragg_step...
>
> Once I replace E_enc with something like energy_enc or enc_energy the data
> is correctly identified as xas data.
>
> So, the easiest solution for data taken at P65 is that I will change the
> header in the data file so that the first column has a name that contains
> "energy". I think i remember that Athena was looking for the first column
> were the value of every point n+1 is larger than the value of point n. it
> then usually correctly identified that column to be the energy axis.
> However, I think labeling the columns correctly, that means following
> common accepted conventions, is much easier than programming a bunch of
> complicated fail safe energy axis recognising procedures.
>
> Cheers,
>
> Edmund
>
>
>
>
>
> On 28.02.23 18:16, Matt Newville wrote:
>
> Hi Siebe,
>
>
>
> Just to agree with Edmund and Mauro: selecting "energy" and changing data
> type from "raw" to "XAS" should help. But, you would have to do this for
> every file, which is not ideal.
>
>
>
> The file reader is trying to guess whether data is in energy (and if so,
> what units are used) and whether it is really XAS data or some other kind
> of data. I suspect that it is defaulting to "raw" because the energy column
> is not labeled "energy", but "col1" -- it sort of looks like the reader is
> not doing a good job of guessing what the columns mean.
>
>
>
> Posting a file would help diagnose why it is guessing wrong. The file
> reading does try to guess what beamline a data file came from, so if you
> send a file, maybe we can add enough hints for files from that beamline to
> work more reliably.
>
>
>
>
>
>
>
> On Tue, Feb 28, 2023 at 6:37 AM Welter, Edmund <edmund.welter at desy.de>
> wrote:
>
> Dear Siebe,
>
>
>
> did you try to change the data type from "raw" to XAS. If you click on the
> small arrow on the right you should have the choice between xas and raw. On
> my computer that works and after that I can choose the unit for the X-axis.
> For some reason "raw" seems to be the preset choice. Maybe Matt could
> change that, I fell into this trap several times.
>
>
>
> Cheers,
>
> Edmund
>
>
> ------------------------------
>
> *From: *"van der Veer, Siebe" <s.van.der.veer at rug.nl>
> *To: *"XAFS Analysis using Ifeffit" <ifeffit at millenia.cars.aps.anl.gov>
> *Sent: *Tuesday, 28 February, 2023 13:16:18
> *Subject: *[Ifeffit] XAS viewer unit detection during data reading
>
>
>
> Dear all,
>
> I am not sure if this mailing list is the correct medium for this
> question. Apologies, if not.
>
>
>
> I am looking to import files in XAS viewer, which worked fine in ATHENA,
> but in XAS viewer the correct units of the data are not detected and I
> cannot manually change them. I can get a plot of the data, but the data is
> not recognized by the software as XAS data, i.e. a lot of
> plotting/conversion options do not show.
>
>
>
> I hope the following screenshots provide sufficient information to
> illustrate my issue.
>
> Reading in a .dat file of fluorescence data. I can't change the units in
> the marked area.
>
>
>
>
>
> The data can be read into the XAS viewer, but I don't have the regular
> plotting options in these dropdown menus.
>
>
>
>
>
> Any advice would be much appreciated.
>
>
>
> Kind regards,
>
>
>
> --
>
> *Siebe van der Veer* | *PhD student*
>
> Nanostructures of Functional Oxides | Zernike Institute for Advanced
> Materials
>
> University of Groningen | Nijenborgh 4, 9747 AG Groningen | The Netherlands
>
> Email: s.van.der.veer at rug.nl |
>
>
> _______________________________________________
> Ifeffit mailing list
> Ifeffit at millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
> _______________________________________________
> Ifeffit mailing list
> Ifeffit at millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
>
>
>
> --
>
> --Matt Newville <newville at cars.uchicago.edu> 630-327-7411
>
>
>
> _______________________________________________
>
> Ifeffit mailing list
>
> Ifeffit at millenia.cars.aps.anl.gov
>
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
>
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
> _______________________________________________
> Ifeffit mailing list
> Ifeffit at millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
>
>
>
> --
>
> --Matt Newville <newville at cars.uchicago.edu> 630-327-7411
>
>
>
> *Caution:* This message originated from outside of the Tufts University
> organization. Please exercise caution when clicking links or opening
> attachments. When in doubt, email the TTS Service Desk at it at tufts.edu or
> call them directly at 617-627-3376.
>
>
>
> _______________________________________________
> Ifeffit mailing listIfeffit at millenia.cars.aps.anl.govhttp://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
> _______________________________________________
> Ifeffit mailing list
> Ifeffit at millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
--
--Matt Newville <newville at cars.uchicago.edu> 630-327-7411
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20230302/56ce92ba/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 64865 bytes
Desc: not available
URL: <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20230302/56ce92ba/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 7781 bytes
Desc: not available
URL: <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20230302/56ce92ba/attachment-0003.png>
More information about the Ifeffit
mailing list