Re: [Ifeffit] Ifeffit Digest, Vol 184, Issue 21
Dear all
Rebinning is always fraught; I would normally not allow a student to do it...
You might be interested in our paper [Schalken, M et al., JSR, 2018] which explicitly avoids rebinning and propagates uncertainty to ifeffit-like [Matt modified] analysis, efeffit.
This was an outcome from Q2XAFS.
We also include code in supplementary information.
Note that this also discusses interpolation and how to preserve information content if that is needed.
Best wishes
Chris
PS looking forward to great discussions at Poland
--------------------------------------------------------------------------------------
Christopher Chantler, Professor, FAIP, Fellow American Physical Society
Editor-in-Chief, Radiation Physics and Chemistry
Chair, International IUCr Commission on XAFS CIT, CCN
President, International Radiation Physics Society
School of Physics, University of Melbourne
Parkville Victoria 3010 Australia
+61-3-83445437 FAX +61-3-93474783
chantler@unimelb.edu.auhttps://owa.unimelb.edu.au/owa/redir.aspx?C=c7BoS0kVVkC1_S95-9x9l5cBu6YTjdAITgSrfUpfDAUV5oUH1LFYBcz08w8xvHMJoosZRdagfQM.&URL=mailto%3achantler%40unimelb.edu.au chantler@me.comhttps://owa.unimelb.edu.au/owa/redir.aspx?C=c7BoS0kVVkC1_S95-9x9l5cBu6YTjdAITgSrfUpfDAUV5oUH1LFYBcz08w8xvHMJoosZRdagfQM.&URL=mailto%3achantler%40me.com
http://optics.ph.unimelb.edu.au/~chantler/xrayopt/xrayopt.htmlhttps://owa.unimelb.edu.au/owa/redir.aspx?C=c7BoS0kVVkC1_S95-9x9l5cBu6YTjdAITgSrfUpfDAUV5oUH1LFYBcz08w8xvHMJoosZRdagfQM.&URL=http%3a%2f%2foptics.ph.unimelb.edu.au%2f%7echantler%2fxrayopt%2fxrayopt.html
http://optics.ph.unimelb.edu.au/~chantler/home.htmlhttps://owa.unimelb.edu.au/owa/redir.aspx?C=c7BoS0kVVkC1_S95-9x9l5cBu6YTjdAITgSrfUpfDAUV5oUH1LFYBcz08w8xvHMJoosZRdagfQM.&URL=http%3a%2f%2foptics.ph.unimelb.edu.au%2f%7echantler%2fhome.html
________________________________
From: Ifeffit
Yes, we measure fast and have taken as many as 20000 points. The problem is not in the shifts that you mention. This is normal and expected. the problem is specificallly in the rebinning algorithm in Demeter. It seems to be different than the one in the old Horae package. I have done a test of this and I attache a coule of figures that show the difference.
I have used 10 continuous scans for this test. The data were taken at the MRCAT beamline, Sector 10 at the APS. The data are for the Fe K-edge and there are about 3400 points per scan with a point density of about 0.35 eV/step. I used both versions of Athena and performed the following steps to give the data groups shown in the plots
new_athena.png Fe_new_rebin_merge - (blue) all 10 scans rebinned at input and then merged Fe_new_merge - (red) all 10 scans merged only Fe_new_merge_rebin - (green) all 10 scans merged then rebinned
old_athena.png Fe_old_rebin_merge - (blue) all 10 scans rebinned at input and then merged Fe_old_merge - (red) all 10 scans merged only Fe_old_merge_rebin - (green) all 10 scans merged then rebinned
comp_athena.png Fe_old_rebin_merge - (blue) Fe_new_rebin_merge - (red)
It is clear that the new Athena (Demeter) is not rebinning the same way as the old one (Horae). The contrast is particularly evident with the last plot. The new rebinning algorithm is introducing more noise. For the moment, I recommend only merging and perhaps smoothing if you can tolerate a bit of amplitude reduction.
I have been thinking that it might even be better to have the data acquisition software do the rebinning on the fly so the data does not have to be manipulated in Athena. I am not sure if this is a good idea yet but I think it would help my users.
Carlo
On Wed, 27 Jun 2018, Edmund Welter wrote:
Dear Carlo,
do you also measure as fast as possible in the sense that for two consecutive scans the points on the energy axis are not at the same positions? This is what happens at my beamline. The differences are typically very small but there are differences and one should not just add all the first points and all the second points and so on because they are not necessarily exactly at the same energy. Sometimes the beamline computer is doing something else in parallel (whatever that might be) and the distance between points A and B is significantly larger than the distance between B and C.
So, the problem is, at which point does it make sense to merge several spectra of the same sample? I presume that Athena is taking care of this when I use it to merge spectra, but it can only do so by interpolating the points in the spectrum onto a common grid before summing up the spectra.
The best solution might be to rebin/interpolate the spectra onto a fixed grid before they are imported into Athena (or any other program), depends on what Athena is exactly doing when it is rebinning data.
Another aspect is that Athena is not very happy about 8600 points/spectrum anyway, at least as long as it using Ifeffit.
Cheers,
Edmund
On 27.06.2018 15:14, Carlo Segre wrote:
Hi Ilya:
We always take data in this mode at APS Sector 10 and I have also find
the rebinning function is not working satisfactorily at this time. I find that for the current version of the software it is better to merge your data and let IFEFFIT interpolate to the dk=0.05 grid that it uses.
Carlo
On Wed, 27 Jun 2018, Ilya Sinev wrote:
Hi all,
I have a question regarding the chi(k) function isolation and rebinning processes. I have some data recorded in ?quasi channel-cut? modus, i.e. with the mono constantly moving and the data points collected with the highest possible rate. With 180 sec measurement in yields to a spectrum of ca. 8600 point, which obviously needs to be rebinned. The rebinned data, however, does not look good in k-space even if multiple data are merged. Moreover, I have an impression that the raw spectrum in k-space does not have those 8000+ points anymore but significantly less. Is there any reduction of
that the
data points number that is not seen (e.g. as a preparation step for FT)? Since the unbinned data has higher quality, does it then make more sense to keep using it for EXAFS analysis?
Thank you
Ilya Sinev
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
-- Carlo U. Segre -- Duchossois Leadership Professor of Physics Interim Chair, Department of Chemistry Director, Center for Synchrotron Radiation Research and Instrumentation Illinois Institute of Technology Voice: 312.567.3498 Fax: 312.567.3494 segre@iit.edu http://phys.iit.edu/~segre segre@debian.org _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
--
--Matt Newville
Dear Chris,
Rebinning is always fraught; I would normally not allow a student to do it...
I thought you would say that! ;-) and I share your concern, actually that is the reason why we store ~8600 data points or more for a scan of 1200 eV. Don't mess with the raw data! But 8600 (or 22000...) is a significant oversampling. We could simply measure for a certain longer time per point, than read the counters and store the values. The results turned out to be less good, some weird technical reason. We could also sum up n points and divide the result by n, so that we would end with 8600/n data points in the final data file. In most cases that would certainly be ok, but information would be irretrievably lost. But, at some point one has to reduce the number of points and make use of all the counts that were collected within a certain reasonable interval like 0.5 eV or 0.05 Ang-1. But it looks as if artifacts might arise from doing it twice, first explicitly by using Athena's rebin function and than again implicitly by mapping the data onto the 0.05 Ang-1 grid. Cheers, Edmund
Hi Edmund and Chris: I totally agree but for the same reason as Edmund, it is much more efficient to let the monochromator and not stop and start all the time. Rebinning is useful if you are sure that it works properly. Carlo On Thu, 28 Jun 2018, Edmund Welter wrote:
Dear Chris,
Rebinning is always fraught; I would normally not allow a student to do it...
I thought you would say that! ;-) and I share your concern, actually that is the reason why we store ~8600 data points or more for a scan of 1200 eV. Don't mess with the raw data! But 8600 (or 22000...) is a significant oversampling. We could simply measure for a certain longer time per point, than read the counters and store the values. The results turned out to be less good, some weird technical reason. We could also sum up n points and divide the result by n, so that we would end with 8600/n data points in the final data file. In most cases that would certainly be ok, but information would be irretrievably lost.
But, at some point one has to reduce the number of points and make use of all the counts that were collected within a certain reasonable interval like 0.5 eV or 0.05 Ang-1. But it looks as if artifacts might arise from doing it twice, first explicitly by using Athena's rebin function and than again implicitly by mapping the data onto the 0.05 Ang-1 grid.
Cheers, Edmund
-- Carlo U. Segre -- Duchossois Leadership Professor of Physics Interim Chair, Department of Chemistry Director, Center for Synchrotron Radiation Research and Instrumentation Illinois Institute of Technology Voice: 312.567.3498 Fax: 312.567.3494 segre@iit.edu http://phys.iit.edu/~segre segre@debian.org
Hi When it comes to rebinning and/or averaging, my preferred way is to use smoothing spline interpolation as implemented by IgorPro from Wavemetrics. With a little care, smooth curves with no artifacts can be obtained, in particular with oversampled data. We use it a lot with XPS data, where in my opinion the usual box averaging is really bad, producing lots of irrelevant wiggles in the base line, whereas smoothing splines do not have this problem. The reference for the algorithm is: "Smoothing by Spline Functions", Christian H. Reinsch, Numerische Mathematik 10. There are probably other implementations to do this. Michael.
Am 28.06.2018 um 20:20 schrieb Carlo Segre
: Hi Edmund and Chris:
I totally agree but for the same reason as Edmund, it is much more efficient to let the monochromator and not stop and start all the time. Rebinning is useful if you are sure that it works properly.
Carlo
On Thu, 28 Jun 2018, Edmund Welter wrote:
Dear Chris,
Rebinning is always fraught; I would normally not allow a student to do it... I thought you would say that! ;-) and I share your concern, actually that is the reason why we store ~8600 data points or more for a scan of 1200 eV. Don't mess with the raw data! But 8600 (or 22000...) is a significant oversampling. We could simply measure for a certain longer time per point, than read the counters and store the values. The results turned out to be less good, some weird technical reason. We could also sum up n points and divide the result by n, so that we would end with 8600/n data points in the final data file. In most cases that would certainly be ok, but information would be irretrievably lost.
But, at some point one has to reduce the number of points and make use of all the counts that were collected within a certain reasonable interval like 0.5 eV or 0.05 Ang-1. But it looks as if artifacts might arise from doing it twice, first explicitly by using Athena's rebin function and than again implicitly by mapping the data onto the 0.05 Ang-1 grid.
Cheers, Edmund
participants (4)
-
Carlo Segre
-
Christopher Chantler
-
Edmund Welter
-
Karl-Michael Schindler