[Ifeffit] The difference of EXAFS oscillation when using Athena and when using Larch

Tue Apr 18 14:15:00 CDT 2017

Dear Matt 

Thank you for detailed explanation about autobk() function. I could understand very well. 

It might be caused by data points being very dense and large as you say.

When I analyzed with Larch using the data which tried to interpolate the energy, the noise certainly decreased (attached as “PdO_q_intp.ex3 and EXAFSanalysis_v3.py”).

I had thought that autobk() function in Ifeffit(Athena) does "quatric interpolation". But, I didn’t know that the boxcar average smoothing was applied in Athena.

Anyway, smoothing may be done using boxcar average if necessary.

Sadly, I am not sure how to use Larch's algorithm of boxcar averaging convolution or Savitzky-Golay method. Although I confirmed Larch HP (http://cars.uchicago.edu/xraylarch/), I could not find out. 

please tell us how to use the algorithm if you can.

Larch is a great library for me. 

I hope to further develop XAFS analysis using Larch.

Best wishes,

Yuji Mahara

From: Ifeffit [mailto:ifeffit-bounces �� millenia.cars.aps.anl.gov] On Behalf Of Matt Newville
Sent: Tuesday, April 18, 2017 11:52 AM
To: XAFS Analysis using Ifeffit <ifeffit �� millenia.cars.aps.anl.gov>
Subject: Re: [Ifeffit] The difference of EXAFS oscillation when using Athena and when using Larch

Hi Yuji,

On Mon, Apr 17, 2017 at 12:22 PM, Yuji Mahara <mahara.yuji �� e.mbox.nagoya-u.ac.jp <mailto:mahara.yuji �� e.mbox.nagoya-u.ac.jp> > wrote:

Dear Matt

Thank you very much for your quick reply.

I attached three files: a raw data obtained from SPring-8 in Japan (PdO.q), .chik file obtained using Athena based on PdO.q (PdO.q.chik), and a python script I actually analyzed with Larch (EXAFSanalaysis.py).

(When opening PdO.q with Athena we use the plugin named "PFBL12C".)

If you put three files in the same directory and run “EXAFSanalysis.py”, you can see the previous PdO.png.

I am looking forward to hearing from you. 

Thank you very much for the nice report.  I can reproduce what you see, and looked into this a bit more.  The energy data you have is very finely spaced (the mu and energy arrays have 3991 energy points), leading to some question about how to convert that finely spaced energy data to the chi(k) grid of 0.05 Ang^-1.

In both Ifeffit and Larch, the autobk() function determine an array for mu0(E) and chi(E) at all the energy data points provided.  But to use later, chi(k) needs to be put onto evenly spaced values of k, with a step size of 0.05 Ang^-1 being both common and appropriate for EXAFS.

For your data (see a modified version of your script, attached), there are 30 energy points between 15.0 and 15.10 Ang^-1.  It's not obvious how to best use those data to give the value chi(k) at k=15.05.

The autobk() function in Ifeffit essentially does a boxcar average, taking those ~15 data points closest to k=15.05 and averaging them with equal weighting. 

The autobk function in Larch takes the spline function of mu0(E) and evaluates that at the k-values of the grid, essentially doing a cubic-splice interpolation of the surrounding points to give the value at a particular k.

I think this difference in interpolation explains the difference in chi(k) and chi(R) you see -- the Ifeffit data is smoother and so lower amplitude at high R.  The high frequency "noise" has been filtered out by the boxcar average.   The data processed by Larch does not impose this filter, so the data retains the higher-frequency components.  

The attached plot (from the the attached modified version of your EXAFSAnalysis.py) includes the data processed with your script, with Athena using the Larch backend, and with Athena using the Ifeffit backend.  There are slight differences at low-R -- I didn't worry too much about setting clamps.

I don't have a strong opinion, but think preserving the noise / high frequency parts of the data is probably preferred.  You can smooth it with a variety of methods either before or after background subtraction (and  Larch comes with a few common smoothing algorithms including boxcar averaging, convolution and the Savitzky-Golay method).  

When I do continuous XAFS, I bin the data to energy points selected to be spaced 0.05Ang^-1 apart.  That effectively boxcar averages the data at the point of collection.   I don't know that there is always a single best way to do that averaging/smoothing, but it would be interesting to hear the approach others take and seeing how we might be able to unify and/or simplify that part of the data reduction.

--Matt

Best,

Yuji

From: Ifeffit [mailto:ifeffit-bounces �� millenia.cars.aps.anl.gov <mailto:ifeffit-bounces �� millenia.cars.aps.anl.gov> ] On Behalf Of Matt Newville
Sent: Tuesday, April 18, 2017 12:45 AM
To: XAFS Analysis using Ifeffit <ifeffit �� millenia.cars.aps.anl.gov <mailto:ifeffit �� millenia.cars.aps.anl.gov> >
Subject: Re: [Ifeffit] The difference of EXAFS oscillation when using Athena and when using Larch

Hi Yuji,

On Mon, Apr 17, 2017 at 9:49 AM, Yuji Mahara <mahara.yuji �� e.mbox.nagoya-u.ac.jp <mailto:mahara.yuji �� e.mbox.nagoya-u.ac.jp> > wrote:

Dear All, 

I am Yuji Mahara at Nagoya University, Japan.

I am using Athena in Demeter 0.9.24 package and Larch 0.9.33 on windows 10. 

At that time, I am worrying about the difference of EXAFS oscillation when using Athena and when using Larch.

k3-weighted Pd K-edge EXAFS and Fourier transform EXAFS spectrum for PdO bulk are shown in attached file(PdO.png).

When XAFS raw data is read using Athena and EXAFS oscillation is obtained, EXAFS spectrum is smooth and noise is small. On the other hand, using Larch, noise is noticeable in k space. 

This phenomenon occurred when analyzing other XAFS data as well.

I tried to analyze XAFS data by Athena and Larch with the almost same conditions.

Does this difference of EXAFS results arise from the difference in autobk's algorithm between Athena and Larch?

The analysis conditions are shown below. 

If someone knows this reason, please let me know.

There may be subtle differences in the results from autobk() from Ifeffit and autobk() from Larch, but they should not be very big.   If I recall correctly, the meaning of the values for the "clamps" is different between the two, but I don't think that accounts for the changes you're seeing.

Like you say, the differences seem to be more about the noise in the data which is much more noticeable in the data processed with Larch than with Athena, as if the data is being smoothed or interpolated differently.  if the data is very finely spaced or binned in some unexpected way from a QXAFS collection, that might help explain the difference, though I'm not sure of that or what behavior would be preferred.  

Is it possible to post one of the data files?

--Matt

Best regards,

Yuji Mahara

******************************************
■ Yuji Mahara
Nagoya University(Japan)  D3
Graduate School of Engineering
Furo-cho, Chikusa-ku, Nagoya-shi, Aichi, 466-8603, Japan
・Phone : +81-52-789-3192 <tel:%2B81-52-789-3192> 
・E-mail :  <mailto:mahara.yuji �� e.mbox.nagoya-u.ac.jp> mahara.yuji �� e.mbox.nagoya-u.ac.jp
******************************************

#####Analysis condition of Athena #####

e0 -->[ set e0 to a fraction of the edge step] --> 24355.9037323484 <tel:%28903%29%20732-3484> 

Normalization order: 3

Pre-edge range: from -150 to -60 eV

Normalization range: from 150 to 1100 eV

Rbkg: 1

k-weight used by spline: 2

Spline clamp: low=None, high=Strong

Spline range in k: from 0 to 17 

Fourier transform k-range: from 3 to 15, dk=0.5, window=’hanning’

#####Analysis condition of Larch #####

dat.e0 = 24355.9037323484 <tel:%28903%29%20732-3484>  <-- e0 was estimated by Athena

pre_edge(dat.e, dat.mu <http://dat.mu> , group=dat, e0=dat.e0, pre1=-150, pre2=-60, norm1=150, norm2=1100, nnorm=2, _larch=mylarch) 

autobk(dat.e, dat.mu <http://dat.mu> , group=dat, e0=dat.e0, rbkg=1, nclamp=5, clamp_lo=0, clamp_hi=24, kstep=0.05, kmin=0, kmax=17, kweight=2, win='hanning', _larch=mylarch) 

xftf(dat.k, dat.chi, group=dat, kmin=3, kmax=15, dk=0.5, window='hanning', kweight=3, _larch=mylarch)

#########################

_______________________________________________
Ifeffit mailing list
Ifeffit �� millenia.cars.aps.anl.gov <mailto:Ifeffit �� millenia.cars.aps.anl.gov> 
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

_______________________________________________
Ifeffit mailing list
Ifeffit �� millenia.cars.aps.anl.gov <mailto:Ifeffit �� millenia.cars.aps.anl.gov> 
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

-------------- next part --------------
HTML����������������������������...
URL: <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20170419/f370465c/attachment-0001.html>
-------------- next part --------------
��������������������������������������������...
����������: EXAFSanalysis_v3.py
��:         application/octet-stream
������:     3046 ������
����:       ����
URL:        <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20170419/f370465c/attachment-0002.obj>
-------------- next part --------------
��������������������������������������������...
����������: PdO_q_intp.ex3
��:         application/octet-stream
������:     14504 ������
����:       ����
URL:        <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20170419/f370465c/attachment-0003.obj>
-------------- next part --------------
��������������������������������������������...
����������: PdO_v3.png
��:         image/png
������:     142566 ������
����:       ����
URL:        <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20170419/f370465c/attachment-0001.png>