[Ifeffit] Ifeffit Digest, Vol 162, Issue 17

Matthew Marcus mamarcus at lbl.gov
Fri Aug 12 17:10:33 CDT 2016 

Topic 3: Incorrect dark current (offset) is just one of many ways that glitches can escape being normalized out, and one of the few you can correct in software.  Others include but are
not limited to

1.	Any detection-system nonlinearity in I0 or detected channel (F or T).  This includes deadtime in fluorescence detectors.

2.	Inadequate detection speed in QXAS so that the recording of I0 lags behind that of fluorescence (or vice versa if using a Lytle detector or similar).

3.	Harmonic/stray beam content varies while going through the glitch.

4.	Beam position or shape varies while going through the glitch.  If the crystals are in a diverging beam, then each part of the crystal sees a slightly
	different incidence angle, so the exact energy at which the two reflections contributing to the glitch both satisfy the Bragg condition varies across
	the crystal surface, resulting in non-uniform reflection.

There are undoubtely many more.
	mam

On 8/12/2016 2:36 PM, Christopher Thomas Chantler wrote:
> Topic 1: We have a routine /edit within ifeffit (our modified version) which propagates and fits uncertainty.
> Working on a couple of minor details before passing it on to Matt and Bruce for general use.
>
> Topic 3: Monochromator glitches should be eliminated in absorption spectra under normal circumstances if the dark current estimation is well defined. I recommend careful measurement and recording of this for all spectra.
>
> ------------------------------------------------------------
> Christopher Chantler, Professor, FAIP
> Editor-in-Chief, Radiation Physics and Chemistry
> Chair, International IUCr Commission on XAFS
> President, International Radiation Physics Society
> School of Physics, University of Melbourne
> Parkville Victoria 3010 Australia
> +61-3-83445437 FAX +61-3-93474783
> chantler at unimelb.edu.au chantler at me.com
> http://optics.ph.unimelb.edu.au/~chantler/xrayopt/xrayopt.html
> http://optics.ph.unimelb.edu.au/~chantler/home.html
>
>
> ________________________________________
> From: Ifeffit [ifeffit-bounces at millenia.cars.aps.anl.gov] on behalf of ifeffit-request at millenia.cars.aps.anl.gov [ifeffit-request at millenia.cars.aps.anl.gov]
> Sent: Friday, 12 August 2016 11:22 PM
> To: ifeffit at millenia.cars.aps.anl.gov
> Subject: Ifeffit Digest, Vol 162, Issue 17
>
> Send Ifeffit mailing list submissions to
>         ifeffit at millenia.cars.aps.anl.gov
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> or, via email, send a message with subject or body 'help' to
>         ifeffit-request at millenia.cars.aps.anl.gov
>
> You can reach the person managing the list at
>         ifeffit-owner at millenia.cars.aps.anl.gov
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Ifeffit digest..."
>
>
> Today's Topics:
>
>    1. Statistical errors in linear combination fits (Joshua Kas)
>    2. Re: Statistical errors in linear combination fits (Bruce Ravel)
>    3. Athena interpolation when removing mono glitches
>       (Michael Gaultois)
>    4. Re: Athena interpolation when removing mono glitches (Bruce Ravel)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 11 Aug 2016 13:31:45 -0700
> From: Joshua Kas <joshua.j.kas at gmail.com>
> To: ifeffit at millenia.cars.aps.anl.gov
> Subject: [Ifeffit] Statistical errors in linear combination fits
> Message-ID:
>         <CAHuhYRm_Dx5+_HAUuOZ0nCwzXwCrS-Subxb9dmBXDsQp8wmSRQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi All,
> I was wondering if it is possible to pass a value for the uncertainty in
> the data to Athena when doing linear combination fits. We have calculated
> chi^2 values using a simple estimate of the statistical uncertainty, and
> find that our values differ by several factors of 10 when compared to the
> reported values from Athena. I assume that this has to do with value of the
> uncertainty that Athena is using, but I certainly could be mistaken. In any
> case, the reduced chi^2 reported by Athena is much smaller than 1, while
> the fit is off by fairly large amounts compared to any reasonable estimate
> of the statistical error.
> Thanks in advance for any help and sorry if this info is already available
> in the archives,
> Josh Kas
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20160811/ebcf5347/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 11 Aug 2016 16:43:36 -0400
> From: Bruce Ravel <bravel at bnl.gov>
> To: XAFS Analysis using Ifeffit <ifeffit at millenia.cars.aps.anl.gov>
> Subject: Re: [Ifeffit] Statistical errors in linear combination fits
> Message-ID: <507e138f-9ab8-9de3-0629-bfd85aad1e1f at bnl.gov>
> Content-Type: text/plain; charset=windows-1252; format=flowed
>
> On 08/11/2016 04:31 PM, Joshua Kas wrote:
>> I was wondering if it is possible to pass a value for the uncertainty in
>> the data to Athena when doing linear combination fits. We have
>> calculated chi^2 values using a simple estimate of the statistical
>> uncertainty, and find that our values differ by several factors of 10
>> when compared to the reported values from Athena. I assume that this has
>> to do with value of the uncertainty that Athena is using, but I
>> certainly could be mistaken. In any case, the reduced chi^2 reported by
>> Athena is much smaller than 1, while the fit is off by fairly large
>> amounts compared to any reasonable estimate of the statistical error.
>
> Not so easy with Ifeffit, quite easier with larch.  It's on my to do
> list, but has not yet been implemented in Athena.
>
> B
>
> --
>   Bruce Ravel  ------------------------------------ bravel at bnl.gov
>
>   National Institute of Standards and Technology
>   Synchrotron Science Group at NSLS-II
>   Building 743, Room 114
>   Upton NY, 11973
>
>   Homepage:    http://bruceravel.github.io/home/
>   Software:    https://github.com/bruceravel
>   Demeter:     http://bruceravel.github.io/demeter/
>
>
> ------------------------------
>
> Message: 3
> Date: Fri, 12 Aug 2016 11:56:47 +0000
> From: Michael Gaultois <mike.g at usask.ca>
> To: "ifeffit at millenia.cars.aps.anl.gov"
>         <ifeffit at millenia.cars.aps.anl.gov>
> Subject: [Ifeffit] Athena interpolation when removing mono glitches
> Message-ID:
>         <CAADTZ036AFLb_nWjkS3XYS=vguPi+R01GVOWXA5+eK0bJuxdew at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Dear members of the Ifeffit list,
>
> I recently collected some EXAFS data with some significant monochromator
> glitches that I am looking to remove. I have used a python script
> graciously written by the beamline scientist to remove the offending
> regions, but when I import the data into Athena, Athena does some funny
> business in an attempt to join together the regions outside of the data
> gap. (See the bending away in the dataset and/or attached image.) I have
> confirmed by plotting with other software that the strange step-like
> behaviour in the mu(E) is present only after importing into Athena (the raw
> data is fine).
>
> I have looked through the mailing list archives and also the user manual,
> but can't seem to find anything that explains it, or other people who have
> experienced this problem in the past. From what I can determine, Athena
> joins together the segments to obtain a linear interpolation in the
> norm(E)? This leads to a warping in the mu(E).
> ==How does Athena try to treat this data?==
>
> I was wondering if other people have had similar issues, and what steps can
> be taken to remedy the problem. For example, replacing removed data points
> with artificial points along a linear interpolation would be possible, but
> the act of adding artificial points that don't exist is concerning to me.
> ==What is the best way to treat data with mono glitches to reduce spurious
> features not intrinsic to the sample?==
>
> If you are interested, I have included links to .prj datasets and images to
> highlight these problems below.
>
> With thanks for your time,
> Michael
>
> ----------
> .prj file with 4 ways of working up the same data:
> http://bit.ly/2bnfNZ5
>
> 1) raw data
> 2) mono glitches removed
> 3) rebinned data
> 4) rebinned and manually removed points (This leads to some strange-looking
> features in k-space, and this would be less than desireable on the many
> datasets we have collected)
>
> images to highlight these problems:
> a) mu(E)
> http://bit.ly/2aYaJIf
>
> b) norm(E)
> http://bit.ly/2aYaX27
>
> c) k
> http://bit.ly/2baWh0h
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/20160812/9b376ceb/attachment-0001.html>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 12 Aug 2016 09:22:22 -0400
> From: Bruce Ravel <bravel at bnl.gov>
> To: XAFS Analysis using Ifeffit <ifeffit at millenia.cars.aps.anl.gov>
> Subject: Re: [Ifeffit] Athena interpolation when removing mono
>         glitches
> Message-ID: <c9b4007d-f976-b62d-f1b1-17c8d18c80e1 at bnl.gov>
> Content-Type: text/plain; charset=windows-1252; format=flowed
>
>
> Michael,
>
> Let's start by talking about where chi(k) comes from in the software.
>
> mu(E) is measured on some grid.  Eventually, we want chi(k) to be on a
> clear, specified, reasonably (but not excessively) dense grid in k.
> chi(E) is (mu(E) - mu0(E)) / mu0(E0).  This places chi(E) on the same
> grid as the original data.
>
> To prepare for the Fourier transform and the comparison with theory,
> chi(E) needs to be converted to k and put onto the specified k grid.
> In our software, this is done by interpolation.  The interpolation is
> local and does not consider the density of the grid in E.  So long as
> the grid in E is not very dense compared to the wavelengths
> represented in the data [1], this works fine.  However, with a very
> dense E-grid, it is possible to come up with a situation where this
> local interpolation -- i.e. using a few points from the original grid
> which surround the point on the target k-grid -- will yield
> undesirable results.
>
> How do I know it's possible for the interpolation to go all wonky like
> this?  You contrived just such a situation :)
>
> Your deglitching step removed about 40 eV worth of data, which
> corresponded to about 0.5/Ang. in the region of 11.5/Ang in k -- about
> 10 points on the k-grid.  This is a BIG gap in the data.
>
> The normal presumption when deglitching is that you are only removing
> a few points of data.  If you are removing a huge gash from your data
> -- as you did -- you need to think hard about how you fill that gash
> back in.
>
> Your data set #2 is a pretty bad solution.  Your data set #4 seems
> like a much better solution.  Let's examine why.
>
> In data set #2 you contrived exactly the problem I described above.
> You left a gash in the data which was about 10 grid steps wide.  When
> going from chi(E) to chi(k), the interpolation was done.  At the first
> data point in the gap, the previous few data points were used in the
> interpolation.  Since those few points were pointing down-ish, the
> first point in the gap was lower than the baseline.  The next point
> interpolated even lower, the next even lower.  Eventually, the data
> had to hook back up with the other side of the gap, so the
> interpolation rose back up.  By cutting a bug gash out of the data and
> doing a simple interpolation, you introduced the weird, downward
> pointing feature at 11.5.
>
> Data set 4 is much more sensible solution because you rebinned before
> deglitching.  Rebinning is implemented in Athena as a convolution with
> a square kernel -- this is often called a box car average.  This
> passes the convolution kernel over the data and evaluates it at the
> target grid points.  The data before rebinning were on a very dense
> grid.  After rebinning, they are on the "conventional" energy grid
> that was used back in the day when we all did step scans and for which
> the background removal algorithm was originally written and optimized.
> By rebinning first, the data are smoother and sparser and less
> susceptible to the interpolation effect that you saw in data set #2.
>
>
> Soooooo ..... is Athena broken?  An argument could be made that the
> interpolation is the problem and the solution should be to make a
> better algorithm.  There is merit to that, but I am going to argue
> something else.
>
> I think the problem is that beamlines are implementing quick scanning
> without thinking about all the needs of their users.  Your original
> data consists of about 4000 points.  It is rebinned to about 600
> points -- about the size of a conventional step scan.
>
> Last week, one of my colleagues here at BNL showed me a quick scan
> which had almost 200,000 points in it.  Yowzers!
>
> My question to the beamline scientists out there is this: what do your
> users want and need?  While it is possible that you might have a power
> user with a good reason to examine the data file with 4000 or 200,000
> points [2], most of your users want the data rebinned onto a
> conventional grid.  So why are beamlines sending their users home with
> data in a format that is not what they want?  That's just bad
> practice.
>
> Had you simply gone home [3] with your quick scan data rebinned onto a
> sensible grid, you would never have even noticed this problem because
> you would have naturally fallen into the case of data set #4.
>
> You should demand that from your beamline scientist.
>
> B
>
>
>
>
>
> [1] I mean the part of the data that is EXAFS, not the part of the
>      data that is unnormalized monochromator glitch.
>
> [2] ... and it is certainly true that the beamline scientist needs to
>      examine the large, dense, raw data file ...
>
> [3] "But what about the sanctity of raw data" is something that I am
>      sure someone is sputtering right now.  Well, to take the example
>      of Diamond, /all/ the data are streamed into an HDF5 file.  Column
>      data files are then written for the convenience of the user.
>      That's a great solution.  The HDF5 file has all the things but the
>      user can interact with the salient representation of the
>      measurement.
>
>
>
>
> On 08/12/2016 07:56 AM, Michael Gaultois wrote:
>> Dear members of the Ifeffit list,
>>
>> I recently collected some EXAFS data with some significant monochromator
>> glitches that I am looking to remove. I have used a python script
>> graciously written by the beamline scientist to remove the offending
>> regions, but when I import the data into Athena, Athena does some funny
>> business in an attempt to join together the regions outside of the data
>> gap. (See the bending away in the dataset and/or attached image.) I have
>> confirmed by plotting with other software that the strange step-like
>> behaviour in the mu(E) is present only after importing into Athena (the
>> raw data is fine).
>>
>> I have looked through the mailing list archives and also the user
>> manual, but can't seem to find anything that explains it, or other
>> people who have experienced this problem in the past. From what I can
>> determine, Athena joins together the segments to obtain a linear
>> interpolation in the norm(E)? This leads to a warping in the mu(E).
>> ==How does Athena try to treat this data?==
>>
>> I was wondering if other people have had similar issues, and what steps
>> can be taken to remedy the problem. For example, replacing removed data
>> points with artificial points along a linear interpolation would be
>> possible, but the act of adding artificial points that don't exist is
>> concerning to me.
>> ==What is the best way to treat data with mono glitches to reduce
>> spurious features not intrinsic to the sample?==
>>
>> If you are interested, I have included links to .prj datasets and images
>> to highlight these problems below.
>>
>> With thanks for your time,
>> Michael
>>
>> ----------
>> .prj file with 4 ways of working up the same data:
>> http://bit.ly/2bnfNZ5
>>
>> 1) raw data
>> 2) mono glitches removed
>> 3) rebinned data
>> 4) rebinned and manually removed points (This leads to some
>> strange-looking features in k-space, and this would be less than
>> desireable on the many datasets we have collected)
>>
>> images to highlight these problems:
>> a) mu(E)
>> http://bit.ly/2aYaJIf
>>
>> b) norm(E)
>> http://bit.ly/2aYaX27
>>
>> c) k
>> http://bit.ly/2baWh0h
>>
>>
>> _______________________________________________
>> Ifeffit mailing list
>> Ifeffit at millenia.cars.aps.anl.gov
>> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
>> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>>
>
>
> --
>   Bruce Ravel  ------------------------------------ bravel at bnl.gov
>
>   National Institute of Standards and Technology
>   Synchrotron Science Group at NSLS-II
>   Building 743, Room 114
>   Upton NY, 11973
>
>   Homepage:    http://bruceravel.github.io/home/
>   Software:    https://github.com/bruceravel
>   Demeter:     http://bruceravel.github.io/demeter/
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Ifeffit mailing list
> Ifeffit at millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
>
> ------------------------------
>
> End of Ifeffit Digest, Vol 162, Issue 17
> ****************************************
>
> _______________________________________________
> Ifeffit mailing list
> Ifeffit at millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>

More information about the Ifeffit mailing list