RE: [Ifeffit] Too many points in pre-edge
Carlo and Matt:
With these files, the location of the edge (and element identification as well) using the first derivative does not work.
I also had a similar problem when we had a somewhat noisy pre-edge and edge data. Often our analysis of the XANES requires high confidence in the edge energy to determine energy shifts and to accurately align our spectra using metal foil references. Of course, I can always manually assign the edge position in Athena, but it would be nice if Ifeffit was a bit smarter about how it chooses Eo since it is critical for Athena operation when a reference standard is used to calibrate a sample spectra. I would guess that there is some robust algorithm that one could implement which would more accurately determine either (1) the maximum in the first derivative or (2) the zero of the second derivative for data with any number of data points or noisy data. I imagine that such an algorithm would include some sort of adjustable "smoothing" parameter to filter the signal from noise. Dave _________________________________ David Barton The Dow Chemical Company Catalysis R&D 1776 Building Midland, MI 48674 USA Ph: (989) 636-4753 Fax: (989) 638-9350 Email: dgbarton@dow.com
-----Original Message----- From: ifeffit-bounces@millenia.cars.aps.anl.gov [mailto:ifeffit-bounces@millenia.cars.aps.anl.gov] On Behalf Of Carlo U. Segre Sent: Thursday, December 02, 2004 9:44 AM To: IFEFFIT Mailing List Subject: [Ifeffit] Too many points in pre-edge
Matt:
I am running into a somewhat annoying problem with continuous scan data imports into athena where there are many datapoints before the edge (I think > 400 or so). With these files, the location of the edge (and element identification as well) using the first derivative does not work. The edge is identified as being somewhere in the pre-edge. As I reduce the number of data points in the pre-edge, the edge identification moves closer to the true edge and then locks in correctly when the number of data points in the pre-edge is small enough (again, I think this is less than 400 points).
I can address this problem by running these data through a rebinning process in the pre-edge but I think that if it is possible to handle this in ifeffit, that would be a good idea.
On a second note, would you tell me whether you or Bruce is planning to maintain the ifeffit perl wrapper? At this time, ifeffit 1.2.6a and horae 046 have different versions of the Ifeffit.pm script (1.3 and 1.301 respectively). I recently ran into a problem because I had the ifeffit version and horae 046.
Carlo
-- Carlo U. Segre -- Professor of Physics Associate Dean for Special Projects, Graduate College Illinois Institute of Technology Voice: 312.567.3498 Fax: 312.567.3494 Carlo.Segre@iit.edu http://www.iit.edu/~segre _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov
On Thursday 02 December 2004 02:17 pm, Barton, David (DG) wrote:
Carlo and Matt:
With these files, the location of the edge (and element identification as well) using the first derivative does not work.
I also had a similar problem when we had a somewhat noisy pre-edge and edge data. Often our analysis of the XANES requires high confidence in the edge energy to determine energy shifts and to accurately align our spectra using metal foil references. Of course, I can always manually assign the edge position in Athena, but it would be nice if Ifeffit was a bit smarter about how it chooses Eo since it is critical for Athena operation when a reference standard is used to calibrate a sample spectra. I would guess that there is some robust algorithm that one could implement which would more accurately determine either (1) the maximum in the first derivative or (2) the zero of the second derivative for data with any number of data points or noisy data. I imagine that such an algorithm would include some sort of adjustable "smoothing" parameter to filter the signal from noise.
Carlo's problme seems to be a little bit different. His data are not particularly noisy. The problem appears to be correlated to the length in steps of the pre-edge. I, too, would be interested in a perfectly robust against noise algorithm for finding the edge energy. However, I suspect that when someone builds a better algorithm, somoeone else will build noisier data ;-) Athena does have a feature that might be some help in this case. In the "bkg" section of the preferences, one of the parameters is "e0". This is set to "derivative" by default, telling Athena to use Ifeffit's first derivative based algorithm to determine e0. Unless the noise in the pre-edge is really large, the "half" option for that parameter might be helpful. It's described right there on the preferences page. Another things that might help is Athena's pre-processing of data as it's imported. If all goes well, then you will have to set e0 by hand for the first data set, but choosing the "set all parameter values" option in the processing dialog should set all the rest to reasonable values. In any case, if you have noisy data, you really must be prepared to do hands-on data analysis. Ifeffit and Athena will only ever be so smart, but data can be arbitrarily crappy! B -- Bruce Ravel ----------------------------------- ravel@phys.washington.edu Code 6134, Building 3, Room 405 Naval Research Laboratory phone: (1) 202 767 2268 Washington DC 20375, USA fax: (1) 202 767 4642 NRL Synchrotron Radiation Consortium (NRL-SRC) Beamlines X11a, X11b, X23b National Synchrotron Light Source Brookhaven National Laboratory, Upton, NY 11973 My homepage: http://feff.phys.washington.edu/~ravel EXAFS software: http://feff.phys.washington.edu/~ravel/software/exafs/
Grant has been using a differential evolution algorithm to take case of noisy data. Perhaps this could help? Carlo On Thu, 2 Dec 2004, Bruce Ravel wrote:
I, too, would be interested in a perfectly robust against noise algorithm for finding the edge energy. However, I suspect that when someone builds a better algorithm, somoeone else will build noisier data ;-)
Athena does have a feature that might be some help in this case. In the "bkg" section of the preferences, one of the parameters is "e0". This is set to "derivative" by default, telling Athena to use Ifeffit's first derivative based algorithm to determine e0. Unless the noise in the pre-edge is really large, the "half" option for that parameter might be helpful. It's described right there on the preferences page.
Another things that might help is Athena's pre-processing of data as it's imported. If all goes well, then you will have to set e0 by hand for the first data set, but choosing the "set all parameter values" option in the processing dialog should set all the rest to reasonable values.
In any case, if you have noisy data, you really must be prepared to do hands-on data analysis. Ifeffit and Athena will only ever be so smart, but data can be arbitrarily crappy!
B
-- Carlo U. Segre -- Professor of Physics Associate Dean for Special Projects, Graduate College Illinois Institute of Technology Voice: 312.567.3498 Fax: 312.567.3494 Carlo.Segre@iit.edu http://www.iit.edu/~segre
I agree that the algorithm to find E0 could be improved, especially for noisy and XANES-only data. The ideas Bruce added to athena seem like good approaches. FWIW, Ifeffit's E0 finding strategy is mostly tuned to avoid glitches in the extended XAFS. Therefore, it choose the point that has the largest first derivative provided that the previous two points have a positive first derivative. It only looks in the first half of the energy array (counting by array index). It's quite possible that it would miss e0 in data with an unexpectedly long pre-edge. Carlo wrote:
Grant has been using a differential evolution algorithm to take case of noisy data. Perhaps this could help?
I have no idea what that is. Does it help you and Grant? --Matt
On Thursday 02 December 2004 04:40 pm, Matt Newville wrote:
I agree that the algorithm to find E0 could be improved, especially for noisy and XANES-only data. The ideas Bruce added to athena seem like good approaches. FWIW, Ifeffit's E0 finding strategy is mostly tuned to avoid glitches in the extended XAFS. Therefore, it choose the point that has the largest first derivative provided that the previous two points have a positive first derivative.
It only looks in the first half of the energy array (counting by array index).
That would explain what was going on in Carlo's data.
It's quite possible that it would miss e0 in data with an unexpectedly long pre-edge.
I can see the argument that ifeffit should have been able to figure this out, but it does seem kind of silly to bin the data in the pre-edge such that the energy steps are about 1/2 volt. That vastly overspecifies the pre-edge.
Carlo wrote:
Grant has been using a differential evolution algorithm to take case of noisy data. Perhaps this could help?
I have no idea what that is. Does it help you and Grant?
Ditto. I would like to see a reference. B -- Bruce Ravel ----------------------------------- ravel@phys.washington.edu Code 6134, Building 3, Room 405 Naval Research Laboratory phone: (1) 202 767 2268 Washington DC 20375, USA fax: (1) 202 767 4642 NRL Synchrotron Radiation Consortium (NRL-SRC) Beamlines X11a, X11b, X23b National Synchrotron Light Source Brookhaven National Laboratory, Upton, NY 11973 My homepage: http://feff.phys.washington.edu/~ravel EXAFS software: http://feff.phys.washington.edu/~ravel/software/exafs/
On Thu, 2 Dec 2004, Bruce Ravel wrote:
On Thursday 02 December 2004 04:40 pm, Matt Newville wrote:
It's quite possible that it would miss e0 in data with an unexpectedly long pre-edge.
I can see the argument that ifeffit should have been able to figure this out, but it does seem kind of silly to bin the data in the pre-edge such that the energy steps are about 1/2 volt. That vastly overspecifies the pre-edge.
As I mentioned earlier, these data are taken in continuous scan mode. The monochromator is set to run at a constant speed and data is taken on a constant time interval basis with a multichannel scaler. This results in a high point density over the entire range. We sum spectra and rebin in the EXAFS region so that we can keep all the statistics possible (we had an extensive discussion of this on this list about a year ago). However, it is of use not to reduce the point density of the edge so we just don't rebin before the edge. Since the pre-edge is normally flat, we chose not to alter it at all. This results in the problem that we are discussing. I think that we might as well change our program to rebin the pre-edge as well and this will solve the problem Carlo -- Carlo U. Segre -- Professor of Physics Associate Dean for Special Projects, Graduate College Illinois Institute of Technology Voice: 312.567.3498 Fax: 312.567.3494 Carlo.Segre@iit.edu http://www.iit.edu/~segre
participants (4)
-
Barton, David (DG) -
Bruce Ravel -
Carlo U. Segre -
Matt Newville