ARE: [Ifeffit] Re: Ifeffit Digest, Vol 5, Issue 1

[Matt Newville]

On Fri, 4 Jul 2003 SCHLEGEL Michel 177447 <schlegel at azurite.cea.fr> wrote:

> > I totally agree.  This can lead to big problems in the data.  We
> > accidentally tried it once and then made sure that the 
> > <If/Io> average was performed instead.
> 
> I think the best way to do it statistically is to weight each 
> individual XAS spectrum with 1/s^2, where s is an estimate of 
> the noise in the spectrum. This makes sure that a noisy 
> spectrum does not alter the overall quality of the fit . Of 
> course, a simpler way to do it is to retain only the spectra 
> of overall similar -and hopefully good - quality, and discard 
> the noisy ones. 

Just to be clear on the original point: setting mu = <If>/<Io> is a
very bad idea. Getting <Io> to look at the variations in Io itself
might be useful, however.

I do agree that weighted averages of data based on the noise in the
data could be helpful, but getting a good estimate for the noise in
data can be tricky.

> I think an evaluation of the spectrum noise may be obtained 
> from the FT contributions peaking at high 'distances' (e.g.,
> > 10Å). Hence, with a correct script, we should be able to:
> - extract chi(k) functions from individual XAS spectra
> - perform FTs for these spectra,
> - perform inverse FT on the spectra over a given limit, and 
>   calculate s from that,
> - usecalculated s values to sum weight-average individual 
>   XAS spectra.
> 
> As I'm a very lousy IFEFFIT user (means that I've stopped 
> short from hitting buttons on Athena and Artemis), I would 
> humbly ask Bruce if there's any chance to see such a methodology
> implemented on Athena one of these days (not in a hurry, though) 

As Bruce pointed out, Ifeffit's chi_noise() command will estimate
the noise in chi(k) with essentially this procedure (averaging
chi(R) between 15 and 25Ang and using Parseval's theorem to give the
noise in chi(k)).  For noisy data with known counting statistics,
and tests with known random noise added to spectra, this works well.

This method will miss noise that depends strongly on R.  These
"non-white" portions are usually interpreted as "systematic errors".  
Whether these are really systematic measurement errors is not
well-established.  Nearly everyone (including me) believes ths
non-whate terms to dominate the noise, though there's not a lot of
hard evidence for this either.

Of course, what you want is the noise in chi that is at the same
frequencies as the signal you're interested in.  That is *very* hard
to estimate.  Scan-to-scan variations can be useful to look at, but
are, by themselves, fairly poor measures of uncertainty in the data.

Scott Calvin wrote:
 
> It is my experience that on some beamlines there are spurious
> effects, such as those related to monochromator lock-in, pumps,
> etc. that introduce high-frequency (i. e. high r) oscillations
> without affecting oscillations in the 1-10 Angstrom range. The
> method IFEFFIT uses for estimating errors in fitted parameters is
> fortunately not dependent on this error estimate, although some
> statistical information, such as the value of chi-squared, is. But
> using high-r noise to weight spectra during merges may rely more
> about the behavior of a pump or a feedback mechanism than on the
> actual noise within the FT's region of interest.

I'd assumed that vibrations would actually cause fairly white noise,
though feedback mechanisms could skew towards high frequency. Other
effects (temperature/pressure/flow fluctuations in ion chamber gases
and optics) might skew toward low-frequency noises.  I have not seen
many studies of vibrations, feedback mechanism, or other
beamline-specific effects on data quality, and none discussing the
spectral weight of the beamline-specific noise.

On the other hand, all data interpolation schemes do some smoothing,
which suppresses high frequency components.  And it usually appears
that the high-frequency estimate of the noise from chi_noise() or
Feffit gives an estimate that is significantly *low*.

Anyway, I think using the 'epsilon_k' that chi_noise() estimates as
the noise in chi(k) is a fine way to do a weighted averages of data.
It's not perfect, but neither is anything else.

--Matt