[Ifeffit] How to do with diffraction peak

[Matt Newville]

Hi Matthew,

On Fri, 14 Dec 2012, Matthew Marcus wrote:

> On 12/14/2012 4:10 PM, Matt Newville wrote:
>> Hi Matthew,
>> 
>> On Thu, Dec 13, 2012 at 11:28 PM, Matthew Marcus <mamarcus at lbl.gov> wrote:
>> 
>> Oh, I meant that for each energy point measured you could use the
>> different fluorescence channels to compute a mean value for I_f and
>> standard deviation for I_f, and then propagate both measurement and
>> uncertainty through the entire analysis.  For a point with a glitch,
>> the standard deviation would be very high, and so that region of k
>> space would receive less weight.
>> 
>> I'd view data not measured not as having a value of 0 but having
>> infinite uncertainty.  In that sense, removing a glitch doesn't do any
>> harm - we're treating it as if we have no measurement at that energy
>> point.
>
> If I understand you correctly, you're saying that 'deglitch with reference 
> scalers' is OK, but with some additional
> error estimate imposed on the points you had to do that to.

No, I'm saying first that it is OK to simply remove a glitch -- 
you didn't measure mu(E) at that energy, you measured something 
else.

Alternatively, one could keep the glitch but say that the 
uncertainty at that energy is extremely large.   This approach 
would require one to propagate the uncertainties per point 
throughout the analysis -- not a bad idea, but not normally 
done.

Your method of replacing an outlier in a set of I_f values 
with the mean value is also fine with me -- it effectively 
removes the bad point, preserving the average I_f.

Alternatively, one could use not the simple sum or mean of 
the set of I_f values, but use both the mean and variance, and 
propagate these both all the way through the analysis.

The advantage of propagating the mean and variance is that one 
may not correctly identifiy small glitches or tails of glitches 
or other artifacts not due to counting statistics,  but their 
effect would be handled as well as possible.

> I've found no reason to suspect that any artifacts were introduced by this 
> procedure because the ratio of the "bad" to the "good"
> counters is quite slowly-varying except in the glitch region, so 
> interpolation is safe.  Therefore, the only extra noise is
> shot noise due to counting with fewer elements.  Now, given how non-rigorous 
> our treatment of noise is, I think that the
> extra noise is not going to influence the fit significantly.
>
> The real problem is what to do when you can't do any kind of signal-based 
> data recovery, but have to either
> acknowledge that you don't have any data there or make some up and assign 
> some sort of uncertainty to your curve-drawing
> artistry.

Right.  I think acknowledging you don't have the data is the 
best approach.

>>>> I agree with your larger point that we can definitely afford the CPU
>>>> cycles to do better analysis than use something simply because it is
>>>> Fast.  I'm not sure that a slow discrete FT would be significantly
>>>> different than an FFT, though.  I think (but am listening...) that the
>>>> continuous v discrete FT is like classic audiophile snobbery about the
>>>> superiority of analog over digital music.  With sufficient sampling
>>>> and resolution and dynamic range in the signal, the difference has to
>>>> be trivial.  I'm willing to believe that a digital recording has less
>>>> dynamic range, but I'm also willing to believe that this is swamped by
>>>> my stereo and ears (and I mostly listen to mp3s through cheap
>>>> headphones these days, so I clearly am not buying a tube amplifier
>>>> anytime soon...).
>>> 
>>> You're right that the DFT is exactly equivalent to the FFT.  However, the
>>> VSFT is different in that
>>> it works on non-uniformly tabulated data with no interpolation.
>>> I've heard of an AB test in which Golden Ears were challenged with two 
>>> amps,
>>> one of which was solid-state
>>> and the other also solid-state but with some 2nd-harmonic distortion and 
>>> (I
>>> think) noise added to simulate
>>> a tube amp.  Guess which won?  Somebody once wrote a parody in which
>>> mercury-filled speaker cables were
>>> advertised for their "liquid, shimmering sound", but I digress.
>> 
>> I agree / understand that a VSFT is or can be different from a DFT.
>> For me the point is that with sufficient sampling and bit depth, the
>> differences go to zero.  And I think we know how to sample XAFS well
>> enough.  For digital audio there's a fair argument about bit depth /
>> dynamic range and whether people can actually detect frequencies about
>> 18KHz.  But there aren't such arguments (yet for EXAFS): no one is
>> (yet) measuring EXAFS reliably to 50 Ang^-1.
>
> Sampling is exactly the problem - if you have gaps in the data, then you're 
> not sampled densely in the
> relevant region.

Yes, sampling is the problem. But, removing a point or two from a 
0.05Ang^-1 grid won't do much damage to the signal below 8 Ang.

You could even mask out a full 0.5 Ang^-1 from the chi(k) and 
still do OK.   This is essentially the earlier suggestion for
using a notched window around a glitch or region of glitches -- 
it is possible to ignore portions of an oscillatory function and 
still recover phase/amplitude.  Sure, it's not ideal, but it's 
actually not that bad for the lower frequencies.

>  If you assume the data to be band-limited, then you could 
> in principle interpolate.

The data is limited. We're not going to be modeling EXAFS beyond 
50 Ang^-1 or beyond 20 Ang anytime soon.  Possibly ever. 
Samplying at 0.05Ang^-1 would let us measure to 31 Ang (well, 
aliasing might push that down,  but only if you believe non-zero 
signal past 31 Ang!).  That is, we over-sample the EXAFS below 
8 Ang well enough to be able to miss a few  points.

> One way to do that is to do the VSFT and then evaluate the 
> resulting weighted sum of  trig functions in the bad region.  Unlike DFT,
> VSFT defines an inverse transform over a continuous range in abscissa.  You 
> can even extrapolate.  When you use Artemis
> in q-space mode and it plots q over a bigger range than your input data, I 
> suspect that it's doing such a summation.

> BTW, I think that argument about detecting high audio frequencies is more 
> subtle than whether you can hear them directly.
> I think the argument is that their presence somehow affects the perception of 
> the lower frequencies.  I'd have to hear
> an AB comparison of a piece of music with and without the 18kHz and above 
> frequencies before I'll believe such a claim.

Sure....   For audio, the detectors are complicated.  ;).

Cheers,

--Matt