Re: [Ifeffit] How to do with diffraction peak

Hi Matthew, On Fri, 14 Dec 2012, Matthew Marcus wrote:

On 12/14/2012 4:10 PM, Matt Newville wrote:

...

Hi Matthew,

On Thu, Dec 13, 2012 at 11:28 PM, Matthew Marcus wrote:

Oh, I meant that for each energy point measured you could use the different fluorescence channels to compute a mean value for I_f and standard deviation for I_f, and then propagate both measurement and uncertainty through the entire analysis. For a point with a glitch, the standard deviation would be very high, and so that region of k space would receive less weight.

I'd view data not measured not as having a value of 0 but having infinite uncertainty. In that sense, removing a glitch doesn't do any harm - we're treating it as if we have no measurement at that energy point.

If I understand you correctly, you're saying that 'deglitch with reference scalers' is OK, but with some additional error estimate imposed on the points you had to do that to.

No, I'm saying first that it is OK to simply remove a glitch -- you didn't measure mu(E) at that energy, you measured something else. Alternatively, one could keep the glitch but say that the uncertainty at that energy is extremely large. This approach would require one to propagate the uncertainties per point throughout the analysis -- not a bad idea, but not normally done. Your method of replacing an outlier in a set of I_f values with the mean value is also fine with me -- it effectively removes the bad point, preserving the average I_f. Alternatively, one could use not the simple sum or mean of the set of I_f values, but use both the mean and variance, and propagate these both all the way through the analysis. The advantage of propagating the mean and variance is that one may not correctly identifiy small glitches or tails of glitches or other artifacts not due to counting statistics, but their effect would be handled as well as possible.

I've found no reason to suspect that any artifacts were introduced by this procedure because the ratio of the "bad" to the "good" counters is quite slowly-varying except in the glitch region, so interpolation is safe. Therefore, the only extra noise is shot noise due to counting with fewer elements. Now, given how non-rigorous our treatment of noise is, I think that the extra noise is not going to influence the fit significantly.

The real problem is what to do when you can't do any kind of signal-based data recovery, but have to either acknowledge that you don't have any data there or make some up and assign some sort of uncertainty to your curve-drawing artistry.

Right. I think acknowledging you don't have the data is the best approach.

...

...

...

I agree with your larger point that we can definitely afford the CPU cycles to do better analysis than use something simply because it is Fast. I'm not sure that a slow discrete FT would be significantly different than an FFT, though. I think (but am listening...) that the continuous v discrete FT is like classic audiophile snobbery about the superiority of analog over digital music. With sufficient sampling and resolution and dynamic range in the signal, the difference has to be trivial. I'm willing to believe that a digital recording has less dynamic range, but I'm also willing to believe that this is swamped by my stereo and ears (and I mostly listen to mp3s through cheap headphones these days, so I clearly am not buying a tube amplifier anytime soon...).

You're right that the DFT is exactly equivalent to the FFT. However, the VSFT is different in that it works on non-uniformly tabulated data with no interpolation. I've heard of an AB test in which Golden Ears were challenged with two amps, one of which was solid-state and the other also solid-state but with some 2nd-harmonic distortion and (I think) noise added to simulate a tube amp. Guess which won? Somebody once wrote a parody in which mercury-filled speaker cables were advertised for their "liquid, shimmering sound", but I digress.

I agree / understand that a VSFT is or can be different from a DFT. For me the point is that with sufficient sampling and bit depth, the differences go to zero. And I think we know how to sample XAFS well enough. For digital audio there's a fair argument about bit depth / dynamic range and whether people can actually detect frequencies about 18KHz. But there aren't such arguments (yet for EXAFS): no one is (yet) measuring EXAFS reliably to 50 Ang^-1.

Sampling is exactly the problem - if you have gaps in the data, then you're not sampled densely in the relevant region.

Yes, sampling is the problem. But, removing a point or two from a 0.05Ang^-1 grid won't do much damage to the signal below 8 Ang. You could even mask out a full 0.5 Ang^-1 from the chi(k) and still do OK. This is essentially the earlier suggestion for using a notched window around a glitch or region of glitches -- it is possible to ignore portions of an oscillatory function and still recover phase/amplitude. Sure, it's not ideal, but it's actually not that bad for the lower frequencies.

If you assume the data to be band-limited, then you could in principle interpolate.

The data is limited. We're not going to be modeling EXAFS beyond 50 Ang^-1 or beyond 20 Ang anytime soon. Possibly ever. Samplying at 0.05Ang^-1 would let us measure to 31 Ang (well, aliasing might push that down, but only if you believe non-zero signal past 31 Ang!). That is, we over-sample the EXAFS below 8 Ang well enough to be able to miss a few points.

One way to do that is to do the VSFT and then evaluate the resulting weighted sum of trig functions in the bad region. Unlike DFT, VSFT defines an inverse transform over a continuous range in abscissa. You can even extrapolate. When you use Artemis in q-space mode and it plots q over a bigger range than your input data, I suspect that it's doing such a summation.

BTW, I think that argument about detecting high audio frequencies is more subtle than whether you can hear them directly. I think the argument is that their presence somehow affects the perception of the lower frequencies. I'd have to hear an AB comparison of a piece of music with and without the 18kHz and above frequencies before I'll believe such a claim.

Sure.... For audio, the detectors are complicated. ;). Cheers, --Matt