[Ifeffit] R-factor uncertainty

[Scott Calvin]

Hi Lisa,



>At 07:23 AM 1/5/2007, you wrote:
>
>I have a question about the R factor: how can I decide if the 
>difference between the R factors of 2 fits is statistically 
>significant, i.e, how can I calculate the uncertainty which has to 
>be associated to the R factor?
>

As I understand it, you can't. The R-factor is not a proper 
statistical measure, as it doesn't incorporate any measure of data 
quality. That's the great weakness of this measure of quality-of-fit. 
It is also its strength, as estimating the uncertainty in EXAFS data 
is notoriously problematic.

The complementary statistic is reduced chi-square. It does 
incorporate a measure of data quality. By default, ifeffit uses noise 
from high in the FT to estimate this. That's a reasonable idea, but 
can be problematic. It has been shown (by Matt and/or Shelly, as I 
recall), that there may in some cases be signal in the part of the FT 
ifeffit is using to estimate noise. There are also cases where the 
noise may not be "white," that is, the noise high in the FT may be a 
poor estimate of the noise low in the FT. Ifeffit does allow you to 
specify a value of the measurement uncertainty instead, so if you 
think you have a way of doing this, go ahead.

What does all this mean in practice? It means, in my opinion, that 
the actual =value= of the reduced chi-square statistic is usually 
meaningless, unless you have a good way of coming up with the 
measurement uncertainty (for example, your sample may be so dilute 
that errors are dominated by counting statistics). But reduced 
chi-square is a great statistic for comparing two fits to a given set 
of data, particularly if the k-range, k-weighting, and k-window are 
the same for the two fits. For example, you can apply statistical 
tests of significance, if you'd like. The R-factor then provides the 
reality check that the fit is "good" at all. The R-factor isn't doing 
anything other than what you can see by looking at a graph, but is a 
nice shorthand for tables showing the results of many fits and 
similar applications. If there's a big R-factor (say, 0.20), the 
question of statistical significance isn't necessary to tell you that 
you haven't got a conclusive positive result: maybe the R-factor is 
big because the fitting model is lousy, or maybe it's big because the 
data quality is lousy, but either way the fit shouldn't be trusted.

I'd also add that your eye tells you considerably more than the 
R-factor, because you can tell the character of the mismatch. Is it 
in the high part of the FT, low, or evenly throughout? Is the miss 
primarily in amplitude, or phase? I often find I choose a fit with an 
R-factor of 0.03 over one with 0.01, if, for example, the 0.03 
reproduces qualitatively all the features in the data but has small 
errors in the amplitude of the peaks, while the 0.01 fits the first 
part of the spectrum perfectly but misses some peak altogether.

Hope that helps...

--Scott Calvin
Sarah Lawrence College