[Ifeffit] Chi^2 and R-factor reporting requirements

[Matt Newville]

Hi Stan,

> I found that Chi^2 and R-factor values are not provided by many
> EXAFS papers. However, according to standards of The
> International XAFS Society (http://ixs.iit.edu/) both these
> values should be reported.

Yep, many papers do not repot chi-square and r-factor.
I've not always reported these myself.

> My understanding is that the reduced Chi^2 should be close to 1 in ideal
> case. In practice however, the value is always greater than 1 because of
> systematic errors.
 
This is the convential wisdom. It's probably even right.  We
almost always try to measure until statistical errors are not
significant. 

> For example, my reduced Chi^2 values are around 300. However,
> the R-factor values are about 3% which doesn't look as "ugly" as
> Chi^2 reduced. On the other hand, ifeffit overestimates
> uncertainties using reduced Chi^2 value. Thus, all errors are
> taken into account.

Well, I wouldn't say 'overestimates'.  It does the equivalent of
adjusting the measurement uncertainty until reducded-chi-square=1.  
Then it does the normal estimate of parameter uncertainties of
increasing (non-reduced) chi-square by 1.

> The final question: Is it appropriate to report just R-factor calculated
> by ifeffit and the obtained uncertainties (which look reasonable) without
> reporting the "ugly" Chi^2/reduced Chi^2 values?

Speaking for myself (not the IXS!!):

I would say this can be appropriate.  I'd also say that it can be
appropriate to present _qualitative_ EXAFS analysis ("spectra A
does not look at all like spectra B or C") in a paper, and not
report any statistics at all -- the IXS recommendation completely
ignores this, and seems to say should not happen.

I want to see chi-square and/or r-factor in a paper if it's really
part of the story... that is, if you're comparing two models or
the quality of the fit is actually in question (generally, the
gentle reader takes it on faith that you're presenting the best
fit and the most reasonable model you could find).

If you're reporting fits to a reasonably clear model and the point
is something along the lines of "see, in this one sample, the near
neighbor distance is different by 0.08Ang", I probably wouldn't
care too much about r-factor and chi-square, unless the parameter
uncertainties were unusual (say, +/- 0.07Ang).  Then again, in a
case like that, I'd probably even be able that something was
different by looking at k*chi(k).

Basically, if the story NEEDS the detailed statistical parameters,
the paper needs them.  If the story doesn't need the detailed
statistical parameters, the paper is probably fine without them.

Depending on the journal and the story, it may be appropriate to
put such details (chi-square, epsilon, fit ranges, etc) into the
'supplemental material'.

--Matt