Hello all, I would like to hear your opinion about reporting Chi^2 and R-factor values in EXAFS studies. I found that Chi^2 and R-factor values are not provided by many EXAFS papers. However, according to standards of The International XAFS Society (http://ixs.iit.edu/) both these values should be reported. My understanding is that the reduced Chi^2 should be close to 1 in ideal case. In practice however, the value is always greater than 1 because of systematic errors. For example, my reduced Chi^2 values are around 300. However, the R-factor values are about 3% which doesn't look as "ugly" as Chi^2 reduced. On the other hand, ifeffit overestimates uncertainties using reduced Chi^2 value. Thus, all errors are taken into account. The final question: Is it appropriate to report just R-factor calculated by ifeffit and the obtained uncertainties (which look reasonable) without reporting the "ugly" Chi^2/reduced Chi^2 values? Stanislav Stanislav Stoupin ----------------------------------- Research Assistant/PhD student BCPS department Illinois Institute of Technology email: stousta@iit.edu voice: 312-567-7983
Hi Stanislav, I'm glad you brought this up--I'm very interested in hearing a debate on this issue. Here's my two cents: in principle, I think reporting a reduced chi^2 is a good idea. The problem, of course, is that a meaningful reduced chi-square requires a meaningful estimate of the measurement error epsilon. The default behavior of Ifeffit is to use the noise in the high-R fourier transform (I believe it uses 15-25 Angstroms) for epsilon. Ifeffit also allows you to override this value if you think you have a better estimate. In some cases, the default ifeffit estimate for epsilon is probably fairly poor. I have used a beamline, for example, which often showed high-R oscillations (due, perhaps, to feedback problems). These high-R oscillations had little or no effect on the EXAFS, but, by increasing Ifeffit's estimate of epsilon, resulted in a lower reduced chi^2 than if I had recorded the data on another beamline! Because of this, I ignore published reduced chi^2's that don't include an argument as to why the epsilon they chose (or let Ifeffit choose) is a reasonable value. I would generally prefer such reduced chi^2's not be included in the article. Having said that, I'd love a discussion as to how people obtain meaningful estimates of epsilon, or under what conditions they consider the Ifeffit default good enough. As you noted, it is worth reiterating that the algorithm Ifeffit uses to find uncertainties in fitted parameters is independent of the choice of epsilon. --Scott Calvin Sarah Larence College
Hello all,
I would like to hear your opinion about reporting Chi^2 and R-factor values in EXAFS studies.
I found that Chi^2 and R-factor values are not provided by many EXAFS papers. However, according to standards of The International XAFS Society (http://ixs.iit.edu/) both these values should be reported.
My understanding is that the reduced Chi^2 should be close to 1 in ideal case. In practice however, the value is always greater than 1 because of systematic errors. For example, my reduced Chi^2 values are around 300. However, the R-factor values are about 3% which doesn't look as "ugly" as Chi^2 reduced.
On the other hand, ifeffit overestimates uncertainties using reduced Chi^2 value. Thus, all errors are taken into account. The final question: Is it appropriate to report just R-factor calculated by ifeffit and the obtained uncertainties (which look reasonable) without reporting the "ugly" Chi^2/reduced Chi^2 values?
Stanislav
On Monday 12 July 2004 03:25 pm, Scott Calvin wrote:
Having said that, I'd love a discussion as to how people obtain meaningful estimates of epsilon, or under what conditions they consider the Ifeffit default good enough.
As you noted, it is worth reiterating that the algorithm Ifeffit uses to find uncertainties in fitted parameters is independent of the choice of epsilon.
I would emphasize that last point. The error bars on parameters are always reasonable because they are computed by scaling the diagonal elements of the covarience matrix by the square root of chi-square. That is *the same thing as* reeveluating the covarience matrix with the value of epsilon that results in a reduced chi square of 1. Thus, if you believe that the fit is a good fit, the error bars are the right size. As far as the actual value of epsilon -- I am not sure it is really all that important. There is plenty of information available to your to evaluate a fit. The R-factor is a start, as are the error bars and ones physical intuition about the system. The use of reduced chi-square is to compare two different models. If you change a fitting model in some way, you expect the reduced chi-square to be sufficiently smaller for the better fit. If reduced chi-square doesn't change enough, then the two fitting models are statistically indistinguishable. Another way of saying that is that the R factor is a useful metric for deciding if THIS fit is reasonable. The reduced chi-square is a useful metric for deciding if THIS fit is better than THAT fit. As to whether you need to report reduced chi-square in a paper, I agree with Matt. If it's an important part of the story, it's an important part of the paper. But if you include it in the paper, you had better be prepared to explain the previous paragraph and to explain why it is so hard to estimate epsilon. Or at least to refer to a paper which explains those things to your satisfaction. B -- Bruce Ravel ----------------------------------- ravel@phys.washington.edu Code 6134, Building 3, Room 405 Naval Research Laboratory phone: (1) 202 767 2268 Washington DC 20375, USA fax: (1) 202 767 4642 NRL Synchrotron Radiation Consortium (NRL-SRC) Beamlines X11a, X11b, X23b National Synchrotron Light Source Brookhaven National Laboratory, Upton, NY 11973 My homepage: http://feff.phys.washington.edu/~ravel EXAFS software: http://feff.phys.washington.edu/~ravel/software/exafs/
Hi Stan,
I found that Chi^2 and R-factor values are not provided by many EXAFS papers. However, according to standards of The International XAFS Society (http://ixs.iit.edu/) both these values should be reported.
Yep, many papers do not repot chi-square and r-factor. I've not always reported these myself.
My understanding is that the reduced Chi^2 should be close to 1 in ideal case. In practice however, the value is always greater than 1 because of systematic errors.
This is the convential wisdom. It's probably even right. We almost always try to measure until statistical errors are not significant.
For example, my reduced Chi^2 values are around 300. However, the R-factor values are about 3% which doesn't look as "ugly" as Chi^2 reduced. On the other hand, ifeffit overestimates uncertainties using reduced Chi^2 value. Thus, all errors are taken into account.
Well, I wouldn't say 'overestimates'. It does the equivalent of adjusting the measurement uncertainty until reducded-chi-square=1. Then it does the normal estimate of parameter uncertainties of increasing (non-reduced) chi-square by 1.
The final question: Is it appropriate to report just R-factor calculated by ifeffit and the obtained uncertainties (which look reasonable) without reporting the "ugly" Chi^2/reduced Chi^2 values?
Speaking for myself (not the IXS!!): I would say this can be appropriate. I'd also say that it can be appropriate to present _qualitative_ EXAFS analysis ("spectra A does not look at all like spectra B or C") in a paper, and not report any statistics at all -- the IXS recommendation completely ignores this, and seems to say should not happen. I want to see chi-square and/or r-factor in a paper if it's really part of the story... that is, if you're comparing two models or the quality of the fit is actually in question (generally, the gentle reader takes it on faith that you're presenting the best fit and the most reasonable model you could find). If you're reporting fits to a reasonably clear model and the point is something along the lines of "see, in this one sample, the near neighbor distance is different by 0.08Ang", I probably wouldn't care too much about r-factor and chi-square, unless the parameter uncertainties were unusual (say, +/- 0.07Ang). Then again, in a case like that, I'd probably even be able that something was different by looking at k*chi(k). Basically, if the story NEEDS the detailed statistical parameters, the paper needs them. If the story doesn't need the detailed statistical parameters, the paper is probably fine without them. Depending on the journal and the story, it may be appropriate to put such details (chi-square, epsilon, fit ranges, etc) into the 'supplemental material'. --Matt
participants (4)
-
Bruce Ravel -
Matt Newville -
Scott Calvin -
Stanislav Stoupin