Hi Scott, here I copy a part of the report: Independent points = 6.222656250 Number of variables = 4.000000000 Chi-square = 247.145092496 Reduced Chi-square = 111.193574128 R-factor = 0.017422216 Guess parameters +/- uncertainties (initial guess): amp = 6.7815290 +/- 1.4687660 (1.0000) enot = 2.2173620 +/- 2.1499920 (0.0000) delr = 0.0514640 +/- 0.0163900 (0.0000) ss = 0.0074020 +/- 0.0025220 (0.0030) I see that the R-factor is pretty good, 1.74%, amp is high cause is correlated with the coordination number and always have big errors., delr has the error of the total distance, so it is ok, but ss and enot have a a really big error, is this normal? Thanks, euG
Hi euG, That amp is not physically reasonable, unless you're using it as a proxy for the coordination number. The uncertainty on the other variables does not seem high to me for a single-shell fit. Well, 2 eV is a bit high for an uncertainty on E0, but not crazy high. There are some approaches that can be used to try to reduce the uncertainties, but you shouldn't even think about that until you get the amp (S02) straightened out. --Scott Calvin Sarah Lawrence College On Aug 18, 2009, at 6:18 PM, Eugenio Otal wrote:
Hi Scott, here I copy a part of the report:
Independent points = 6.222656250 Number of variables = 4.000000000 Chi-square = 247.145092496 Reduced Chi-square = 111.193574128 R-factor = 0.017422216
Guess parameters +/- uncertainties (initial guess): amp = 6.7815290 +/- 1.4687660 (1.0000) enot = 2.2173620 +/- 2.1499920 (0.0000) delr = 0.0514640 +/- 0.0163900 (0.0000) ss = 0.0074020 +/- 0.0025220 (0.0030)
I see that the R-factor is pretty good, 1.74%, amp is high cause is correlated with the coordination number and always have big errors., delr has the error of the total distance, so it is ok, but ss and enot have a a really big error, is this normal? Thanks, euG
Dear Eugenio,
I see that the R-factor is pretty good, 1.74%, amp is high cause is correlated with the coordination number and always have big errors., delr has the error of the total distance, so it is ok, but ss and enot have a a really big error, is this normal?
What leads you to conclude that the uncertainty in delr (of ~0.016
Ang) is OK, but the uncertainty in ss (~0.0025 Ang^2) and enot (~2 eV)
are really big? I don't know how to interpret the value for "amp" or
assess why it would have a "big error" (is that a big error? how is
"amp" used in your model?).
Generally, for "decent data" and a good fit, R has uncertainty of 0.01
or 0.02 Ang, E0 has uncertainty of 0.5 to 1.0 eV, and the amplitude
factors (N and sigma^2) are good to 10% or so. Without knowing more
details, I'd say that your results fit within the "normal" range.
Having a "reasonable R-factor" of a few percent misfit and a reduced
chi-square of ~100 means the misfit is much larger than the estimated
uncertainty in the data. This is not at all unusual. It does not
necessarily mean (as Scott implies) that this is because the
uncertainty in data is unreasonably low, but can also mean that there
are systematic problems with the FEFF calculations that do not account
for the data as accurately as it can be measured. For most "real"
data, it is likely that both errors FEFF and a slightly low estimate
for the uncertainty in the data contribute to making reduced
chi-square much larger than 1.
And, yes, the community-endorsed recommendation is to report either
chi-square or reduced chi-square as well as an R-factor. I think some
referees might find it a little deceptive to report R-factor because
it is "acceptably small" but not reduced chi-square because it is "too
big".
--Matt
On Tue, Aug 18, 2009 at 5:18 PM, Eugenio Otal
Hi Scott, here I copy a part of the report:
Independent points = 6.222656250 Number of variables = 4.000000000 Chi-square = 247.145092496 Reduced Chi-square = 111.193574128 R-factor = 0.017422216
Guess parameters +/- uncertainties (initial guess): amp = 6.7815290 +/- 1.4687660 (1.0000) enot = 2.2173620 +/- 2.1499920 (0.0000) delr = 0.0514640 +/- 0.0163900 (0.0000) ss = 0.0074020 +/- 0.0025220 (0.0030)
I see that the R-factor is pretty good, 1.74%, amp is high cause is correlated with the coordination number and always have big errors., delr has the error of the total distance, so it is ok, but ss and enot have a a really big error, is this normal? Thanks, euG
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Matt, Is this the most recent IXAS report on error reporting standards? http://www.i-x-s.org/OLD/subcommittee_reports/sc/err-rep.pdf It uses a rather expansive definition of epsilon, which explicitly includes "imperfect" ab initio standards such as FEFF calculations. It indicates that statistical methods such as that used by ifeffit for estimating measurement error yields a lower limit for epsilon, and thus an overestimate of chi square. So I think my statement and yours are entirely compatible. As far as what should be reported, I do deviate from the IXAS recommendations by not reporting chi-square. Of course, I tend to work in circumstances where the signal-to-noise ratio is very high, and thus the statistical uncertainties make a very small contribution to the overall measurement error. In such cases I have become convinced that the R-factor alone provides as much meaningful information as the chi-square values, and that in fact the chi-square values can be confusing when listed for fits on different data. For those working with dilute samples, on the other hand, I can see that chi-square might be a meaningful quantity. At any rate, I strongly agree that the decision of which measurements of quality of fit to produce should not be dependent on what "looks good"! That would be bad science. The decision of what figures of merit to present should be made a priori. --Scott Calvin Sarah Lawrence College On Aug 18, 2009, at 10:40 PM, Matt Newville wrote:
Having a "reasonable R-factor" of a few percent misfit and a reduced chi-square of ~100 means the misfit is much larger than the estimated uncertainty in the data. This is not at all unusual. It does not necessarily mean (as Scott implies) that this is because the uncertainty in data is unreasonably low, but can also mean that there are systematic problems with the FEFF calculations that do not account for the data as accurately as it can be measured. For most "real" data, it is likely that both errors FEFF and a slightly low estimate for the uncertainty in the data contribute to making reduced chi-square much larger than 1.
And, yes, the community-endorsed recommendation is to report either chi-square or reduced chi-square as well as an R-factor. I think some referees might find it a little deceptive to report R-factor because it is "acceptably small" but not reduced chi-square because it is "too big".
OK, I feel I have to weigh in. I'm on a microprobe line where sample motions contribute to noise and I rarely find that the noise
quality of
the EXAFS signal, as measured by running a high-order polynomial through the data and looking at the residuals, matches the number
of counts per point.
Also, if you are using an analog counter like an ion chamber, then you can't measure the true number of detected quanta, so you
can't get the shot-noise
limit. Further, there will be systematics like background-subtraction artifacts which will act as other than white noise. For all
these reasons, I think
that an attempt to use a literal chi-squared isn't going to succeed. I don't think I've ever seen anyone report the true noise
quality of their data, anyway.
Occasionally, someone might report the number of counts/point, but as I said, that's an upper limit to the noise quality. What is
more intuitive, though
less rigorous to report, is the R value.
mam
----- Original Message -----
From: "Scott Calvin"
Matt,
Is this the most recent IXAS report on error reporting standards?
http://www.i-x-s.org/OLD/subcommittee_reports/sc/err-rep.pdf
It uses a rather expansive definition of epsilon, which explicitly includes "imperfect" ab initio standards such as FEFF calculations. It indicates that statistical methods such as that used by ifeffit for estimating measurement error yields a lower limit for epsilon, and thus an overestimate of chi square.
So I think my statement and yours are entirely compatible.
As far as what should be reported, I do deviate from the IXAS recommendations by not reporting chi-square. Of course, I tend to work in circumstances where the signal-to-noise ratio is very high, and thus the statistical uncertainties make a very small contribution to the overall measurement error. In such cases I have become convinced that the R-factor alone provides as much meaningful information as the chi-square values, and that in fact the chi-square values can be confusing when listed for fits on different data. For those working with dilute samples, on the other hand, I can see that chi-square might be a meaningful quantity.
At any rate, I strongly agree that the decision of which measurements of quality of fit to produce should not be dependent on what "looks good"! That would be bad science. The decision of what figures of merit to present should be made a priori.
--Scott Calvin Sarah Lawrence College
On Aug 18, 2009, at 10:40 PM, Matt Newville wrote:
Having a "reasonable R-factor" of a few percent misfit and a reduced chi-square of ~100 means the misfit is much larger than the estimated uncertainty in the data. This is not at all unusual. It does not necessarily mean (as Scott implies) that this is because the uncertainty in data is unreasonably low, but can also mean that there are systematic problems with the FEFF calculations that do not account for the data as accurately as it can be measured. For most "real" data, it is likely that both errors FEFF and a slightly low estimate for the uncertainty in the data contribute to making reduced chi-square much larger than 1.
And, yes, the community-endorsed recommendation is to report either chi-square or reduced chi-square as well as an R-factor. I think some referees might find it a little deceptive to report R-factor because it is "acceptably small" but not reduced chi-square because it is "too big".
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Hi Scott,
Is this the most recent IXAS report on error reporting standards?
http://www.i-x-s.org/OLD/subcommittee_reports/sc/err-rep.pdf
Yes. To be clear, the main value of reduced chi-square is that it can be used (even if with some inherent uncertainty) to compare two models with different number of variables. Many analysis programs report only a value like R-factor (ie, the misfit not scaled by the measurement uncertainty or number of free parameters in the data). Again, this is an OK measure of the misfit, though it too is scaled somewhat arbitrarily, and cannot be used to compare models with different number of variables.
... In such cases I have become convinced that the R-factor alone provides as much meaningful information as the chi-square values, and that in fact the chi-square values can be confusing when listed for fits on different data. For those working with dilute samples, on the other hand, I can see that chi-square might be a meaningful quantity.
... I strongly agree that the decision of which measurements of quality of fit to produce should not be dependent on what "looks good"! That would be bad science. The decision of what figures of merit to present should be made a priori.
The subcommittee that looked into agreed (after some debate) on wording and recommendations of such topics also thought it should be done a priori, though they also thought is should be done without regard to quality of the data or type of samples. You're free to disagree with this report. --Matt
participants (4)
-
Eugenio Otal -
Matt Newville -
Matthew -
Scott Calvin