What confidence limit is implied in the error bars for least-squares fitting in Athena
Hi everyone, I'm fitting data with the least-squares method in Athena and am not sure what the error bars correspond to. What I mean, are these 95% confidence limits, or is it some other statistical way for error analysis? If Athena says the weight of one phase is 0.049 (0.003), what confidence are in the 0.003 error bar reported by Athena? (I have done a PCA of the XANES, and XRDs of the samples and am pretty confident in the phases that I'm fitting). I greatly appreciate the help! Andrew Campos
On Thursday 02 October 2008 16:49:27 Andrew wrote:
I'm fitting data with the least-squares method in Athena and am not sure what the error bars correspond to. What I mean, are these 95% confidence limits, or is it some other statistical way for error analysis? If Athena says the weight of one phase is 0.049 (0.003), what confidence are in the 0.003 error bar reported by Athena?
They are 1-sigma error bars, with the caveat that they assume that the only source of noise is statistical noise. Since an XAS experiment is (almost) never dominated by statistical noise, it's generally a very conservative (in the sense that it almost certainly understates the confidence) 1-sigma. The dominant source of error in a typical XAS experiment is often something like sample inhomogeneity. I am not really certain how to quantify that sort of thing, so I don't actually know how to report a better error bar. B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973 My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
Hi Bruce,
On Thu, Oct 2, 2008 at 4:12 PM, Bruce Ravel
On Thursday 02 October 2008 16:49:27 Andrew wrote:
I'm fitting data with the least-squares method in Athena and am not sure what the error bars correspond to. What I mean, are these 95% confidence limits, or is it some other statistical way for error analysis? If Athena says the weight of one phase is 0.049 (0.003), what confidence are in the 0.003 error bar reported by Athena?
They are 1-sigma error bars, with the caveat that they assume that the only source of noise is statistical noise. Since an XAS experiment is (almost) never dominated by statistical noise, it's generally a very conservative (in the sense that it almost certainly understates the confidence) 1-sigma.
Does it? I thought it used data_uncertainty=1 unless the uncertainty was explicitly specified. Does Athena use a value determined from the sigma when merging data, or is something? Should I read the Users Guide more closely? --Matt
On Thursday 02 October 2008 18:06:19 Matt Newville wrote:
Does it? I thought it used data_uncertainty=1 unless the uncertainty was explicitly specified. Does Athena use a value determined from the sigma when merging data, or is something? Should I read the Users Guide more closely?
Hmmmm... perhaps I am confused. Athena uses Ifeffit's minimize function. As you say, the data uncertainty is set to one, so the diagonals of the covarience matrix will be orders of magnitude too small. My understanding is that Ifeffit rescales the error bars on the variable parameters in the same manner as the feffit function does -- that is, by the square root of reduced chi-square. In that case, the error bars are reported are 1-sigma error bars given the assumption that the fit is, in fact, a good fit. Or do I not understand what minimize is doing? Or are there too many possible meanings of the word "sigma" in this context such that we are talking about different things? B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973 My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
Athena uses Ifeffit's minimize function. As you say, the data uncertainty is set to one, so the diagonals of the covarience matrix will be orders of magnitude too small. My understanding is that Ifeffit rescales the error bars on the variable parameters in the same manner as the feffit function does -- that is, by the square root of reduced chi-square.
In that case, the error bars are reported are 1-sigma error bars given the assumption that the fit is, in fact, a good fit.
Or do I not understand what minimize is doing? Or are there too many possible meanings of the word "sigma" in this context such that we are talking about different things?
Yes, that is all correct (and the reference manual is no doubt confusing!!) It asserts the fit is good (and that all the data points are independent), and so asserts the uncertainty in the data is the typical misfit of data-model. And with that estimate, it reports 1-sigma uncertainties in the parameters. But that's not using an actual estimate of sigma from the data (unless explicitly provided). So the value of chi-square is not scaled correctly. --Matt
Hi Andrew, If your issue is whether or not a phase with a amplitude of 0.049(3) is actually present in your sample, then you want to use an F-test. The probability of of the F-distribution is the probability that the improvement in the fit (due in this case to adding a component with an amplitude of 0.049) could be due to random error. The general rule of thumb is that if the probability is less than 0.05, than improvement in the fit is significant, and the component can be considered to be observed. Unlike the standard deviations of the fitting parameters, the F-test does not depend on having an accurate description of the uncertainty of the data. Sincerely, Wayne Matt Newville wrote:
Athena uses Ifeffit's minimize function. As you say, the data uncertainty is set to one, so the diagonals of the covarience matrix will be orders of magnitude too small. My understanding is that Ifeffit rescales the error bars on the variable parameters in the same manner as the feffit function does -- that is, by the square root of reduced chi-square.
In that case, the error bars are reported are 1-sigma error bars given the assumption that the fit is, in fact, a good fit.
Or do I not understand what minimize is doing? Or are there too many possible meanings of the word "sigma" in this context such that we are talking about different things?
Yes, that is all correct (and the reference manual is no doubt confusing!!) It asserts the fit is good (and that all the data points are independent), and so asserts the uncertainty in the data is the typical misfit of data-model. And with that estimate, it reports 1-sigma uncertainties in the parameters.
But that's not using an actual estimate of sigma from the data (unless explicitly provided). So the value of chi-square is not scaled correctly.
--Matt _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
On Thursday 02 October 2008 18:06:19 Matt Newville wrote:
Does it? I thought it used data_uncertainty=1 unless the uncertainty was explicitly specified. Does Athena use a value determined from the sigma when merging data, or is something? Should I read the Users Guide more closely?
Chapter 8 of the Ifeffit Reference Manual is a bit ambiguous about what is reported as an error bar by minimize, but it does lead one to believe that the error bars have been rescaled by the suqare root of reduced chi-square -- like the feffit function. B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973 My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
Hi Andrew, If you are fitting XANES spectra with standards, a large source of uncertainty can come from the representation of the standards in matching the species in the sample. Shelly ________________________________ From: ifeffit-bounces@millenia.cars.aps.anl.gov [mailto:ifeffit-bounces@millenia.cars.aps.anl.gov] On Behalf Of Andrew Sent: Thursday, October 02, 2008 3:49 PM To: ifeffit@millenia.cars.aps.anl.gov Subject: [Ifeffit] What confidence limit is implied in the error bars forleast-squares fitting in Athena Hi everyone, I'm fitting data with the least-squares method in Athena and am not sure what the error bars correspond to. What I mean, are these 95% confidence limits, or is it some other statistical way for error analysis? If Athena says the weight of one phase is 0.049 (0.003), what confidence are in the 0.003 error bar reported by Athena? (I have done a PCA of the XANES, and XRDs of the samples and am pretty confident in the phases that I'm fitting). I greatly appreciate the help! Andrew Campos
participants (5)
-
Andrew -
Bruce Ravel -
Kelly, Shelly -
Matt Newville -
Wayne Lukens