On Oct 25, 2010, at 8:25 AM, Wayne W Lukens Jr wrote:
A more useful way to look at this is that the probabilities that A, B and C are present are 99.99999999%, 93%, and 77%, respectively.
An excellent post, Wayne, but I don't think that last statement is quite right. If the F-test gives a probability of 0.23 for material C, I believe it's saying that there is a 23% that, given the noise level in the data, the fit would indicate that C was present when it was not. That is not the same thing as saying there is a 77% chance of C being present.
To see this, imagine very, very noisy data. Including C in the fit might very well improve the fit in the sense of an R-factor--maybe, in fact, there's a 45% chance of a modest improvement with a given set of very noisy data, even if there's no C present. That does not mean that a result like that should lead to the conclusion that C is more likely than not present (55%).
--Scott Calvin
Sarah Lawrence College