Hi Scott,
Is this the most recent IXAS report on error reporting standards?
http://www.i-x-s.org/OLD/subcommittee_reports/sc/err-rep.pdf
Yes. To be clear, the main value of reduced chi-square is that it can be used (even if with some inherent uncertainty) to compare two models with different number of variables. Many analysis programs report only a value like R-factor (ie, the misfit not scaled by the measurement uncertainty or number of free parameters in the data). Again, this is an OK measure of the misfit, though it too is scaled somewhat arbitrarily, and cannot be used to compare models with different number of variables.
... In such cases I have become convinced that the R-factor alone provides as much meaningful information as the chi-square values, and that in fact the chi-square values can be confusing when listed for fits on different data. For those working with dilute samples, on the other hand, I can see that chi-square might be a meaningful quantity.
... I strongly agree that the decision of which measurements of quality of fit to produce should not be dependent on what "looks good"! That would be bad science. The decision of what figures of merit to present should be made a priori.
The subcommittee that looked into agreed (after some debate) on wording and recommendations of such topics also thought it should be done a priori, though they also thought is should be done without regard to quality of the data or type of samples. You're free to disagree with this report. --Matt