On Fri, Jul 13, 2012 at 7:54 AM, Bruce Ravel <bravel@bnl.gov> wrote:
> On Thursday, July 12, 2012 05:00:00 PM Matt Newville wrote:
>> > I would challenge a reviewer to come up with a reliable study
>> > demonstrating that Feff8 or Feff9 provides a statistically significant
>> > improvement to EXAFS analysis. Obviously, I am not saying that full
>> > multiple scattering, self-consistency, and the various potential
>> > improvements in Feff9 are not relevant to XANES calculations or to
>> > Feff's treatment of other spectroscopies. What I am saying is that it
>> > has not been rigorously demonstrated that those things have an impact
>> > on the calculation of the extended EXAFS such that the analysis is
>> > improved in a statistically meaningful way.
>> >
>> > In short, I find that such arguments from reviewers boil down to
>> > "Feff8 is 33% better than Feff6 because 8 is 33% bigger than 6!"
>>
>> I would say that there are some demonstrated cases where Feff8 is
>> slightly better than Feff6 at modeling EXAFS. The most notable cases
>> are when H is in the input file -- Feff6 is terrible at this.
>>
>> Feff8 has gone through many revisions, so minor version might matter
>> here. Some of the earliest versions actually gave worse fits (in
>> terms of goodness-of-fit statistics) to EXAFS than Feff6. Versions
>> around 8.2.4 (if memory serves) ended up with approximately the same
>> goodness-of-fits as Feff6. And, although few cases have been tested,
>> and the process is hard to generalize, if multi-pole loss terms are
>> used in Feff8.5 or later, the improvement in goodness of fit is
>> actually quite noticeable.
>>
>> Whether these differences have any impact of the accuracy of fitted
>> values is harder to determine, of course.
>
>
> Alrighty, fair enough. I admit to being a little vague about what the
> different minor versions of Feff8 do. And I certainly agree that the
> multi-pole loss terms represent a substantial improvement for the
> interpretation of EXAFS.
>
> That said, your last sentence is the gist of my comment.
>
> To my knowledge (and I freely admit that my knowledge may be limited
> and inaccurate), no one has done a rigorous statistical assessment of
> how EXAFS analysis is affected by the various bits of theory that are
> in Feff8 and are not in Feff6.
>
> It's a shame that hasn't been done. It seems like rather low-hanging
> fruit. Regardless of the outcome, such a study would certainly have
> an impact on how I develop and promote my software.

John and I have run statistical tests of fits using various feff versions in the past.  I still have the test suite, and I believe they've run the suite several times on various versions of Feff8.  Getting definitive tests is not as easy as it might appear, as you'd really want perfect data on simple, well-known structures (like, I would not expect Mn2O3 or even MnO or FeO to be good candidates).   I believe that we used some Cu metal data you and I measured long ago, and some Ag and Au metal foil data.  I have been made skeptical of the quality of the Ag data used.   In any event, the tests were all on dense metal systems, which is a very skewed test, and not highly relevant for many analyses.

Like you say, while Feff6 is not perfect, demonstrating a robust improvement is challenging.    And I agree that the issue is worth re-visiting.   What we would need is a representative test suite of very good data on simple structures, probably with XRD data.

--Matt