[Ifeffit] fitting a specific k range

Fri Dec 6 08:24:25 CST 2013

Hi Matt,

Yes--we clearly disagree. 

The word "complaining" has come up a couple of times. I didn't think of it that way, although clearly it's coming off that way. I said it was my least favorite warning, and that usually one of the first things I tell students to do is to change the preferences to turn it off. That isn't a complaint--it's a description of what I do. I think descriptions of the actual behavior of users, whatever their level of expertise, is useful to developers and community. If that's counterproductive, wouldn't it also be counterproductive to say "I never use the button for corrected Fourier transform, because I think it encourages confusion" or "I don't see what the benefit is to fitting in k or q space, and so I don't do it?" 

> Having a warning about this is vastly preferable to not having such a warning.

To be clear, my view is that false positives can indeed be a significant problem--the medical community, for example, knows this very well. I've repeatedly witnessed people starting out with Artemis who fall in to a pattern of ignoring all warnings because of this particular false positive.

It seems to me that a user has to know a bit more to generate the true positive in this case than the false one. They have to add free parameters for paths outside the fitting range--Artemis doesn't do that for them. And then they get another strong sign that something has gone wrong because, as you say, the error bars blow up. So, unlike the consequences that can come from treating the false positive as a true one (removing a path that actually contributes significantly to the fitting range), in the case of a true positive it will at least be clear something is wrong. The message helps diagnose the problem, but the lack of the message wouldn't cause someone to think everything is hunky-dory when it's not (unless they don't even understand the concept and importance of error bars) .

So yes, it's closer to my view that the rule is always wrong and not worth fixing, although of course that's a bit oversimplified. For certain kinds of fitting strategies, it can be useful, and it's good that the software offers it as an option. But I don't think it should be the default option, as I think it confuses or misleads novices more often than it helps them. So I would suggest that the out of the box behavior for the warning to be off.

That's not an attack on the software in general, or on the warning feature it includes. It is my opinion, and it seems to me that this mailing list is an appropriate place to express that kind of opinion.

--Scott Calvin
Sarah Lawrence College

On Dec 6, 2013, at 12:29 AM, Matt Newville <newville at cars.uchicago.edu> wrote:

> Hi Scott,
> 
> On Thu, Dec 5, 2013 at 4:56 PM, Scott Calvin <scalvin at sarahlawrence.edu> wrote:
>> I love the warnings Artemis gives! They're not just for novices--they often catch when I've made a dumb mistake somewhere. I praise them, defend them, and generally think Bruce has done a wonderful thing by having them.
>> 
>> The out of range default warning, however, I find counterproductive and confusing to novices. There are two reasons for my opinion (and it is, of course, my opinion--as Bruce points out, if others differ, they can set the defaults differently.) One has to do with the kind of example I mentioned earlier. Here's the other reason:
>> 
>> The default behavior is to warn if there is a path beyond 1.1 times the top of the range, correct? (It's kind of a pain for me right now to fire up the most recent version of Artemis, so I can't actually easily confirm that at the moment.) The default R-max is, if I recall correctly, 3.0 angstroms. Thus, by default, a warning is generated if there is a path above 3.3 angstroms in the list.
>> 
>> But, as we know, the path list uses distances which are half path lengths, while the Fourier transform range is in terms of the conjugate variable to k. For edges around the third row of the periodic table, the peaks corresponding to a path tend to show up about 0.3 to 0.5 angstroms lower in the Fourier transform than their half path-length. And, of course, that's just the peak--the path has significant amplitude a bit below (and above) that.
>> 
>> So, a novice user fires up Artemis, imports her data, and uses atoms to generate a feff file. Because she's appropriately thoughtful about what she's doing, she looks at what the unfitted paths of the FEFF calculation look like. She sees the fitting range goes up to 3 angstroms, and then selects all the paths that contribute significant amplitude to that range. That might include a path with a half path length of 3.4 angstroms. She then runs a fit--and Artemis gives her a warning that something may be wrong.
>> 
>> At that point, she could stick to her guns and tell the fit to continue. Since that's going to happen with pretty much every fit she runs, it becomes very tempting not to read the warning each time, but just dismiss it. And at that point, if there's a highly useful warning, she'll miss it.
>> 
>> Or, she could decide that she's the novice, and what she's doing isn't that unusual, so maybe she shouldn't be including that path at 3.4 angstroms, and take it out. She is now getting distorted results, because she's leaving out a path that has significant amplitude in the region.
> 
> It's reasonable to question whether the threshold for such a warning
> is too conservative or to suggest (better yet, issue a pull request
> for) a fix for a particular feature.  Saying (as you did) that turning
> the warning off "usually one of the first things" you teach students
> seems counterproductive.   To be clear, you did not teach them to
> change the default value of reff_margin. I would have said Mend it,
> don't End it.   Perhaps the rule should be (Reff+0.5)*1.1?  But this
> was not your suggestion.  Your suggestion was to ignore the rule.
> 
> If you are relying on these codes for your work, and teaching others
> to use them, and find things you don't like or could be improved, it
> is your responsibility to provide constructive, specific feedback and
> fixes to the actual code.  Yes, I actually do mean you, Scott.   This
> would be the perfect place for you to contribute.  See
>     https://github.com/bruceravel/demeter/blob/master/lib/Demeter/Fit/Sanity.pm
>     https://github.com/bruceravel/demeter/blob/master/lib/Demeter/configuration/warnings.demeter_conf
> 
>> I think, not just from personal preference, but also from a consideration of what is best for people learning to use
>> the software, that the warning is set too conservatively. And I'm not even clear what misstep it's trying to prevent.
> 
> I cannot tell if you are joking.   It is, of course, to prevent one
> from including paths (say, the second and third shells with variable
> parameters when the R range being fit (say the first shell) will not
> allow those variables to alter the fit.  Having such variables can
> grossly distort the fit and will generally prevent good error bars
> from being calculated.  I agree that it's not so much the presence of
> Paths well beyond Rmax as it is the presence of variables that only
> affect Paths outside the fitting range.  But that is much harder to
> automatically detect, and the case described above is a likely way to
> get into that situation.  Having a warning about this is vastly
> preferable to not having such a warning.
> 
> The details of the thresholds may be tweaked, but you didn't complain
> about the details of the rules, but the existence of the rule itself.
> 
> Perhaps your view is that this rule is just always wrong and not worth
> fixing, but I think many of us would disagree with you.
> 
> --Matt
> 
> _______________________________________________
> Ifeffit mailing list
> Ifeffit at millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit