RE: [Ifeffit] testing ravelware
Hi Matt,
Hi Shelly,
Bruce says: P.S. Once again, on the topic of how people can contribute to the ifeffit project without writing code, volunteering to conceive of a battery of tests and then actually running them would be an excellent contribution. Any takers?
I would think that someone would want to write a perl script to execute a bunch of Athena/Artimis commands and then check the results. Then have the script write out a report. (Kind of like what tkperl does when you do install the software?) Can Athena/Airtimes commands be executed from a script? If someone was willing to figure out how to do this for one or two of the commands, I would be willing to extend it to the rest.
Testing ifeffit commands with the Ifeffit perl module is definitely possible. That may miss some artemis/athena capabilities and potential problems, but would still be useful. You could certainly emulate much of athena's functionality into perl scripts to do background subtraction, FFTs, and merging of data with a wide range of parameters and data sets.
A script like the simple one below might be pretty good place to start. (This could be done in languages other than perl and English). There is also a test.pl with the ifeffit source perl module (I think it's called t/ifeffit.t in horae), which mostly tests passing of data between perl and Ifeffit. Since the script below essentially runs an ifeffit script, there are several test/example scripts in the examples/ folder that could be used as the basis of other tests.
The main difficulty is deciding when the analysis or calculated values are "right", which makes detecting problems hard to automate.
If anyone wants more pointers on how to use the ifeffit from scripts (perl, python, tcl, ...), please let us know.
--Matt
#! /usr/bin/perl use Ifeffit; use Ifeffit qw(get_scalar);
my $xmu_file = "a.xmu"; my $expected_e0 = 8980.0; my $toler = 0.5;
print "Test Finding E0 for $xmu_file: ";
my $script =<<"END"; read_data(file = $xmu_file, prefix = my, type = xmu) pre_edge(my.energy, my.xmu) END
$ret = ifeffit($script); # should test $ret for errors. $e0 = get_scalar("e0"); print " found $e0, which is ";
if ( abs($e0 - $expected_e0) le $toler) { print "ok\n"; } else { print "outside of expected range\n"; }
##---- end test_e0.pl
This is the kind of script is what I had in mind. As you mentioned this script checks ifeffit directly. I would like to extend it so that Athena is checked. Does anyone know how to do that? This makes me think that if one was cleaver about the way that the test is written that it could be done to check both Athena and/or Ifeffit. That would make the tests more general diagnostic tools. Shelly
Hi, I wanted to make some comments on the discussion from last week mostly between Matt and Shelly in response to my request for testing volunteers. Matt's suggestion were mostly about *regression testing* of the ifeffit library. I was hoping to inspire volunteers to do *use testing* of Athena and Artemis. Both are needed. Regression testing of ifeffit is undoubtedly important and would be a splendid way for a volunteer to contribute something of genuine value to the ifeffit project. One could imagine creating some sort of database using standards data. Then every time a new version of ifeffit is released, the standards data can be checked against this data base. One series of regression tests might be to Fourier transform a standard chi(k) data file using a sequence of different parameters (i.e. different k-ranges, different weights, phase correction, different window types, and so on -- essentially every option listed in the Ifeffit Reference Manual for the fftf() function could be poked during the regression testing). The data base would contain the trusted results [1] of each of the permutations of the basic Fourier transform operation. The regression tests would then use some kind of R-factor-like metric to determine that the Fourier transforms under each set of conditions are sufficiently close to the trusted results that the Fourier transforming operation can be said to still be working in the new release of ifeffit. This kind of thing could be implemented in C or perl or python or whatever, whatever makes the implementer happy. This kind of automated, low-level testing is very much necessary. Although Matt has done an extraordinary job [2] of providing a stable, high-quality analysis engine, there have been examples in recent months of errors being introduced in a new release. In some cases, these mistakes would not have survived good regression testing. Thus, the contribution of a good set of regression tests would help the core ifeffit library immensely. The other day, when I asked for test volunteers, I was asking for something a little different from regression testing. The problem with a GUI is that it encompasses not only algorithms but a user implementation as well. Testing the underlying algorithms is necessary but inadequate. It is quite possible for a GUI to have properly-functioning algorithms but to have serious problems between the on-screen user experience and those properly-functioning algorithms. Thus a GUI must be used by a human [3] to be tested. My understanding of how companies do that is to have a crew of folks who use the program repeatedly and report any indication that something is misbehaving. My concept of how my codes might be tested would be for a number of people to volunteer for some aspect of the program. For example, one person might volunteer to read files into Athena and write them out in every form that is supported. Then, plot the output data independently of Athena to verify that writing them out worked as expected. Another person might volunteer to put the alignment dialog through its paces. And so on. Not only is that kind of user-level interaction difficult to automate, automating it might fail to turn up some problem in the user interface [4]. If enough people volunteered to check out some aspect of the code each time a new release comes out, then no one would actually have to do that much work [5]. Indeed, a network of use testing volunteers could be incorporated into the release process such that a version does not get released until the problems the testers uncover get fixed. Soooooo... regression testing and use testing... any takers? Regards, B Footnotes: [1] The sense in which I mean "trusted" is a similar sense as in cryptogrophy or security research. That is, the trusted results are the ones that we presume to be true or have convinced ourselves are true by some form of scrutiny outside the scope of the regression tests. If we allow a faulty operation to slip into the "trusted results", then future regression tests will trun up false positives when they preserve that faulty result. However, with enough care and scrutiny, a set of trusted results can be made that are, indeed, trustworthy. [2] Really, he has! Has everyone remembered to thank Matt again recently for all he has done? [3] This is essentially the same reason that faculty have grad students rather than well-trained bonobos. ;-) [4] And it certainly would for Athena and Artemis. I did not consider the possibility of automating the on-screen behavior when I first designed the programs. There really isn't a good way to automate screen events within my codes in the manner of, say, writing VB scripts for Microsoft Office applications. And retooling the codes to allow for that would be a huge job that would take me away from implementing new features for weeks or months for relatively little benefit. [5] Except me, of course! -- Bruce Ravel ----------------------------------- ravel@phys.washington.edu Code 6134, Building 3, Room 222 Naval Research Laboratory phone: (1) 202 767 5947 Washington DC 20375, USA fax: (1) 202 767 1697 NRL Synchrotron Radiation Consortium (NRL-SRC) Beamlines X11a, X11b, X23b National Synchrotron Light Source Brookhaven National Laboratory, Upton, NY 11973 My homepage: http://feff.phys.washington.edu/~ravel EXAFS software: http://feff.phys.washington.edu/~ravel/software/exafs/
Bruce, I believe one can programmatically fill in form data and send virtual events to a Tk-based GUI, which is essentially pushing all the buttons, and may have been what Shelly was getting at. The callback routines for "the real calculations" can definitely be run programmatically with various input data (and wasn't the original post about phase-corrections???). This might be difficult or easy for athena/artemis, depending on how they're implemented: Perl's anonymous closures can only be tested by sending virtual events. But if it's worth the time to formally test these programs, it might be worth some effort to make them more easily testable, no? To be honest, I'm not sure how many bugs regression tests for Ifeffit would have found in the past year: most of the problems have been in new features, obscure corners that may have been missed even by "thorough tests", or platform-specific bugs. There were several problems dealing with data files with unexpected formats, which is hard to predict, and hard to say that "failure" of a test necessarily means the program should be changed. Still, I think any such testing (by primates of any sort or by machine, of any GUI or of underlying libraries) is a fine thing to do. But without the "trusted results" you mentioned, I don't see a huge difference between using and testing. And I'd guess that deciding on these trusted results is 90% of the work. --Matt
On Monday 10 November 2003 03:09 pm, Matt Newville wrote:
I believe one can programmatically fill in form data and send virtual events to a Tk-based GUI, which is essentially pushing all the buttons, and may have been what Shelly was getting at. The callback routines for "the real calculations" can definitely be run programmatically with various input data (and wasn't the original post about phase-corrections???). This might be difficult or easy for athena/artemis, depending on how they're implemented: Perl's anonymous closures can only be tested by sending virtual events. But if it's worth the time to formally test these programs, it might be worth some effort to make them more easily testable, no?
Well, from where I'm sitting, that falls in the catagory of "easier said than done". I can imagine how to tool my own code to allow for regression testing via generation of virtual events. I have a little more trouble thinking of a robust way to implement regression tests via virtual events involving arbitrary widgets from the widget library (such as the file dialog). I have no idea at all how to do a regression test on, say, deglitching, which involves faithfully reproducing a virtual event in the pgplot window. Since a lot of people seem to consider Athena and Artemis important parts of their workday (a fact that, frankly, blows my mind -- just look at the yahoo that wrote them!), it seems reasonable to ask that a few of them spend a half hour every so often doing manual tests, reporting problems, and making suggestions. So far, three volunteers seem to agree with that. Of course, if the concensus is that tools for automated regression testing are more important than new analysis features, then I can be convinced to spend my time implementing them. B -- Bruce Ravel ----------------------------------- ravel@phys.washington.edu Code 6134, Building 3, Room 222 Naval Research Laboratory phone: (1) 202 767 5947 Washington DC 20375, USA fax: (1) 202 767 1697 NRL Synchrotron Radiation Consortium (NRL-SRC) Beamlines X11a, X11b, X23b National Synchrotron Light Source Brookhaven National Laboratory, Upton, NY 11973 My homepage: http://feff.phys.washington.edu/~ravel EXAFS software: http://feff.phys.washington.edu/~ravel/software/exafs/
Hi all, I recently installed the ifeffit package on my new Linux system. When I installed the python wrapper (including gifeffit) I didn't notice any error message, but trying to start gifeffit results in the following output: --------------- Traceback (most recent call last): File "/usr/local/ifeffit/bin/gifeffit", line 3, in ? GIFeffit.GIFeffit(sys.argv[1:]) File "/usr/lib/python2.3/site-packages/Ifeffit/GIFeffit.py", line 287, in __init__ self.createMenubar(self.root) File "/usr/lib/python2.3/site-packages/Ifeffit/GIFeffit.py", line 393, in createMenubar command=Command(self.read_cmndfile, self.root)) File "/usr/lib/python2.3/site-packages/Pmw/Pmw_1_1/lib/PmwMenuBar.py", line 155, in addmenuitem self._addHotkeyToOptions(menuName, kw, 'label', traverseSpec) File "/usr/lib/python2.3/site-packages/Pmw/Pmw_1_1/lib/PmwMenuBar.py", line 201, in _addHotkeyToOptions underline = string.atoi( File "/usr/lib/python2.3/string.py", line 220, in atoi return _int(s, base) TypeError: int() can't convert non-string with explicit base ----------- (and gifeffit is not starting, of course). Does anyone of you have an idea what is going wrong? (I'm not a linux expert. Maybe this is a silly beginners mistake!) Thanks, Peter -- -------------------------------------------------------------- Peter Pfalzer Universitaet Augsburg Tel: +49-821-598-3215 Lehrstuhl fuer Experimentalphysik II Fax: +49-821-598-3411 Universitaetsstr. 1 D-86135 Augsburg Germany Peter.Pfalzer@physik.uni-augsburg.de --------------------------------------------------------------
Hi Peter, I have similar problem on MacOS X before. What I did to bypass it is not to use explicit base in atoi() in string.py as the error message said: def atoi(s, *args): return _int(s, *args)
File "/usr/lib/python2.3/site-packages/Pmw/Pmw_1_1/lib/PmwMenuBar.py", line 155, in addmenuitem self._addHotkeyToOptions(menuName, kw, 'label', traverseSpec) File "/usr/lib/python2.3/site-packages/Pmw/Pmw_1_1/lib/PmwMenuBar.py", line 201, in _addHotkeyToOptions underline = string.atoi( File "/usr/lib/python2.3/string.py", line 220, in atoi return _int(s, base) TypeError: int() can't convert non-string with explicit base
Tsu-Chien <Chien>001000</Chien>
Hi Peter, Nope, this is not a silly beginner's mistakes. And as Tsu-Chien points out, it looks like a real problem between python2.3 and Pmw1.1 when converting strings to integers. I have not tried Pmw with python2.3 on linux or OS X myself, but will investigate. --Matt On Wed, 12 Nov 2003, Peter Pfalzer wrote:
Hi all,
I recently installed the ifeffit package on my new Linux system. When I installed the python wrapper (including gifeffit) I didn't notice any error message, but trying to start gifeffit results in the following output: --------------- Traceback (most recent call last): File "/usr/local/ifeffit/bin/gifeffit", line 3, in ? GIFeffit.GIFeffit(sys.argv[1:]) File "/usr/lib/python2.3/site-packages/Ifeffit/GIFeffit.py", line 287, in __init__ self.createMenubar(self.root) File "/usr/lib/python2.3/site-packages/Ifeffit/GIFeffit.py", line 393, in createMenubar command=Command(self.read_cmndfile, self.root)) File "/usr/lib/python2.3/site-packages/Pmw/Pmw_1_1/lib/PmwMenuBar.py", line 155, in addmenuitem self._addHotkeyToOptions(menuName, kw, 'label', traverseSpec) File "/usr/lib/python2.3/site-packages/Pmw/Pmw_1_1/lib/PmwMenuBar.py", line 201, in _addHotkeyToOptions underline = string.atoi( File "/usr/lib/python2.3/string.py", line 220, in atoi return _int(s, base) TypeError: int() can't convert non-string with explicit base ----------- (and gifeffit is not starting, of course). Does anyone of you have an idea what is going wrong? (I'm not a linux expert. Maybe this is a silly beginners mistake!)
Thanks, Peter
Hi Peter, Tsu-Chien, Sorry to all for following-up my own post so quickly: If you're up for it, there is a Pmw version 1.2 that claims to work with python2.3. This would be the first thing I try after getting python2.3 and Pmw installed (and looks likely to work, as this _had_ to be noticed before). You might be able to do this faster than me. Installing Pmw usually just involves copying the source Pmw_1_X tree to PYTHON_DIR/site-packages/Pmw/ (with PYTHON_DIR probably being /usr/lib/python2.3 on linux, /sw/lib/python2.3 with OS X 10.2/Fink, and probably something like Library/Python on OS X 10.3). --Matt
Hi Matt, Tsu-Chien,
If you're up for it, there is a Pmw version 1.2 that claims to work with python2.3. This would be the first thing I try after getting python2.3 and Pmw installed (and looks likely to work, as this _had_ to be noticed before). You might be able to do this faster than me.
Installing Pmw usually just involves copying the source Pmw_1_X tree to PYTHON_DIR/site-packages/Pmw/ (with PYTHON_DIR probably being /usr/lib/python2.3 on linux, /sw/lib/python2.3 with OS X 10.2/Fink, and probably something like Library/Python on OS X 10.3).
thanks for the quick help! I installed Pmw 1.2 and gifeffit does okay now. Peter -- -------------------------------------------------------------- Peter Pfalzer Universitaet Augsburg Tel: +49-821-598-3215 Lehrstuhl fuer Experimentalphysik II Fax: +49-821-598-3411 Universitaetsstr. 1 D-86135 Augsburg Germany Peter.Pfalzer@physik.uni-augsburg.de --------------------------------------------------------------
Hi Peter, On Wed, 12 Nov 2003, Peter Pfalzer wrote:
I installed Pmw 1.2 and gifeffit does okay now.
Excellent! Thanks very much for checking this out. I'll be sure to put Pmw 1.2 in the next source release, and use it for the binaries for Mac OS 10.3 (when I get 10.3, that is!). --Matt
participants (5)
-
Bruce Ravel -
Kelly, Shelly D. -
Matt Newville -
Peter Pfalzer -
Tsu-Chien Weng