This is great to hear, Matt! While I'm going to be too busy to give it a real look in the next few months, after that I'll be very interested. Writing it in Python will make it possible for me to compile the code myself, which I never succeeded in doing with the Fortran-based programs on the Mac. That means I may be able to help add new features and the like. --Scott Calvin Sarah Lawrence College On Jun 27, 2012, at 6:36 PM, Matt Newville wrote:
Hi Folks,
I wanted to give an update on progress of my Ifeffit 2 project, which I've been developing under the code name Larch, a name which I think might stick. This has been a long, slow (and sometimes nearly stopped) process over several years, but a few events have conspired so that I've focused enough attention on it over the past month or so that enough progress has actually been made to report on it. One of the conspiring events is the upcoming XAFS conference, where I will present several features of Larch.
First, a summary:
Larch is a complete rewrite of the functionality of Ifeffit. It has a much richer and flexible syntax, and much better data processing functionality. Though currently focused on XAFS analysis, the intention is to add functionality for related techniques such as XRF. Like Ifeffit, Larch works either as a command-line with a scriptable interface, or as a library, allowing GUIs to be built upon this foundation. Larch is written entirely in Python, using a large set of existing and supported scientific computing libraries.
By throwing out all of Ifeffit1 and Fortran and re-implementing in Python, a number of recurring problems we've add with Ifeffit 1 go away. The most import of these is that there is no built-in limits on memory usage. A secondary issue is that the macro language exposed to the user is much, much better -- you can write real programs in Larch (it is a dialect of Python itself) and have much richer data structures. A third part of the enhancement is that development and extending functionality is simpler, and the code is much more modular. This not only means that maintenance is easier for me, but that many existing libraries can be immediately used. It also means that others should have a much easier time playing with the code, which I encourage.
Though it may be easy to see that replacing Fortran for Python would give code that is more flexible, easier to read and maintain, it might be counter-intuitive to expect it to be more efficient. But it turns out that for basic data processing (not including graphics), Larch is often considerably faster than Ifeffit 1. A typical "read data, do background subtractions, and FFTs" is about 2.5x faster in my tests. That should make processing large amounts of QEXAFS and hyper-spectral data noticeably faster. As an aside, the graphics are slower, but are of much better quality.
Larch is still in active development, but here's a short list of what is currently working:
Basic functionality: 1. basic mathematical, array manipulation. 2. reading / writing data from ASCII files. 3. reading data from HDF5. 4. line plotting of x, y data, with much better graphics than Ifeffit, with interactive zooming, plot configuration, etc. 5. simple 2d image display of 2d array data, I(x,y). 6. a complete python-like language for writing complex scripts. 7. an easy-to-use interface to general-purpose fitting of data, for example to Gaussian lineshapes. 8. running either as a command-line program or as a Python library, or in 'server mode', so that another process may interact with a background server process. 9. There is a start of a proof-of-principle general purpose GUI, but it's not very far along.
XAFS Functionality: 1. Pre-edge subtraction, normalization. Currently done as in Ifeffit, and ready to be expanded. 2. Autobk background subtraction. A few missing features here. 3. XAFS Fourier transforms (forward and reverse) with a variety of window types. 4. Feffit fits to sums of Feff Paths from feff.dat files. Support for multiple-k-weight fits and multiple-data-set fits are included. 5. Uncertainties calculations are automatically included and match very closely to those from Feffit/Ifeffit despite being a new implementation. 6. Standard reference data for several x-ray values can be retrieved from tables of Elam etal (absorption coefficients, x-ray edges and lines), Chantler etal (f', f''), Waasmaier and Kirfel (f0), and Keski-Rahkonen and Krause (core-hole widths).
Where possible, the syntax of the XAFS functionality is largely unchanged. Some of the functionality listed above is an incomplete port of Ifeffit's functionality. For example, the autobk() function does not yet support spline clamps, and may have an error when and there is not yet an equivalent of the Cromer-Libermann Background, ala MBACK. Some functionality is much better, of course.
In essence, all of Ifeffit has been ported and is approximately ready for testing for the brave. The build and installation process is not highly streamlined yet -- a windows executable does not yet exist. But code has been actively developed on Windows, Mac OS X, and Linux, and installs and runs without modification (there are some built in Windows-specific code but the user never needs to think about this). The documentation is fairly scant, especially for the EXAFS functions. This will improve over the next months.
Much of the current status is expected to be the starting point for further expansion, so this is the right time to think about and propose real enhancements. For example, including better linear algebra / PCA methods would be near the top of list of things to add. My own needs include XRF Analysis, data collection, and I fully expect Larch to be able to all of these things.
Since most people here are mostly using the Athena/Artemis/Dathena/Dartemis GUIs I'll point out that the 'server mode' mentioned above will allow Demeter to communicate with Larch, and even allow Demeter to use 2 or more instances of background Larch processes running at the same time (say, one per core of your machine). That is not yet all worked out, but we know how to do it.
Code and Documentation so far are at http://github.com/xraypy/xraylarch and http://xraypy.github.com/xraylarch
At this point, I'm looking for a couple types of feedback: 1. comments and suggestions on the process and design so far. 2. suggestions for enhancements or features. This is the time to send your wish list! 3. collaborators on implementing these features or adding new functionality.
--Matt _______