[Ifeffit] Larch / Ifeffit2 development status

[Matt Newville]

Hi Folks,

I wanted to give an update on progress of my Ifeffit 2 project, which
I've been developing under the code name Larch, a name which I think
might stick.  This has been a long, slow (and sometimes nearly
stopped) process over several years, but a few events have conspired
so that I've focused enough attention on it over the past month or so
that enough progress has actually been made to report on it.  One of
the conspiring events is the upcoming XAFS conference, where I will
present several features of Larch.

First, a summary:

 Larch is a complete rewrite of the functionality of Ifeffit.
 It has a much richer and flexible syntax, and much better
 data processing functionality.  Though currently focused on
 XAFS analysis, the intention is to add functionality for
 related techniques such as XRF.  Like Ifeffit, Larch works
 either as a command-line with a scriptable interface, or as
 a library, allowing GUIs to be built upon this foundation.
 Larch is written entirely in Python, using a large set of
 existing and supported scientific computing libraries.

By throwing out all of Ifeffit1 and Fortran and re-implementing in
Python, a number of recurring problems we've add with Ifeffit 1 go
away.  The most import of these is that there is no built-in limits on
memory usage.  A secondary issue is that the macro language exposed to
the user is much, much better -- you can write real programs in Larch
(it is a dialect of Python itself) and have much richer data
structures.  A third part of the enhancement is that development and
extending functionality is simpler, and the code is much more modular.
 This not only means that maintenance is easier for me, but that many
existing libraries can be immediately used.  It also means that others
should have a much easier time playing with the code, which I
encourage.

Though it may be easy to see that replacing Fortran for Python would
give code that is more flexible, easier to read and maintain, it might
be counter-intuitive to expect it to be more efficient.  But it turns
out that for basic data processing (not including graphics), Larch is
often considerably faster than Ifeffit 1.  A typical "read data, do
background subtractions, and FFTs" is about 2.5x faster in my tests.
That should make processing large amounts of QEXAFS and hyper-spectral
data noticeably faster.   As an aside, the graphics are slower, but
are of much better quality.

Larch is still in active development, but here's a short list of what
is currently working:

Basic functionality:
   1. basic mathematical, array manipulation.
   2. reading / writing data from ASCII files.
   3. reading data from HDF5.
   4. line plotting of x, y data, with much better graphics than
Ifeffit, with interactive zooming, plot configuration, etc.
   5. simple 2d image display of 2d array data, I(x,y).
   6. a complete python-like language for writing complex scripts.
   7. an easy-to-use interface to general-purpose fitting of data, for
example to Gaussian lineshapes.
   8. running either as a command-line program or as a Python library,
or in 'server mode', so that another process may interact with a
background server process.
   9. There is a start of a proof-of-principle general purpose GUI,
but it's not very far along.

XAFS Functionality:
   1. Pre-edge subtraction, normalization.  Currently done as in
Ifeffit, and ready to be expanded.
   2. Autobk background subtraction. A few missing features here.
   3. XAFS Fourier transforms (forward and reverse) with a variety of
window types.
   4. Feffit fits to sums of Feff Paths from feff.dat files.  Support
for multiple-k-weight fits and multiple-data-set fits are included.
   5. Uncertainties calculations are automatically included and match
very closely to those from Feffit/Ifeffit despite being a new
implementation.
   6. Standard reference data for several x-ray values can be
retrieved from tables of Elam etal (absorption coefficients, x-ray
edges and lines), Chantler etal (f', f''), Waasmaier and Kirfel (f0),
and Keski-Rahkonen and Krause (core-hole widths).

Where possible, the syntax of the XAFS functionality is largely
unchanged.   Some of the functionality listed above is an incomplete
port of Ifeffit's functionality.  For example, the autobk() function
does not yet support spline clamps, and may have an error when and
there is not yet an equivalent of the Cromer-Libermann Background, ala
MBACK.   Some functionality is much better, of course.

In essence, all of Ifeffit has been ported and is approximately ready
for testing for the brave.  The build and installation process is not
highly streamlined yet -- a windows executable does not yet exist.
But code has been actively developed on Windows, Mac OS X, and Linux,
and installs and runs without modification (there are some built in
Windows-specific code but the user never needs to think about this).
The documentation is fairly scant, especially for the EXAFS functions.
 This will improve over the next months.

Much of the current status is expected to be the starting point for
further expansion, so this is the right time to think about and
propose real enhancements.  For example, including better linear
algebra / PCA methods would be near the top of list of things to add.
 My own needs include XRF Analysis, data collection, and I fully
expect Larch to be able to all of these things.

Since most people here are mostly using the
Athena/Artemis/Dathena/Dartemis GUIs I'll point out that the 'server
mode' mentioned above will allow Demeter to communicate with Larch,
and even allow Demeter to use 2 or more instances of background Larch
processes running at the same time (say, one per core of your
machine).  That is not yet all worked out, but we know how to do it.

Code and Documentation so far are at
http://github.com/xraypy/xraylarch and
http://xraypy.github.com/xraylarch

At this point, I'm looking for a couple types of feedback:
  1. comments and suggestions on the process and design so far.
  2. suggestions for enhancements or features.  This is the time to
send your wish list!
  3. collaborators on implementing these features or adding new functionality.

--Matt