[Ifeffit] Larch / Ifeffit2 development status

[Scott Calvin]

This is great to hear, Matt! While I'm going to be too busy to give it a real look in the next few months, after that I'll be very interested. Writing it in Python will make it possible for me to compile the code myself, which I never succeeded in doing with the Fortran-based programs on the Mac. That means I may be able to help add new features and the like.

--Scott Calvin
Sarah Lawrence College

On Jun 27, 2012, at 6:36 PM, Matt Newville wrote:

> Hi Folks,
> 
> I wanted to give an update on progress of my Ifeffit 2 project, which
> I've been developing under the code name Larch, a name which I think
> might stick.  This has been a long, slow (and sometimes nearly
> stopped) process over several years, but a few events have conspired
> so that I've focused enough attention on it over the past month or so
> that enough progress has actually been made to report on it.  One of
> the conspiring events is the upcoming XAFS conference, where I will
> present several features of Larch.
> 
> First, a summary:
> 
> Larch is a complete rewrite of the functionality of Ifeffit.
> It has a much richer and flexible syntax, and much better
> data processing functionality.  Though currently focused on
> XAFS analysis, the intention is to add functionality for
> related techniques such as XRF.  Like Ifeffit, Larch works
> either as a command-line with a scriptable interface, or as
> a library, allowing GUIs to be built upon this foundation.
> Larch is written entirely in Python, using a large set of
> existing and supported scientific computing libraries.
> 
> By throwing out all of Ifeffit1 and Fortran and re-implementing in
> Python, a number of recurring problems we've add with Ifeffit 1 go
> away.  The most import of these is that there is no built-in limits on
> memory usage.  A secondary issue is that the macro language exposed to
> the user is much, much better -- you can write real programs in Larch
> (it is a dialect of Python itself) and have much richer data
> structures.  A third part of the enhancement is that development and
> extending functionality is simpler, and the code is much more modular.
> This not only means that maintenance is easier for me, but that many
> existing libraries can be immediately used.  It also means that others
> should have a much easier time playing with the code, which I
> encourage.
> 
> Though it may be easy to see that replacing Fortran for Python would
> give code that is more flexible, easier to read and maintain, it might
> be counter-intuitive to expect it to be more efficient.  But it turns
> out that for basic data processing (not including graphics), Larch is
> often considerably faster than Ifeffit 1.  A typical "read data, do
> background subtractions, and FFTs" is about 2.5x faster in my tests.
> That should make processing large amounts of QEXAFS and hyper-spectral
> data noticeably faster.   As an aside, the graphics are slower, but
> are of much better quality.
> 
> Larch is still in active development, but here's a short list of what
> is currently working:
> 
> Basic functionality:
>   1. basic mathematical, array manipulation.
>   2. reading / writing data from ASCII files.
>   3. reading data from HDF5.
>   4. line plotting of x, y data, with much better graphics than
> Ifeffit, with interactive zooming, plot configuration, etc.
>   5. simple 2d image display of 2d array data, I(x,y).
>   6. a complete python-like language for writing complex scripts.
>   7. an easy-to-use interface to general-purpose fitting of data, for
> example to Gaussian lineshapes.
>   8. running either as a command-line program or as a Python library,
> or in 'server mode', so that another process may interact with a
> background server process.
>   9. There is a start of a proof-of-principle general purpose GUI,
> but it's not very far along.
> 
> XAFS Functionality:
>   1. Pre-edge subtraction, normalization.  Currently done as in
> Ifeffit, and ready to be expanded.
>   2. Autobk background subtraction. A few missing features here.
>   3. XAFS Fourier transforms (forward and reverse) with a variety of
> window types.
>   4. Feffit fits to sums of Feff Paths from feff.dat files.  Support
> for multiple-k-weight fits and multiple-data-set fits are included.
>   5. Uncertainties calculations are automatically included and match
> very closely to those from Feffit/Ifeffit despite being a new
> implementation.
>   6. Standard reference data for several x-ray values can be
> retrieved from tables of Elam etal (absorption coefficients, x-ray
> edges and lines), Chantler etal (f', f''), Waasmaier and Kirfel (f0),
> and Keski-Rahkonen and Krause (core-hole widths).
> 
> Where possible, the syntax of the XAFS functionality is largely
> unchanged.   Some of the functionality listed above is an incomplete
> port of Ifeffit's functionality.  For example, the autobk() function
> does not yet support spline clamps, and may have an error when and
> there is not yet an equivalent of the Cromer-Libermann Background, ala
> MBACK.   Some functionality is much better, of course.
> 
> In essence, all of Ifeffit has been ported and is approximately ready
> for testing for the brave.  The build and installation process is not
> highly streamlined yet -- a windows executable does not yet exist.
> But code has been actively developed on Windows, Mac OS X, and Linux,
> and installs and runs without modification (there are some built in
> Windows-specific code but the user never needs to think about this).
> The documentation is fairly scant, especially for the EXAFS functions.
> This will improve over the next months.
> 
> Much of the current status is expected to be the starting point for
> further expansion, so this is the right time to think about and
> propose real enhancements.  For example, including better linear
> algebra / PCA methods would be near the top of list of things to add.
> My own needs include XRF Analysis, data collection, and I fully
> expect Larch to be able to all of these things.
> 
> Since most people here are mostly using the
> Athena/Artemis/Dathena/Dartemis GUIs I'll point out that the 'server
> mode' mentioned above will allow Demeter to communicate with Larch,
> and even allow Demeter to use 2 or more instances of background Larch
> processes running at the same time (say, one per core of your
> machine).  That is not yet all worked out, but we know how to do it.
> 
> Code and Documentation so far are at
> http://github.com/xraypy/xraylarch and
> http://xraypy.github.com/xraylarch
> 
> At this point, I'm looking for a couple types of feedback:
>  1. comments and suggestions on the process and design so far.
>  2. suggestions for enhancements or features.  This is the time to
> send your wish list!
>  3. collaborators on implementing these features or adding new functionality.
> 
> --Matt
> _______