[Ifeffit] Ifeffit Digest, Vol 109, Issue 21

[Matt Newville]

Hi Qingyu,

On Fri, Mar 30, 2012 at 1:32 PM, KONG Quingyu
<kong at synchrotron-soleil.fr> wrote:
> Thank you for all your helpful suggestions.
>
> Actually, our beamline has recently installed a fast detector with a
> repetation of 20KHz, which could collected data in a time resolution
> of few tens of microsecond and could easily produce a large amount
> of data. We have already accepted three users for microsecond time
> resolved studies, each of them collected millions of spectra.
>
> Together with the company that provide the detector and our software
> engineer we started to write a Python program to treat the data from
> November last year, however our engineer quitted recently, this make
> us to look for other solutions.

Bruce wrote earlier,

   As a closing comment, isn't it somehow strange that you and your
   colleagues went to all the trouble of building a fancy dispersive
   beamline without thinking ahead about how you were going to analyze
   your data?

Sadly, Bruce's observation is not unusual at all.  Everyone is
experiencing data overloads, and almost no one is investing in
computing resources.  Yours may be slightly larger than most XAFS
users, but only by one or two orders of magnitude.  Then again, you
had a software engineer at your beamline at least for a while.

To be clear, Ifeffit, Athena, Demeter, Sixpack, and so on are written
primarily by beamline scientists like yourself.  Your choices are to
use what someone else gives you, write your own, or hire someone to do
it for you. It appears that the last choice rarely works very well.
That you get to use Bruce's codes at all (and for free!) is not an
accident: it is part of the design, and an outcome of the programs
being written by beamline scientists.

The current software base is more than ten years old, and showing its
age especially with respect to large data sets and multiple
processors.    This is a really good opportunity for you and your
colleagues (and others in similar situations) to join in.  That is,
you should write code to deal with your fast data rates.  If you write
that with the Ifeffit family of programs, not only will people at ESRF
be able to use it, but so will anyone else.

> If Demeter would works well for large amount of data treatment, it
> would be great. As I know ID24 at ESRF has also installed the same
> detector and would probably have similar demanding for such a program.

Looking into Demeter is one option, though I don't know exactly that
is perfect for the linear combination analysis you want to use.  Most
of the core algorithms in Demeter still use Ifeffit 1.2, which are
going to be limited for your needs, though they can be worked around
if careful.   Transferring Demeter to use the newer Ifeffit2 should be
possible, but neither Bruce or I have done more than
proof-of-principle tests.     Looking into using Larch / IFeffit2
might be better for you, and I'd be happy to help you do that.  For
one thing, this approach has no memory limitations beyond your
machine, is in Python and can read HDF5 files (which SOLEIL uses, as
far as I understand) directly.   It also has the possibility of using
multicore machines well, but this has not yet been automated.   It is
also possible that the linear algebra techniques used for x-ray
fluorescence map analysis, including code from Gerd Wellenreuther and
Armando Sole would be useful for your needs.

> Probably due to the limited memory of my pc, I can only open around
> 200 spectra simultanously with Athena.

You almost certainly want to not use Athena here anyway.   You want
automated processing programs.

Like Bruce hinted at, you've invested heavily in a detector that
allows you to collect data at a rate orders of magnitude faster than
you could before.  It would have been incredibly naive to believe that
decade-old software would have been able to keep up with that.   This
is a good opportunity for you to help out yourself and others.

Cheers,

--Matt