Re: [Ifeffit] Ifeffit Digest, Vol 109, Issue 24

2 Apr 2012

      Thanks a lot Lello and Matt,

Now, it is clear to us that we anyway need our own program to automatically treat the big data set.

I tried XOP with XAID extension following your suggestions, it is nice to have the graphic operation, but for a big data set with thousands of spectra we need a program. Since we have already part of the program and hope we will finish it soon to a kind of platform for automatic data analysis.

Regards,
Qingyu

-----Original Message-----
From: ifeffit-bounces@millenia.cars.aps.anl.gov [mailto:ifeffit-bounces@millenia.cars.aps.anl.gov] On Behalf Of ifeffit-request@millenia.cars.aps.anl.gov
Sent: Saturday, March 31, 2012 9:47 PM
To: ifeffit@millenia.cars.aps.anl.gov
Subject: Ifeffit Digest, Vol 109, Issue 24

Send Ifeffit mailing list submissions to
	ifeffit@millenia.cars.aps.anl.gov

To subscribe or unsubscribe via the World Wide Web, visit
	http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
or, via email, send a message with subject or body 'help' to
	ifeffit-request@millenia.cars.aps.anl.gov

You can reach the person managing the list at
	ifeffit-owner@millenia.cars.aps.anl.gov

When replying, please edit your Subject line so it is more specific than "Re: Contents of Ifeffit digest..."

Today's Topics:

   1. Re: Ifeffit Digest, Vol 109, Issue 23 (Carmelo Prestipino)

----------------------------------------------------------------------

Message: 1
Date: Sat, 31 Mar 2012 21:47:19 +0200
From: Carmelo Prestipino 
To: ifeffit@millenia.cars.aps.anl.gov
Subject: Re: [Ifeffit] Ifeffit Digest, Vol 109, Issue 23
Message-ID:

Content-Type: text/plain; charset="iso-8859-1"

Hi
if I can add my two cent...(of EURO)

At the ESRF for this kind of studies some work has been done.

There is the module XAID  of the code XOP wrote by Manuel Sanchez del Rio http://www.esrf.eu/computing/scientific/xop2.1/documentation.html
It is written in IDL from scratch.
It have normalization and linear combination fit features, it is fully graphically interfaced  and in terms of performances is quite impressive.
However, the program has a complicate structure that could difficult at the first approach (but Manuel is very kind, don't hesitate  to contact him  ).

Otherwise  there is the code PrestoPronto.
code.google.com/p/*prestopronto*/
It is mainly a fully graphic python  wrapping of ifeffit for sequential analysis.
In terms of speed is very similar to Athena but it do automatically the tricks of cleaning memory But it is still a very young code......
It still has not any interface to linear combination analysis.
We never tested for more than 1000.. spectra

Probably Mark Newton  could give you some more suggestions Regards Lello

On Sat, Mar 31, 2012 at 7:00 PM,
wrote:
...
Send Ifeffit mailing list submissions to
       ifeffit@millenia.cars.aps.anl.gov
To subscribe or unsubscribe via the World Wide Web, visit
       http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
or, via email, send a message with subject or body 'help' to
       ifeffit-request@millenia.cars.aps.anl.gov
You can reach the person managing the list at
       ifeffit-owner@millenia.cars.aps.anl.gov
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Ifeffit digest..."
Today's Topics:
1. Re: Ifeffit Digest, Vol 109, Issue 21 (KONG Quingyu)
  2. Re: Ifeffit Digest, Vol 109, Issue 21 (Matt Newville)
----------------------------------------------------------------------
Message: 1
Date: Fri, 30 Mar 2012 18:32:49 +0000
From: KONG Quingyu 
To: "ifeffit@millenia.cars.aps.anl.gov"

Subject: Re: [Ifeffit] Ifeffit Digest, Vol 109, Issue 21
Message-ID:
       <
30F62EE734A9EA45A3F211513CB6200324BD1922@SUN-DAG1.synchrotron-soleil.f
r>
Content-Type: text/plain; charset="iso-8859-1"
Thank you for all your helpful suggestions.
Actually, our beamline has recently installed a fast detector with a
repetation of 20KHz, which could collected data in a time resolution
of few tens of microsecond and could easily produce a large amount of
data. We have already accepted three users for microsecond time
resolved studies, each of them collected millions of spectra.
Together with the company that provide the detector and our software
engineer we started to write a Python program to treat the data from
November last year, however our engineer quitted recently, this make
us to look for other solutions.
If Demeter would works well for large amount of data treatment, it
would be great. As I know ID24 at ESRF has also installed the same
detector and would probably have similar demanding for such a program.
Probably due to the limited memory of my pc, I can only open around
200 spectra simultanously with Athena.
Best,
Qingyu Kong
Beamline ODE
Soci?t? civile Synchrotron SOLEIL
L'Orme des Merisiers
Saint-Aubin - BP 48
91192 GIF-sur-YVETTE CEDEX
Tel: +33 (0)1 69 35 97 84 (office)
     +33 (0)1 69 35 97 16 (beamline)
email: kong@synchrotron-soleil.fr
________________________________________
From: ifeffit-bounces@millenia.cars.aps.anl.gov [
ifeffit-bounces@millenia.cars.aps.anl.gov] on behalf of
ifeffit-request@millenia.cars.aps.anl.gov [
ifeffit-request@millenia.cars.aps.anl.gov]
Sent: Friday, March 30, 2012 3:29 PM
To: ifeffit@millenia.cars.aps.anl.gov
Subject: Ifeffit Digest, Vol 109, Issue 21
Send Ifeffit mailing list submissions to
       ifeffit@millenia.cars.aps.anl.gov
To subscribe or unsubscribe via the World Wide Web, visit
       http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
or, via email, send a message with subject or body 'help' to
       ifeffit-request@millenia.cars.aps.anl.gov
You can reach the person managing the list at
       ifeffit-owner@millenia.cars.aps.anl.gov
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Ifeffit digest..."
Today's Topics:
1. delr and ss parameterization (Ebrahim Rezaei)
  2. program (KONG Quingyu)
  3. Re: program (Carlo Segre)
  4. Re: program (Matt Newville)
  5. Re: delr and ss parameterization (Bruce Ravel)
  6. Re: program (Bruce Ravel)
----------------------------------------------------------------------
Message: 1
Date: Thu, 29 Mar 2012 21:57:21 -0700 (PDT)
From: Ebrahim Rezaei 
To: ifeffit@millenia.cars.aps.anl.gov
Subject: [Ifeffit] delr and ss parameterization
Message-ID:
<1333083441.66190.YahooMailClassic@web120501.mail.ne1.yahoo.com>
Content-Type: text/plain; charset="utf-8"
Hi everyone,
I am working with bulk and nano structures of MnO, MnO2 and Mn2O3. I
have two questions regarding how to parameterize delr and ss in
Artemis.
It is recommended to
use either delr=alfa*Reff or relate it to unit cell structure. Can
anybody tell me how to do the second approach?
Also is that a good idea to let all ss to float? Or it should be
constrained based on type and distance of atoms? I have seen an
example about ZnO just letting the first two shell?s ss to float and
let the rest to be the same. I would appreciate it very much if
anybody could help me with this too.
Regards,
Ebrahim Rezaei
PhD Candidate, Chem. Eng. Dep., University of Saskatchewan
Saskatoon, SK, Canada
-------------- next part -------------- An HTML attachment was
scrubbed...
URL: <
http://millenia.cars.aps.anl.gov/pipermail/ifeffit/attachments/2012032
9/e13d64a5/attachment-0001.htm
...
------------------------------
Message: 2
Date: Fri, 30 Mar 2012 06:46:38 +0000
From: KONG Quingyu 
To: "ifeffit@millenia.cars.aps.anl.gov"

Subject: [Ifeffit] program
Message-ID:
       <
30F62EE734A9EA45A3F211513CB6200324BD1794@SUN-DAG1.synchrotron-soleil.f
r>
Content-Type: text/plain; charset="iso-8859-1"
We recently collected 3000 EXAFS spectra to follow a chemical reaction
process, we would like to fit the experimental data with reference
spectra using linear combination fitting. Does anyone know there are
available or commercial program to normalize the data and make linear
combination fitting. We tried to do it with Athena manually, but 3000
spectra take a long time for us and it is not a practical way.
Thanks a lot.
Qingyu Kong
Beamline ODE
Soci?t? civile Synchrotron SOLEIL
L'Orme des Merisiers
Saint-Aubin - BP 48
91192 GIF-sur-YVETTE CEDEX
Tel: +33 (0)1 69 35 97 84 (office)
     +33 (0)1 69 35 97 16 (beamline)
email: kong@synchrotron-soleil.fr
------------------------------
Message: 3
Date: Fri, 30 Mar 2012 01:56:57 -0500 (CDT)
From: Carlo Segre 
To: XAFS Analysis using Ifeffit 
Subject: Re: [Ifeffit] program
Message-ID: 
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Hi Qingyu
On Fri, 30 Mar 2012, KONG Quingyu wrote:
...
We recently collected 3000 EXAFS spectra to follow a chemical
reaction process, we would like to fit the experimental data with
reference spectra using linear combination fitting. Does anyone know
there are available or commercial program to normalize the data and
make linear combination fitting. We tried to do it with Athena
manually, but 3000 spectra take a long time for us and it is not a practical way.
You might wish to look into Bruce Ravel's new suite of programs
(called Demeter).  I believe they are built to allow easy scripting of
massive quantities of data such as yours.
Carlo
--
Carlo U. Segre -- Duchossois Leadership Professor of Physics Associate
Dean for Graduate Admissions, Graduate College Illinois Institute of
Technology
Voice: 312.567.3498            Fax: 312.567.3494
segre@iit.edu   http://phys.iit.edu/~segre   segre@debian.org
------------------------------
Message: 4
Date: Fri, 30 Mar 2012 07:56:05 -0500
From: Matt Newville 
To: XAFS Analysis using Ifeffit 
Subject: Re: [Ifeffit] program
Message-ID:

...
Content-Type: text/plain; charset=ISO-8859-1
Hi Qingyu,
...
We recently collected 3000 EXAFS spectra to follow a chemical
reaction
On Fri, Mar 30, 2012 at 1:46 AM, KONG Quingyu
 wrote:
process, we would like to fit the experimental data with reference
spectra using linear combination fitting. Does anyone know there are
available or commercial program to normalize the data and make linear
combination fitting. We tried to do it with Athena manually, but 3000
spectra take a long time for us and it is not a practical way.
Ifeffit is designed to be scriptable for batch processing.   This can
be done with a raw Ifeffit script (which isn't very flexible) or from
Perl or Python, which are more flexible.   Carlo mentioned Demeter,
which is a high-level Perl wrapping of the lower-level functionality.
  Working with 3000 spectra may be challenging with Ifeffit version
1.2, as the total amount of available memory is limited, but with some
care it should be possible.
FWIW, the next version of Ifeffit will eliminate this memory
limitation, and make both scripting and extending the functionality
much easier and better.  This is still a work in progress (see
https://github.com/xraypy/xraylarch) and poorly documented, but the
essential idea is to write all of Ifeffit's functionality in Python,
for all the reasons just stated, and this far enough along that I am
confident it will be a big improvement.   So far (for XAFS), this can
read data, do simple pre-edge subtraction, normalization, remove
backgrounds ala autobk, and do XAFS Fourier transforms.   It can do
simple "general purpose fitting", but does not yet expose a
simple-to-use linear combination fitting functionality (shouldn't be
too hard, as all the linear algebra pieces are readily available) or
fitting spectra to Feff calculations (a bit more involved).   Work is
slow (mostly due to other obligations),  but progress is being made,
and it's close to ready for an initial release.   If you are (or
anyone else is) interested in trying this or helping out, let me know.
--Matt
------------------------------
Message: 5
Date: Fri, 30 Mar 2012 09:13:11 -0400
From: Bruce Ravel 
To: XAFS Analysis using Ifeffit 
Subject: Re: [Ifeffit] delr and ss parameterization
Message-ID: <201203300913.12166.bravel@bnl.gov>
Content-Type: Text/Plain;  charset="utf-8"
On Friday, March 30, 2012 12:57:21 am Ebrahim Rezaei wrote:
...
I am working with bulk and nano structures of MnO, MnO2 and Mn2O3. I
have two questions regarding how to parameterize delr and ss in
Artemis.
It is recommended to
use either delr=alfa*Reff or relate it to unit cell structure. Can
anybody
tell me how to do the second approach?
Also is that a good idea to let all ss to float? Or it should be
constrained based on type and distance of atoms? I have seen an
example
about ZnO just letting the first two shell?s ss to float and let the
rest to be the same. I would appreciate it very much if anybody
could help me with this too.
The alpha*reff trick is only strictly valid for cubic, monoatomic
materials.  For everything else in the world, it is an increasingly
poor approximation.  That's not to say it is not a helpful
parameterization, just that you need to be mindful of its limitations.
As for the sigma^2 parameter, you have to balance when is physically
reasonable with what can be reliably extracted from a real data
analysis.
- Two atoms of a different species at the same distance should have
   different sigma^2 because they have different masses.
- Two atoms of the same species at different distances should have
   different sigma^2 because sigma^2 is a mean square deviation in
   distance.
So, in some sense, all scatterers might have their own sigma^2
parameters.  But that is not to say that you data are sufficient to
independently determine that many parameters.  When you run up against
the limited information content of your data, that is when you start
making approximations of the sort that lead to setting sigma^2
parameters for different scatterers to be the same.
However, there are sensible and physically justifiable reasons why the
sigma^2 for certain multiple scattering paths should be related to
those of single scattering paths.  See
http://dx.doi.org/10.1103/PhysRevB.54.156
Two of my favorite papers that show excellent data analysis solving
real-world problems and which are written in a way that is of
pedagogical value for XAS beginners are Scott Calvin's paper on doped
ferrites (http://dx.doi.org/10.1103/PhysRevB.66.224405) and Shelly
Kelly's paper on uranyl binding to biomass
(http://dx.doi.org/10.1016/S0016-7037(02)00947-X).  For more
interesting papers, look through the archives of the mailing list and
see who gives good answers to questions, then look up their papers.
Another extremely useful resource which includes extensive discussion
of the sorts of things you are asking about is Shelly's review
chapter.  Here's the amazon link to that book:
http://www.amazon.com/Methods-Soil-Analysis-Part-Mineralogical/dp/0891
188460/ The book is a bit pricy, but if you are a clever googler, you
may find a PDF copy of her chapter.
Regards,
B
--
Bruce Ravel  ------------------------------------ bravel@bnl.gov
National Institute of Standards and Technology  Synchrotron Methods
Group at NSLS --- Beamlines U7A, X24A, X23A2  Building 535A  Upton NY,
11973
My homepage:    http://xafs.org/BruceRavel
 EXAFS software: http://cars9.uchicago.edu/ifeffit/Demeter
------------------------------
Message: 6
Date: Fri, 30 Mar 2012 09:29:10 -0400
From: Bruce Ravel 
To: XAFS Analysis using Ifeffit 
Subject: Re: [Ifeffit] program
Message-ID: <201203300929.11143.bravel@bnl.gov>
Content-Type: Text/Plain;  charset="iso-8859-1"
On Friday, March 30, 2012 02:46:38 am KONG Quingyu wrote:
...
We recently collected 3000 EXAFS spectra to follow a chemical
reaction process, we would like to fit the experimental data with
reference
spectra
using linear combination fitting. Does anyone know there are
available or commercial program to normalize the data and make
linear combination fitting. We tried to do it with Athena manually,
but 3000 spectra take a long time for us and it is not a practical way.
Hi Qingyu,
Carlo is correct, Demeter could be used to write a special purpose
program for automating this sort of thing.  The performance in terms
of speed may not be exactly what you want, but the performance in
terms of quickly prototyping a program to process and analyze a large
data volume is excellent.
The Demeter code base comes with extensive (although still incomplete)
documentation as well as plenty of example scripts, including examples
for the individual chores you mention above.
Here's Demeter's documentation: http://bruceravel.github.com/demeter/
As a closing comment, isn't it somehow strange that you and your
colleagues went to all the trouble of building a fancy dispersive
beamline without thinking ahead about how you were going to analyze
your data?
B
--
Bruce Ravel  ------------------------------------ bravel@bnl.gov
National Institute of Standards and Technology  Synchrotron Methods
Group at NSLS --- Beamlines U7A, X24A, X23A2  Building 535A  Upton NY,
11973
My homepage:    http://xafs.org/BruceRavel
 EXAFS software: http://cars9.uchicago.edu/ifeffit/Demeter
------------------------------
_______________________________________________
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
End of Ifeffit Digest, Vol 109, Issue 21
****************************************
------------------------------
Message: 2
Date: Fri, 30 Mar 2012 16:36:58 -0500
From: Matt Newville 
To: XAFS Analysis using Ifeffit 
Subject: Re: [Ifeffit] Ifeffit Digest, Vol 109, Issue 21
Message-ID:

...
Content-Type: text/plain; charset=ISO-8859-1
Hi Qingyu,
On Fri, Mar 30, 2012 at 1:32 PM, KONG Quingyu
 wrote:
...
Thank you for all your helpful suggestions.
Actually, our beamline has recently installed a fast detector with a
repetation of 20KHz, which could collected data in a time resolution
of few tens of microsecond and could easily produce a large amount
of data. We have already accepted three users for microsecond time
resolved studies, each of them collected millions of spectra.
Together with the company that provide the detector and our software
engineer we started to write a Python program to treat the data from
November last year, however our engineer quitted recently, this make
us to look for other solutions.
Bruce wrote earlier,
As a closing comment, isn't it somehow strange that you and your
  colleagues went to all the trouble of building a fancy dispersive
  beamline without thinking ahead about how you were going to analyze
  your data?
Sadly, Bruce's observation is not unusual at all.  Everyone is
experiencing data overloads, and almost no one is investing in
computing resources.  Yours may be slightly larger than most XAFS
users, but only by one or two orders of magnitude.  Then again, you
had a software engineer at your beamline at least for a while.
To be clear, Ifeffit, Athena, Demeter, Sixpack, and so on are written
primarily by beamline scientists like yourself.  Your choices are to
use what someone else gives you, write your own, or hire someone to do
it for you. It appears that the last choice rarely works very well.
That you get to use Bruce's codes at all (and for free!) is not an
accident: it is part of the design, and an outcome of the programs
being written by beamline scientists.
The current software base is more than ten years old, and showing its
age especially with respect to large data sets and multiple
processors.    This is a really good opportunity for you and your
colleagues (and others in similar situations) to join in.  That is,
you should write code to deal with your fast data rates.  If you write
that with the Ifeffit family of programs, not only will people at ESRF
be able to use it, but so will anyone else.
...
If Demeter would works well for large amount of data treatment, it
would be great. As I know ID24 at ESRF has also installed the same
detector and would probably have similar demanding for such a program.
Looking into Demeter is one option, though I don't know exactly that
is perfect for the linear combination analysis you want to use.  Most
of the core algorithms in Demeter still use Ifeffit 1.2, which are
going to be limited for your needs, though they can be worked around
if careful.   Transferring Demeter to use the newer Ifeffit2 should be
possible, but neither Bruce or I have done more than
proof-of-principle tests.     Looking into using Larch / IFeffit2
might be better for you, and I'd be happy to help you do that.  For
one thing, this approach has no memory limitations beyond your
machine, is in Python and can read HDF5 files (which SOLEIL uses, as
far as I understand) directly.   It also has the possibility of using
multicore machines well, but this has not yet been automated.   It is
also possible that the linear algebra techniques used for x-ray
fluorescence map analysis, including code from Gerd Wellenreuther and
Armando Sole would be useful for your needs.
...
Probably due to the limited memory of my pc, I can only open around
200 spectra simultanously with Athena.
You almost certainly want to not use Athena here anyway.   You want
automated processing programs.
Like Bruce hinted at, you've invested heavily in a detector that
allows you to collect data at a rate orders of magnitude faster than
you could before.  It would have been incredibly naive to believe that
decade-old software would have been able to keep up with that.   This
is a good opportunity for you to help out yourself and others.
Cheers,
--Matt
------------------------------
_______________________________________________
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
End of Ifeffit Digest, Vol 109, Issue 23
****************************************
--
Carmelo Prestipino
Sciences Chimiques de Rennes UMR 6226
Solid State and Materials Chemistry group (CSM) Campus de Beaulieu, B?t 10B, bureau 122
F-35042 RENNES
France
Tel: (+33) 22323 6531
FAX: (+33) 22323 5959