Problems importing larch plugins to python
Hello everyone, I recently installed the latest development version of larch from github and now I'm having some problems importing larch plugins to my python scripts. I read in the documentation that there is a new method for doing this but that the old way should still work, this is what happens if I try to do it the way I've done it before: johan@johan-Latitude-E6430:~$ python Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from larch import use_plugin_path use_plugin_path('xafs') from pre_edge import pre_edge Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/johan/.larch/plugins/xafs/pre_edge.py", line 13, in <module> from larch_plugins.std import parse_group_args ImportError: No module named larch_plugins.std
I also tried the new method described in the docs: johan@johan-Latitude-E6430:~$ python Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import larch from larch_plugins.xafs import autobk Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named larch_plugins.xafs
I'm running ubuntu 14.04. Any ideas of what is going on here? Best regards, Johan
Hi Johann,
On Wed, Sep 30, 2015 at 7:48 AM, Johan Nilsson
Hello everyone,
I recently installed the latest development version of larch from github and now I'm having some problems importing larch plugins to my python scripts. I read in the documentation that there is a new method for doing this but that the old way should still work, this is what happens if I try to do it the way I've done it before:
johan@johan-Latitude-E6430:~$ python Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from larch import use_plugin_path use_plugin_path('xafs') from pre_edge import pre_edge Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/johan/.larch/plugins/xafs/pre_edge.py", line 13, in <module> from larch_plugins.std import parse_group_args ImportError: No module named larch_plugins.std
I also tried the new method described in the docs:
johan@johan-Latitude-E6430:~$ python Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import larch from larch_plugins.xafs import autobk Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named larch_plugins.xafs
I'm running ubuntu 14.04. Any ideas of what is going on here?
Best regards, Johan
Sorry for the trouble. I broke this recently. For now, do import larch larch.enable_plugins() from larch_plugins.xafs import autobk I'll fix this soon (and then look for a better solution for what I thought I was trying to accomplish by not doing this automatically!). --Matt Newville
Hello Matt,
I was able to import pre_edge, autobk, and xftf without problems. Is it possible to import feff paths and do a fit to exafs data in a python script? I was able to do
larch.enable_plugins()
from larch_plugins.xafs import feffdat
path1 = feffdat.feffpath('feff0001.dat')
but I could not figure out how to get the feffit_transform and feffit_dataset functions to work. Before I upgraded I was able to use those two in a python script but I was not able to get the feffit function to run to do the actual fit.
Larch is really a great piece of software and I'm very thankful for all your hard work you put in developing it.
Best regards,
Johan
________________________________
From: ifeffit-bounces@millenia.cars.aps.anl.gov [ifeffit-bounces@millenia.cars.aps.anl.gov] on behalf of Matt Newville [newville@cars.uchicago.edu]
Sent: 30 September 2015 15:22
To: XAFS Analysis using Ifeffit
Subject: Re: [Ifeffit] Problems importing larch plugins to python
Hi Johann,
On Wed, Sep 30, 2015 at 7:48 AM, Johan Nilsson
from larch import use_plugin_path use_plugin_path('xafs') from pre_edge import pre_edge Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/johan/.larch/plugins/xafs/pre_edge.py", line 13, in <module> from larch_plugins.std import parse_group_args ImportError: No module named larch_plugins.std
I also tried the new method described in the docs: johan@johan-Latitude-E6430:~$ python Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import larch from larch_plugins.xafs import autobk Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named larch_plugins.xafs
I'm running ubuntu 14.04. Any ideas of what is going on here? Best regards, Johan Sorry for the trouble. I broke this recently. For now, do import larch larch.enable_plugins() from larch_plugins.xafs import autobk I'll fix this soon (and then look for a better solution for what I thought I was trying to accomplish by not doing this automatically!). --Matt Newville
Hi Johan,
On Wed, Sep 30, 2015 at 10:53 AM, Johan Nilsson
Hello Matt,
I was able to import pre_edge, autobk, and xftf without problems. Is it possible to import feff paths and do a fit to exafs data in a python script? I was able to do
larch.enable_plugins() from larch_plugins.xafs import feffdat
path1 = feffdat.feffpath('feff0001.dat')
but I could not figure out how to get the feffit_transform and feffit_dataset functions to work. Before I upgraded I was able to use those two in a python script but I was not able to get the feffit function to run to do the actual fit.
Sorry, the use from python is not as well documented as it should be. A larch script for a simple use of feffit would be: ## simple feffit in larch # read chi(k) cu_data = read_ascii('../xafsdata/cu.chi', labels='k, chi') # define fitting parameter group pars = group(amp = param(1, vary=True), del_e0 = param(0.1, vary=True), sig2 = param(0.002, vary=True), del_r = param(0., vary=True) ) # define a Feff Path, give expressions for Path Parameters path1 = feffpath('feffcu01.dat', s02 = 'amp', e0 = 'del_e0', sigma2 = 'sig2', deltar = 'del_r') # set tranform / fit ranges trans = feffit_transform(kmin=3, kmax=17, kw=2, dk=4, window='kaiser', rmin=1.4, rmax=3.0) # define dataset to include data, pathlist, transform dset = feffit_dataset(data=cu_data, pathlist=[path1], transform=trans) # perform fit out = feffit(pars, dset) # print result print feffit_report(out) ########################################### Translating that into pure Python would be: ## simple feffit in python import larch from larch import Group, Parameter from larch_plugins.io import read_ascii from larch_plugins.xafs import FeffPathGroup from larch_plugins.xafs.feffit import (TransformGroup, FeffitDataSet, feffit, feffit_report) #create larch session session = larch.Interpreter() # read chi(k) cu_data = read_ascii('../xafsdata/cu.chi', labels='k, chi', _larch=session) # define fitting parameter group pars = larch.Group(amp = Parameter(1, vary=True), del_e0 = Parameter(0.1, vary=True), sig2 = Parameter(0.002, vary=True), del_r = Parameter(0., vary=True) ) # define a Feff Path, give expressions for Path Parameters path1 = FeffPathGroup('feffcu01.dat', s02 = 'amp', e0 = 'del_e0', sigma2 = 'sig2', deltar = 'del_r', _larch=session) # set tranform / fit ranges trans = TransformGroup(kmin=3, kmax=17, kw=2, dk=4, window='kaiser', rmin=1.4, rmax=3.0, _larch=session) # define dataset to include data, pathlist, transform dset = FeffitDataSet(data=cu_data, pathlist=[path1], transform=trans, _larch=session) # perform fit out = feffit(pars, dset, _larch=session) # print result print feffit_report(out, _larch=session) ################################################## So, it's FeffPathGroup, TransformGroup and FeffitDataSet that you were looking for -- python classes, with the corresponding larch functione just return an instance of these classes. And you have to pass in a '_larch=session' to all of these classes and fnctions, as these all assume they can read/write to the session-wide larch symbol table (and, while fitting, they actually do). Yeah, a couple of these might be possible to eliminate, and it's not how you'd do it in pure python, but you'd probably build a Feffit class, and be passing around 'self', which is really not so different from "reference to global state". Anyway, it is doable in non-larch Python. Suggestions for improvement welcome! --Matt
Hello Matt,
Thank you for providing the examples. I tried to to write my own script based on your examples and things appear to be working fine until I get to the point where I want to run feffit, then I get the traceback I've pasted here below. I've attached an archive with the script and the data, do you have any idea what the problem is here?
Best regards,
Johan
johan@johan-Latitude-E6430:~/misc/larch/pd_metal_python$ python pd_metal_python.py
Traceback (most recent call last):
File "pd_metal_python.py", line 49, in <module>
out = feffit(pars, dset, _larch=mylarch)
File "/usr/local/lib/python2.7/dist-packages/larch/__init__.py", line 38, in wrapper
return fcn(*args, **keywords)
File "/home/johan/.larch/plugins/xafs/feffit.py", line 411, in feffit
fit.leastsq()
File "/usr/local/lib/python2.7/dist-packages/larch/fitting/minimizer.py", line 398, in leastsq
lsout = leastsq(self.__residual, self.vars, **lskws)
File "/usr/local/lib/python2.7/dist-packages/larch/fitting/minimizer.py", line 222, in leastsq
shape = _check_func('leastsq', 'func', func, x0, args, n)
File "/usr/lib/python2.7/dist-packages/scipy/optimize/minpack.py", line 19, in _check_func
res = atleast_1d(thefunc(*((x0[:numinputs],) + args)))
File "/usr/local/lib/python2.7/dist-packages/larch/fitting/minimizer.py", line 325, in __residual
return self.userfcn(self.paramgroup, *self.userargs, **self.userkws)
File "/home/johan/.larch/plugins/xafs/feffit.py", line 399, in _resid
return concatenate([d._residual() for d in datasets])
File "/home/johan/.larch/plugins/xafs/feffit.py", line 266, in _residual
_larch=self._larch, group=self.model)
File "/usr/local/lib/python2.7/dist-packages/larch/__init__.py", line 38, in wrapper
return fcn(*args, **keywords)
File "/home/johan/.larch/plugins/xafs/feffdat.py", line 447, in _ff2chi
path._calc_chi(k=k, kstep=kstep, kmax=kmax)
File "/home/johan/.larch/plugins/xafs/feffdat.py", line 333, in _calc_chi
if self._larch.symtable._sys.paramGroup is not None:
AttributeError: 'NoneType' object has no attribute 'symtable'
johan@johan-Latitude-E6430:~/misc/larch/pd_metal_python$
________________________________
From: ifeffit-bounces@millenia.cars.aps.anl.gov [ifeffit-bounces@millenia.cars.aps.anl.gov] on behalf of Matt Newville [newville@cars.uchicago.edu]
Sent: 01 October 2015 06:16
To: XAFS Analysis using Ifeffit
Subject: Re: [Ifeffit] Problems importing larch plugins to python
Hi Johan,
On Wed, Sep 30, 2015 at 10:53 AM, Johan Nilsson
On 10/02/2015 06:27 AM, Johan Nilsson wrote:
Thank you for providing the examples. I tried to to write my own script based on your examples and things appear to be working fine until I get to the point where I want to run feffit, then I get the traceback I've pasted here below. I've attached an archive with the script and the data, do you have any idea what the problem is here?
Johan, This reminds me a lot of one of the things I really struggled with when I started working with Larch. It turns out that, when you use larch as a puthon script (as opposed to running it through larch itself), every damn thing needs the '_larch=mylarch' argument. I am guessing (although I have not tried running your script) that your problem is that the calls defining "pars" and "path1" need a _larch=mylarch argument. Although it seems pretty clear that you basically get how to use larch for fitting exafs data, you may find some examples helpful. Here is one such: https://github.com/bruceravel/SCFtests/blob/master/Copper/Copper.py Each of the materials at https://github.com/bruceravel/SCFtests/ has a similar script in it. In each case, the fit is wrapped inside of function. This is because that project is an automated testing framework and, in each case, the fit is called by some other script. But the structure of the fit is the same as what you are trying to do. Every call to a larch-y function needs a "_larch=mylarch". Good luck, B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/
On 10/02/2015 09:09 AM, Bruce Ravel wrote:
It turns out that, when you use larch as a python script (as opposed to running it through larch itself), every damn thing needs the '_larch=mylarch' argument.
Johan, I explain why this is necessary at https://bruceravel.gitbooks.io/larch-plugin-tutorial/content/mback/details.h... Well, that is to say, I explain my best understanding of why it's necessary. Some of the inner workings of larch are still a bit opaque to me :) In any case, when you are at the larch command line (or using larch to interpret a larch script), it is unambiguous which instance of the larch interpreter each function is supposed to use. When using larch as a module in a python script, it is conceivable that more than one larch interpreter exists. So you have to be explicit. B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/
Hello Bruce, I added the _larch argument to my feff path and parameters and the script seems to be working now, thank you for spotting that. I'll have a look at the links you provided and I'll try to do some more exafs fitting with larch and hopefully I will get a better understanding of how it all works. Best regards, Johan ________________________________________ From: ifeffit-bounces@millenia.cars.aps.anl.gov [ifeffit-bounces@millenia.cars.aps.anl.gov] on behalf of Bruce Ravel [bravel@bnl.gov] Sent: 02 October 2015 15:17 To: XAFS Analysis using Ifeffit Subject: Re: [Ifeffit] Problems importing larch plugins to python On 10/02/2015 09:09 AM, Bruce Ravel wrote:
It turns out that, when you use larch as a python script (as opposed to running it through larch itself), every damn thing needs the '_larch=mylarch' argument.
Johan, I explain why this is necessary at https://bruceravel.gitbooks.io/larch-plugin-tutorial/content/mback/details.h... Well, that is to say, I explain my best understanding of why it's necessary. Some of the inner workings of larch are still a bit opaque to me :) In any case, when you are at the larch command line (or using larch to interpret a larch script), it is unambiguous which instance of the larch interpreter each function is supposed to use. When using larch as a module in a python script, it is conceivable that more than one larch interpreter exists. So you have to be explicit. B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/ _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
On 10/02/2015 09:58 AM, Johan Nilsson wrote:
I added the _larch argument to my feff path and parameters and the script seems to be working now, thank you for spotting that.
Yay! If you use a programmable editor, you might consider binding the insertion of "_larch=mylarch" to a keyboard shortcut. You will find yourself typing that A LOT as you use larch in this way. :) B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/
Hi Johan, Bruce,
Thanks. And, yes lots of the functions need a working interpreter passed
in. I'm not sure how I would do it, but it might be reasonable, and more
pythonic to do
session = larch.Interpreter()
session.FeffPathGroup(..)
session.feffit()
and so on. You'd still be typing *something* a lot, but it might be easier
to follow.
On Fri, Oct 2, 2015 at 9:04 AM, Bruce Ravel
On 10/02/2015 09:58 AM, Johan Nilsson wrote:
I added the _larch argument to my feff path and parameters and the script seems to be working now, thank you for spotting that.
Yay!
If you use a programmable editor, you might consider binding the insertion of "_larch=mylarch" to a keyboard shortcut. You will find yourself typing that A LOT as you use larch in this way. :)
B
-- Bruce Ravel ------------------------------------ bravel@bnl.gov
National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973
Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/ _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
-- --Matt Newville <newville at cars.uchicago.edu> 630-252-0431
Hello Matt,
I think that doing something like that would be quite reasonable, it would probably be a bit easier to follow which functions that need a larch interpreter. I don't know if it makes sense to think about the interpreter as a class and the functions that use interpreter as methods of that class, I'm not a very advanced python user and that is probably how I would think about it if it was written like that.
Johan
________________________________
From: ifeffit-bounces@millenia.cars.aps.anl.gov [ifeffit-bounces@millenia.cars.aps.anl.gov] on behalf of Matt Newville [newville@cars.uchicago.edu]
Sent: 02 October 2015 16:47
To: XAFS Analysis using Ifeffit
Subject: Re: [Ifeffit] Problems importing larch plugins to python
Hi Johan, Bruce,
Thanks. And, yes lots of the functions need a working interpreter passed in. I'm not sure how I would do it, but it might be reasonable, and more pythonic to do
session = larch.Interpreter()
session.FeffPathGroup(..)
session.feffit()
and so on. You'd still be typing *something* a lot, but it might be easier to follow.
On Fri, Oct 2, 2015 at 9:04 AM, Bruce Ravel
participants (3)
-
Bruce Ravel -
Johan Nilsson -
Matt Newville