Larch user base?

older
Phase correction not working in...

Sean Fackler

14 Aug 2016 14 Aug '16

9:35 p.m.

Hello all! Do you have an idea of Larch user base or Larch package download stats? Thanks for any information. Regards, Sean Fackler, PhD Joint Center for Artificial Photosynthesis Lawrence Berkeley National Lab 1 Cyclotron Rd Mail Stop 30R0205 Berkeley CA, 94720 Mobile: 609-613-8734

Attachments:

attachment.html (text/html — 1.6 KB)

Show replies by date

Matt Newville

15 Aug 15 Aug

4:35 a.m.

Hi Sean, On Sun, Aug 14, 2016 at 4:35 PM, Sean Fackler wrote:

...

Hello all!

Do you have an idea of Larch user base or Larch package download stats? Thanks for any information.

Funny you should ask. It turns out that a kind and generous NSF Program Manager recently gave me a friendly reminder that I promised to include such information in a report to them. I think I haven't been doing enough to promote or discuss development of Larch, so while we're at it, I should probably give an update on the status of Larch and where it looks to be going over the next couple years. First, your question: The recommended way to download / install Larch in now via Anaconda.org: https://anaconda.org/newville/xraylarch, which conveniently keeps statistics. This reports more than 180 downloads in 2016. That's low compared to Athena/Artemis, but not terrible. Many of these are probably from users of my beamline, where Larch includes the Visualization tools for the XRF maps we make, but then again, this was a major motivation for Larch. For another metric, I counted about 1 discussion per month on this list about Larch. There are at least that many discussions where Bruce or I eventually say "what you're asking would be much easier with Larch". While low compared to questions about using Demeter many of the Demeter questions posed don't really discuss the underlying analysis engine. Anecdotally, I've had many discussions over the past year about Larch, including getting "advanced" Athena/Artemis users and beamline scientists to use it. The conversations always seem positive to me, but I don't know that many people have really taken it up from these discussions. I've also had several discussions with people (beamline scientists or visiting scientists) at the APS in a similar manner -- many seem interested, but I'm not sure the adoption rate is very high. There are definitely a handful of folks trying to use it from Python. Athena and Artemis can already use Larch, and I think both Bruce and I are expecting that Larch will be expected to be the default over the next year. Larch status and near-term directions: I'm about to merge a large set of changes (https://github.com/xraypy/ xraylarch/pull/139) and push a new release of Larch (0.9.30), probably later this week. The main changes / updates for this release are: 1. initial support for Python 3. All the basic stuff is working, but there may still be some small problems with unicode vs. strings, and with the wxPython GUIs 2. support for wxPhoenix, the not-yet-officially-released version of wxPython that will support Python 3. Again, there may be a few problems still to uncover, but stuff is mostly working for me with wxPhoenix on wxPython. 3. improvements and simplifications to the internal "read-eval-print" loop code (some of the oldest code in larch) and to the presentation of exceptions / error messages. 4. A reader for Athena project files to Larch groups. A writer will soon follow, I expect. As mentioned, we have funding and have hired a scientist to work on parts of Larch. This is to integrate and/or improve 4 aspects: 1. handling micro-XRD data, and automating phase ID from micro-XRD maps. 2. better fitting / quantitative XRF analysis. 3. better XANES fitting and linear analysis methods (PCA, and more). 4. improved EXAFS analysis, incorporating dedicated Feff85EXAFS. We think we know how to do all of these steps, but we'd love help with any part of this. The person we've hired is tackling #1 first, For my part, I'll be working on #3, and a dedicated GUI for XANES analysis over the next several months. There are several related projects (mostly python), including https://github.com/xraypy/feff85exafs, https://github.com/scikit-beam/, https://github.com/lmfit/lmfit-py. LMFIT grew out of Larch, but has grown substantially from outside collaborators. I expect that part of the XANES and XRF fitting work will migrate Larch's fitting to just depend on LMFIT. It's possible that some of the domain-specific libraries for analysis (Xray databases, for example at https://github.com/scikit-beam/XrayDB) will migrate to scikit-beam, with Larch then depending on that. Larch is definitely ready to be used. Many aspects are still in development, but the core EXAFS analysis is working. If you or anyone is interested in trying it or working with the code, don't hesitate to give it a try. Sorry for such a long answer to a short question! --Matt

Sean Fackler

5:23 p.m.

Thanks Matt for the thorough explanation. Actually it was Jamie Sethian who asked. I am working on using Larch as a code basis for doing high-throughput NEXAFS analysis. You can imagine automating just basic normalizing and background subtraction steps would be invaluable in this case. I am working with CAMERA ( Alex Hexamer, Ronald Pandolfi ) on making a GUI and Materials Project ( Patrick Huck ) along with ALS ( Alpha N'Daiye ) to have some sort of database contribution route. Data is coming from ALS beam line 6.3.1. I will be calling on you as I learn Python concurrently with implementing Larch. For example, I got read_ascii to work but couldn't get the plot functions to work and get "newplot not defined." I am using 2.7 on Mac OSX El Capitan. Any ideas? Thank you! Sean On Sunday, August 14, 2016, Matt Newville wrote:

...

Hi Sean,

On Sun, Aug 14, 2016 at 4:35 PM, Sean Fackler javascript:_e(%7B%7D,'cvml','swfackler@lbl.gov');> wrote:

...
Hello all!

Do you have an idea of Larch user base or Larch package download stats? Thanks for any information.

Funny you should ask. It turns out that a kind and generous NSF Program Manager recently gave me a friendly reminder that I promised to include such information in a report to them. I think I haven't been doing enough to promote or discuss development of Larch, so while we're at it, I should probably give an update on the status of Larch and where it looks to be going over the next couple years.

First, your question:

The recommended way to download / install Larch in now via Anaconda.org: https://anaconda.org/newville/xraylarch, which conveniently keeps statistics. This reports more than 180 downloads in 2016. That's low compared to Athena/Artemis, but not terrible. Many of these are probably from users of my beamline, where Larch includes the Visualization tools for the XRF maps we make, but then again, this was a major motivation for Larch.

For another metric, I counted about 1 discussion per month on this list about Larch. There are at least that many discussions where Bruce or I eventually say "what you're asking would be much easier with Larch". While low compared to questions about using Demeter many of the Demeter questions posed don't really discuss the underlying analysis engine.

Anecdotally, I've had many discussions over the past year about Larch, including getting "advanced" Athena/Artemis users and beamline scientists to use it. The conversations always seem positive to me, but I don't know that many people have really taken it up from these discussions. I've also had several discussions with people (beamline scientists or visiting scientists) at the APS in a similar manner -- many seem interested, but I'm not sure the adoption rate is very high. There are definitely a handful of folks trying to use it from Python.

Athena and Artemis can already use Larch, and I think both Bruce and I are expecting that Larch will be expected to be the default over the next year.

Larch status and near-term directions:

I'm about to merge a large set of changes (https://github.com/xraypy/xra ylarch/pull/139) and push a new release of Larch (0.9.30), probably later this week. The main changes / updates for this release are: 1. initial support for Python 3. All the basic stuff is working, but there may still be some small problems with unicode vs. strings, and with the wxPython GUIs 2. support for wxPhoenix, the not-yet-officially-released version of wxPython that will support Python 3. Again, there may be a few problems still to uncover, but stuff is mostly working for me with wxPhoenix on wxPython. 3. improvements and simplifications to the internal "read-eval-print" loop code (some of the oldest code in larch) and to the presentation of exceptions / error messages. 4. A reader for Athena project files to Larch groups. A writer will soon follow, I expect.

As mentioned, we have funding and have hired a scientist to work on parts of Larch. This is to integrate and/or improve 4 aspects: 1. handling micro-XRD data, and automating phase ID from micro-XRD maps. 2. better fitting / quantitative XRF analysis. 3. better XANES fitting and linear analysis methods (PCA, and more). 4. improved EXAFS analysis, incorporating dedicated Feff85EXAFS.

We think we know how to do all of these steps, but we'd love help with any part of this. The person we've hired is tackling #1 first, For my part, I'll be working on #3, and a dedicated GUI for XANES analysis over the next several months.

There are several related projects (mostly python), including https://github.com/xraypy/feff85exafs, https://github.com/scikit-beam/, https://github.com/lmfit/lmfit-py. LMFIT grew out of Larch, but has grown substantially from outside collaborators. I expect that part of the XANES and XRF fitting work will migrate Larch's fitting to just depend on LMFIT. It's possible that some of the domain-specific libraries for analysis (Xray databases, for example at https://github.com/scikit- beam/XrayDB) will migrate to scikit-beam, with Larch then depending on that.

Larch is definitely ready to be used. Many aspects are still in development, but the core EXAFS analysis is working. If you or anyone is interested in trying it or working with the code, don't hesitate to give it a try.

Sorry for such a long answer to a short question!

--Matt

-- Sean Fackler, PhD Joint Center for Artificial Photosynthesis Lawrence Berkeley National Lab 1 Cyclotron Rd, Mail Stop 30R0205 Berkeley CA, 94720

Matt Newville

7:59 p.m.

Hi Sean On Mon, Aug 15, 2016 at 12:23 PM, Sean Fackler wrote:

...

Thanks Matt for the thorough explanation.

Actually it was Jamie Sethian who asked. I am working on using Larch as a code basis for doing high-throughput NEXAFS analysis. You can imagine automating just basic normalizing and background subtraction steps would be invaluable in this case. I am working with CAMERA ( Alex Hexamer, Ronald Pandolfi ) on making a GUI and Materials Project ( Patrick Huck ) along with ALS ( Alpha N'Daiye ) to have some sort of database contribution route. Data is coming from ALS beam line 6.3.1.

Yes, this is exactly the sort of thing Larch is meant to do!

...

I will be calling on you as I learn Python concurrently with implementing Larch. For example, I got read_ascii to work but couldn't get the plot functions to work and get "newplot not defined." I am using 2.7 on Mac OSX El Capitan. Any ideas?

Do you have wxPython installed and get the other dependencies (including wxmplot) installed? FWIW, I run it on a Macbook with El Capitan all the time, and have seen many successful installs with Anaconda. If you are using Anaconda python on OS X, there is a weirdness with wxPython that requires using "pythonw" instead of "python" as the interpreter (some Framework thing), but that "just" makes running larch from a command-line or script fail immediately with a message about not being able to draw to the screen. The "larch_makeicons" makes mini-Apps that avoids this, and the next version will fix this even for the commandline programs. --Matt

Sean Fackler

1 Sep 1 Sep

12:25 a.m.

Hi Matt, I do have wxPython installed and wxmplot. I use anaconda which should handle the obvious dependencies. Is there a list of known dependencies I should double check?

...

If you are using Anaconda python on OS X, there is a weirdness with wxPython that requires using "pythonw" instead of "python" as the interpreter (some Framework thing), but that "just" makes running larch from a command-line or script fail immediately with a message about not being able to draw to the screen. The "larch_makeicons" makes mini-Apps that avoids this, and the next version will fix this even for the commandline programs.

I am using Anaconda. I am a newb at Python. What do you mean I use pythonw as the interpreter? I am trying to script with larch and I don’t see any obvious places to replace python with pythonw. I tried installing a new Anaconda environment but didn’t see an obvious point at which to change the interpreter to pythonw. Thanks for any further explanation. I get new_plot to work fine in the larch command line but I want to create new “modules” which I understand needs to be done by scripting larch in a python environment. Sean

...

On Aug 15, 2016, at 12:59 PM, Matt Newville wrote:

Hi Sean

On Mon, Aug 15, 2016 at 12:23 PM, Sean Fackler mailto:swfackler@lbl.gov> wrote: Thanks Matt for the thorough explanation.

Actually it was Jamie Sethian who asked. I am working on using Larch as a code basis for doing high-throughput NEXAFS analysis. You can imagine automating just basic normalizing and background subtraction steps would be invaluable in this case. I am working with CAMERA ( Alex Hexamer, Ronald Pandolfi ) on making a GUI and Materials Project ( Patrick Huck ) along with ALS ( Alpha N'Daiye ) to have some sort of database contribution route. Data is coming from ALS beam line 6.3.1.

Yes, this is exactly the sort of thing Larch is meant to do!

I will be calling on you as I learn Python concurrently with implementing Larch. For example, I got read_ascii to work but couldn't get the plot functions to work and get "newplot not defined." I am using 2.7 on Mac OSX El Capitan. Any ideas?

Do you have wxPython installed and get the other dependencies (including wxmplot) installed? FWIW, I run it on a Macbook with El Capitan all the time, and have seen many successful installs with Anaconda.

If you are using Anaconda python on OS X, there is a weirdness with wxPython that requires using "pythonw" instead of "python" as the interpreter (some Framework thing), but that "just" makes running larch from a command-line or script fail immediately with a message about not being able to draw to the screen. The "larch_makeicons" makes mini-Apps that avoids this, and the next version will fix this even for the commandline programs.

--Matt _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

Matt Newville

3:19 p.m.

Hi Sean, On Wed, Aug 31, 2016 at 7:25 PM, Sean Fackler wrote:

...

Hi Matt,

I do have wxPython installed and wxmplot. I use anaconda which should handle the obvious dependencies. Is there a list of known dependencies I should double check?

if you're using Anaconda, the main dependencies are included in the basic setup, and doing "conda install -c newville xraylarch" should install everything else you need. For reference (or if not using Anaconda), the hard dependencies are numpy, scipy, matplotlib, six, h5py, sqlalchemy, wxmplot, and wxutils. The first six of those are common general purpose or scientific python tools (and may include compiled code or other dependencies that I don't list, but will get installed with them), and the last two are libraries I maintain (and are pure Python and add no other dependencies). There are two optional packages: termcolor, and pyepics that may be used if available but are not required. Termcolor is used by the command-line program to give color to the output text and pyepics is useful if you want to communicate with the Epics control system (which is used at some beamlines, especially in the US).

...

If you are using Anaconda python on OS X, there is a weirdness with wxPython that requires using "pythonw" instead of "python" as the interpreter (some Framework thing), but that "just" makes running larch from a command-line or script fail immediately with a message about not being able to draw to the screen. The "larch_makeicons" makes mini-Apps that avoids this, and the next version will fix this even for the commandline programs.

I am using Anaconda. I am a newb at Python. What do you mean I use pythonw as the interpreter? I am trying to script with larch and I don’t see any obvious places to replace python with pythonw. I tried installing a new Anaconda environment but didn’t see an obvious point at which to change the interpreter to pythonw.

For Anaconda on Mac OS X, using the standard "python" executable cannot draw to the screen with wxPython. Instead, one must use ~> pythonw my_larch_using_script.py instead of ~> python my_larch_using_script.py You can also put "#!/usr/bin/env pythonw" at the top of an executable script, not "#!/usr/bin/env python". The installed programs "larch", "larch_gui", and so forth do this automatically on Mac OS X, and the Apps in the Larch folder built with "larch_makeicons" use the windowed interpreter. For clarity, this irritating feature of "this program cannot draw to the window, it can only run from a terminal/console" was a common issue ten or more years ago on both Windows and Mac (Unices that used X11 solved this issue before anyone named Bush or Clinton was president), I'm sure you're asking "why don't they just rename 'pythonw' 'python' and be done with it"? I don't know why this doesn't work, but I can tell you from experience that it does not. Also, pythonw is not needed on Windows or Mac using "python.org python" or "macports python" (and probably not in "homebrew python"). I have complained to Continuum, as have others.

...

Thanks for any further explanation. I get new_plot to work fine in the larch command line but I want to create new “modules” which I understand needs to be done by scripting larch in a python environment.

There are two basic scenarios: A) you want to write code that can be used inside Larch. For this, writing plain Python and just import your code from within Larch should work fine. If you want to write code that creates Larch Groups or builds on existing functionality, you can import larch and even its plugins. B) you want to use Larch code from within plain Python. For this, you sort of need to know where in the plugin hierarchy the functions you want to use are, but can then do something like: import larch from larch_plugins.xafs import autobk, pre_edge To *use* many of the Larch plugin functions, it is necessary (or, at least, advised) to pass in an existing larch "session", using the '_larch' keyword argument to all (or at least the vast majority) of the plugin functions, like this: _session = larch.Interpreter() outgroup = larch.Group() autobk(endata, mudata, rbkg=1.0, out=outgroup, _larch=_session) Hope that's a good start! --Matt

3003

Age (days ago)

3021

Last active (days ago)

List overview

Download

5 comments

2 participants

participants (2)

Matt Newville
Sean Fackler

Larch user base?

Sean Fackler

Matt Newville

Sean Fackler

Matt Newville

Sean Fackler

Matt Newville

tags

participants (2)