Hi Bruce, Thanks for the useful answer, as usual : Bruce Ravel wrote:

Hi Mauro,

Such an interesting email! I can think of a number of avenues that might be useful for you to pursue -- most imporantly: http://cars9.uchicago.edu/iffwiki/tdl

TDL? Why creating a new language instead of using Python directly? Well, in any case TDL will be Python-compatible and this is (very) good in my point of view. I appreciate the Python-oriented Ifeffit 2.

Perhaps you could tell us what you are interested in doing. What new capabilities are you interested in implementing? What idea do you have about making that happen?

Well, my two long-term projects about XAFS analysis are : 1. integrating DFT ab-initio calculations with EXAFS (as we discussed on March 2007 on the mailing-list); 2. analysis of reflectivity EXAFS spectra taken in grazing incidence (total reflection mode), without the introduction of approximations (that is considering the real and imaginary part of the refractive index). In addition, minor utilities intended to solve repetitive operations during fitting procedure of common materials classes (ex. varying temperature, concentration, etc.). At the moment the solutions I prefer are to work with the Python Ifeffit wrapper for big projects and Demeter for automation of fitting procedures after doing the first working theoretical model with Artemis.

PS. In the last few weeks, I have been making a lot of progress on Demeter after a long time away. I doubt that I have fixed any particular complaint, since I have been working on some stuff that is pretty far from the user (or even application programmer) end of

Yes, I'm aware of your last weeks modifications because with the last SVN revision Demeter refuses to start... I will investigate the error and in case of problems I will ask your help. ;)

things. I don't want to discourage you from pursuing other avenues, but it would be helpful to me to know what you think the obvious short-comings are.

I think TDL/Python for Ifeffit2 and Perl for Demeter are great solutions and represent the right tools to build user-specific XAFS applications. On the other hand, I think that to use these development tools we (the users / application programmers) should follow the coding process, that is sharing the ongoing work. I'm thinking at an "ifeffit-dev" mailing-list but I don't know if it is a good idea. In any case is useful to know in which direction are going Ifeffit & friends. M. -- Mauro Rovezzi PhD student - GILDA BM08 c/o ESRF (Grenoble, FR)

Greetings,

I do think an ifeffit-dev list and/or wiki area is a fine idea. I don't know how many others would be interested in this, but it might allow for more open development, and keep the ifeffit mailing list more for usage questions. Any other opinions on this?

I second the idea. A -dev list would also help avoid confusion. I can see that people might get really confused if one set of people are talking about some software in development and other people are talking about the program that people are using.

Mauro Rovezzi wrote:

...

For these reasons I would like to know your point of view on the idea to find a standard in the creation of Ifeffit applications in order to permit the interested community to contribute/share in the programming effort. Practically speaking, if someone would like to implement a new functionality based on the Ifeffit library which directives should follow to integrate the existing applications?

I think that there are several possibilities but I think it would best to look at where the codebase is going. The development of tdl and demeter look like good solutions. The question is how can other people connect into these things. For example, a plugin like structure might be used. Someone adding a new function could for example write a plugin to do something like: 1. ask ifeffit for data and other needed information 2. do some tranform/filter/magic on data 3. return data to ifeffit Other people could use plugin by calling it from within ifeffit. You can do this now with the wrappers but it appears to me that this might be easier to do with tdl and/or demeter as these things seem better designed to do just that. Perhaps Matt or Bruce would like to comment further. Or people could work more directly with tdl and demeter. In the end I think that a modular approach would be best. It could be called the XAFS-toolbox or something. ;-)

Well, now that question (TDL?? Are you nuts??) is a fine and fair question, and one I've been asked before and in fact ask myself on a regular basis.

I also kind of wondered as I am am normally sceptical of creating a new language. However, one advantage of creating a mini language is that you can tailor it to the specific application. This way you can get the language to 'do what I mean' in a given situation. Cheers, Adam -- ---------------------------------------------------------- Dr. M. Adam Webb HASYLAB at DESY Notkestrasse 85 D-22607 Hamburg Germany Phone: +49 40 8998-1994, Fax: +49 40 8998-2787 E-mail: adam.webb@desy.de -----------------------------------------------------------

So low level support for interacting at the "transform/filter/magic" level exists. I suspect, however, that you are not really asking about low level support. I suspect that you are looking for something more fleshed out:

-- Suppose you were doing some quick-XAS and had 10,000 data files to process. That kind of problem demands a high level of automation. You really need some more structure than the low-level hook provided by the swig wrapper.

-- Suppose you wanted to use Feff to compute the EXAFS spectrum from the list of atoms resulting from a molecular dynamics calculation. You'd need a way to loop through the box, find all atom pairs, and figure out how to have Feff run on each pair. Again, swig's hook is at a level that's a bit too low.

Enabling automation and solving problems like those (as well as small one-off chores, as well as GUI-writing, as well as ...) is what I am aiming for with Demeter.

Is Demeter what you are looking for? Hard to say. It's written in Perl, which might not be your cup of tea. It's still pretty far from done, so it's hard to evaluate. Most of the design specs that I am working towards are inconveniently located in my head (although that could be fixed). I've never had a collaborator that I've worked with on the same code base, so I am not in the habit of doing so. All that said, if you are interested in Demeter -- I'd be thrilled to figure out how to work together.

B

Exactly, The question is how best to utilize the existing software. For trivial bits it easier for me to just do something in python. However, for larger things like your examples I would be unsure what would be best. On the one hand it is possible to use the wrappers but I think it misses out on the good functionality that is already present in the whole suite of programs. I think Matt had it bang on. The ideal would be to be able to call the functions of ifeffit or Hephaestus etc. For example, to use an external program which may be any language. I also think it is possible but not necessarily easy. I have also been rather reluctant to re-code a lot of things. So far I have been successful by working around it. I can do something and then move the results by hand. This works for small projects but it is limiting. I am not certain what the exact answer is but I will give it some serious thought. Off the top of my head, some sort of API seems obvious but not what form this should take. (client-server, web, plug-in, ....) Out of curiosity. Would demeter be able to 'speak' TDL? Or would TDL call demeter? I am uncertain what the status is of things like pyperl or inline::python. These were supposed to allow python and perl to talk to each other. .Net/Mono is advertised to do such things but I don't know if anyone wants to go it that direction. cheers, Adam ---------------------------------------------------------- Dr. M. Adam Webb HASYLAB at DESY Notkestrasse 85 D-22607 Hamburg Germany Phone: +49 40 8998-1994, Fax: +49 40 8998-2787 E-mail: adam.webb@desy.de -----------------------------------------------------------

On Nov 20, 2007 8:13 AM, Webb, Adam wrote:

Greetings,

...

I do think an ifeffit-dev list and/or wiki area is a fine idea. I don't know how many others would be interested in this, but it might allow for more open development, and keep the ifeffit mailing list more for usage questions. Any other opinions on this?

I second the idea. A -dev list would also help avoid confusion. I can see that people might get really confused if one set of people are talking about some software in development and other people are talking about the program that people are using.

Agreed. I will set up a devel list (it may be a couple weeks). The docs for development should also be improved.

For example, a plugin like structure might be used. Someone adding a new function could for example write a plugin to do something like: 1. ask ifeffit for data and other needed information 2. do some tranform/filter/magic on data 3. return data to ifeffit Other people could use plugin by calling it from within ifeffit. You can do this now with the wrappers but it appears to me that this might be easier to do with tdl and/or demeter as these things seem better designed to do just that. Perhaps Matt or Bruce would like to comment further.

As Bruce said, with the simple interfaces to Ifeffit from C/perl/python/tcl, one can do the steps you outline, though they are perhaps at a lower level than would be desirable. One important missing item is that these interfaces only go to the base library, and cannot connect to other libraries built on top of it. While a python script can call ifeffit, it cannot call Demeter or Hephaestus. If the idea is to use the ifeffit library for core XAFS functionality, this is not a terrible approach, but it does have serious drawbacks if the goal is a more comprehensive and extensible set of tools. I think the problem is challenging but solvable. With TDL, it will be much easier to add functionality "to the core" -- as any python module can be exposed to TDL. But that doesn't solve all the problems, specifically the cases where one would want to use procedures coded in a language such as Perl or Java or Mathematica (ie, anything other than C, Fortran, or Python), which Python cannot readily use. And that, I think, may be a part of what both you and Mauro are struggling with. You'd like to use some "core functionality", but hopefully wrapped up at a higher level as Demeter is, and perhaps some other libraries (DFT, processing reflectivity data, etc) that may be implemented in a different language. At some point, we're going to have to figure out how to avoid these language conflicts. Again there are ways to solve them, but we haven't gotten there yet. I've struggled with this too. I have some of Hephaestus' capabilities in Python, but by no means all of them. I've been reluctant to re-code it all in Python, but at some point I might break down and do this. Personally, I'd love to have a web-based Hephaestus, with a documented API, so that I could get the data. --Matt