RE: FEFF Output File Headers

["Steve Zabinsky" <stevez@microsoft.com>]

I'd like to reply to Matt's and Sven's comments.

First of all, Matt is correct in that any convention is workable as long
as it doesn't change. The current convention was designed for the
situation in place at the time (a long time ago!) and does have enough
markers in it that it can be parsed. So in some sense it is good enough.

On the other hand, changing the format to something directly useful to
the user within FEFF rather than requiring a filter to modify it would
be nice. Sven's suggestion would not be too hard to implement, but it's
not likely that every possible requirement would be discovered in the
first design attempt.

A third possibility is to add an output option that writes feff output
in either a heavily labelled text or binary format that contained all
the information but was very easy to parse. Without the constraint that
it be easy to read on a printout, this would be easy to design and
implement. A binary file would preserve full accuracy but there could be
some possible difficulty moving files across platforms due to
big-endian/little-endian conventions, so probably a text-based file
would be better although it will be larger and won't preserve all
possible significant digits.

Under UNIX, filters and pipes are designed to handle exactly the
difficulty of moving data between programs, and, although irritating, it
does work. On PCs running Windows it's a little more trouble, but using
the command prompt or Perl it can be done the same way. Another option
under Windows would be to use a third party unix shell that gives you
the same command line interface as UNIX.

So "do nothing" would be an OK solution, as would the output changes
proposed by Matt and Sven and me.

The better solution was proposed by Matt, that is, a library that can
called by the user however they wish, including the option of writing
their own output files, etc.

When we originally designed FEFF as one stand-alone FORTRAN program,
distributed as one fortran source file, portability across multiple
platforms was our single biggest concern. This approach worked well. The
constraints we had at the time were that the code was being developed as
a research project and simultaneously being used and tested by several
groups around the world. [Many thanks to those who acted as 'beta sites'
at this time. FEFF would not be the useful tool it is today without your
help and feedback!]

The situation is not the same today. FEFF is now both an ongoing
research project and also a widely used diagnostic tool. Matt's
suggestion makes a great deal of sense for FEFF in its role as a widely
used tool.

The design of "fefflib' will be tricky. I still believe that the ability
to run on multiple platforms is very important (working at Microsoft,
where our codes run on Windows and only on Windows is MUCH simpler,
especially for GUI and network programming) and also difficult. There
are a lot of technologies designed to handle this: COM for Windows,
equivalent sharable objects for other operating systems, command line
programs tied together with perl, etc., but most are platform specific.

For portability, breaking FEFF into stand-alone fortran codes and
letting the user tie these pieces together with perl is probably best.
This could also be done with the current version of FEFF without
modification. I do not know of a portable executable format that will
run on the many UNIX platforms, and it is even harder to find a solution
that will run on the many UNIX platforms and also on (cheap and common)
Windows based PCs.

An obvious option is to give up on portability. If we assume that most
users are running problems of a scale that can be handled by a modern PC
(say a 1Ghz process with 526Mbytes of RAM), we could design a library
that ran under Window (requiring users to have a PC that ran WinXP)
using COM or some other Windows specific interface utility. This way,
the code would be available from all Microsoft languages (Visual Basic,
VC++, etc). COM is complex to use, and I suspect that most users would
not be willing to learn it.

Or, we could design a library that ran under Linux (requiring users to
have a PC that ran some specific verison of Linux) using the programming
interface utilities available there and making the library available to
all Linux coding tools. Similar to the Windows solution, it will take a
fairly expert UNIX person to use this interface.

This will be a big project, but the user-base may justify the expense.

My own prejudice is toward the original design [not surprisingly since I
designed it :)], one FORTRAN progam or a set of FORTRAN programs that
can be called in a flexible manner. The learning curve for perl is no
worse than for accessing a COM based interface to Visual Basic, or for
accessing the similar UNIX tools. Parsing the FEFF output can be done
directly in perl, or can be done using another language with the filter
being called by the perl script. (There are other scripting languages
around for those who don't like perl.)

(I don't know how the FORTRAN / FORTRAN90 / C / C++ / Java debate is
shaping up in the physics or XAFS communities. If the concensus has
moved away from FORTRAN, clearly FEFF should be released in whatever
language is most commonly in use.)



-----Original Message-----
From: Matt Newville [mailto:newville@cars.uchicago.edu] 
Sent: Friday, January 11, 2002 8:08 AM
To: FEFF Software Development
Subject: Re: FEFF Output File Headers


Hi All,

In principle, any convention should be OK and a convention does already
exist.  And unless something can be done about updating previous
versions, analysis programs will have to accept the old format for a
long time.  Still, I vote against 'do nothing'.

I find the '@#' convention hard to deal with.  First, it comes at the
end of the comment lines, signaling that all the lines read so far were
comments, even if there were lines of numbers-only columns seen (as some
xmu.dat files do). Second, it means looking at the end of line instead
of the beginning, which can be tricky when files move between Unix,
Windows, and Mac.  I don't know of any plotting programs (gnuplot,
Kaleidagraph, IGOR, SigmaPlot, Origin, or Excel) that has anything like
the '@#' convention.

Putting # (or %, ;, !, ...)  in column 1 of output files should be easy.
Any program that already uses feff.dat could easily be modified to
ignore column 1, I think.  If the license changes, I'll donate fortran
and C routines that reads these feff.dat files with or without these
characters in column 1.

Having other options to tailor the output files for plotting program
could be nice.  That's more work, but if the i/o were well isolated it
would be easy to add additional output formats.  For example, some
programs (Excel, Kaleidagraph) seem to work best when a predictable
number of lines are 'headers to ignore'.  So letting the user specify
the exact number of comment lines (including none) would be a nice
option, and easy to do, assuming Feff had well isolated i/o.

Beyond that, Feff should be converted into a library of 'high level
functions' that can be used within customized applications. This would
allow Feff calculations to be done directly by analysis programs that
already have plotting capabilities, or used as parts of Excel macros, VB
script, Perl or Hypercard programs, etc.  In that scenario, Feff doesn't
need fancy i/o and can let the calling program customize i/o (column
labels in French, say), or provide a simple routine to 'write a regular
xmu.dat file'.

Again, my view is that putting '#' or something else in column 1 and
simplifying the column labels is worthwhile, but I do not have a strong
preference for which details are adopted.  Whatever the convention is
(including the present), it should be documented, and routines to read
the files should be provided.  More than that, and Feff should be
converted from a a monolithic program that attempts to satisfy everyone
into a library of tools.

--Matt

|= Matthew Newville        mailto:newville@cars.uchicago.edu
|= GSECARS, Bldg 434A      voice: (630) 252-0431 / 1713
|= Argonne Natl Lab        fax:   (630) 252-0443
|= 9700 South Cass Ave     http://cars9.uchicago.edu/~newville/
|= Argonne, IL 60439 USA