Hi Vivek, Carlo, Bruce, Everyone, On Mon, 6 Oct 2003, Carlo U. Segre wrote:

Athena does not handle that many columns gracefully. We use a Python program to split these data up into one file for each of the elements and then work from there.

Oops, I was wrong earlier. In fact, I usually use ifeffit to read files with many columns or beamline-specific software to add columns together. But the file at http://cars9.uchicago.edu/ifeffit/contrib/MED_FILES/mn_x.001 is an example of a multi-element detector file that can be read directly with ifeffit. It has 56 columns, and lines that are 830+ characters long. Ifeffit can see all columns with read_data(mn_x.001, group=mn) The file http://cars9.uchicago.edu/ifeffit/contrib/MED_FILES/x.001 is not EXAFS data, but has 73 columns, and lines with 1090+ characters. Ifeffit reads it fine. But, Carlo is right: athena does not handle that many columns. In fact, it shows "only" the first 35 in its data-column interface for these files, though the file viewer shows the full line length. I suspect that Bruce never noticed this. I don't know if that was the original problem, but I thought I'd try to clear up the confusion. --Matt

On Monday 06 October 2003 06:35 pm, Matt Newville wrote:

Oops, I was wrong earlier. In fact, I usually use ifeffit to read files with many columns or beamline-specific software to add columns together. But the file at http://cars9.uchicago.edu/ifeffit/contrib/MED_FILES/mn_x.001

<...snip...>

But, Carlo is right: athena does not handle that many columns. In fact, it shows "only" the first 35 in its data-column interface for these files, though the file viewer shows the full line length. I suspect that Bruce never noticed this.

To the contrary, I most certainly have. What I haven't noticed, up until this week, was much complaint about how Athena handles multi-column data. Except for a few special situations, Athena relies entirely upon two aspects of Ifeffit to read in data and construct the data import dialog, i.e. this thing: http://leonardo.phys.washington.edu/~ravel/software/exafs/images/athena/colu... For almost all data files, Athena uses ifeffit's read_data() function. read_data() sets a scalar called $column_label. Athena parses the contents of this scalar to determine what the columns are. The grid of columns presented in the file import dialog is construsted from this. There is no hard limit to how many columns Athena is willing to present, however it will not present any columns not reported in $column_label. Try reading the mn_x.001 file that Matt put on his web site. I concur that ifeffit reads in all the columns. What it does not do is report all the columns in the $column_label scalar. And the only reason that Athena reads as many columns as it does is because the mn_x.001 file has nice short column lables, like D1 and D17. If your column lables were, say, "my_favorite_column_label" and "my_second_favorite_column_label", the space in $column_label would be used up with far fewer columns. So, the obvious solution is for Athena to ignore the $column_label scalar (which seems not to be serving its purpose) and instead to parse the output of "show @group" for the newly read group. The other solution is for Matt to allow for a much longer string to be placed in $column_label. However, it seems to me that a bit of preprocessing, as Carlo does, is still a good idea. If the many-column data file has a well-known structure, then a few lines of specialized python (or perl or VBScript or whatever) will be an easier way to handle dozens of columns than Athena's column import dialog. This is particularly true if you need to do deadtime correction, which Athena still does not handle for you. B P.S. Of course, none of this discussion addresses the original complain, which was Athena barfing for a particular file. I would still like to see that file. -- Bruce Ravel ----------------------------------- ravel@phys.washington.edu Code 6134, Building 3, Room 222 Naval Research Laboratory phone: (1) 202 767 5947 Washington DC 20375, USA fax: (1) 202 767 1697 NRL Synchrotron Radiation Consortium (NRL-SRC) Beamlines X11a, X11b, X23b National Synchrotron Light Source Brookhaven National Laboratory, Upton, NY 11973 My homepage: http://feff.phys.washington.edu/~ravel EXAFS software: http://feff.phys.washington.edu/~ravel/software/exafs/

On Tuesday 07 October 2003 11:49 am, Matt Newville wrote:

Bruce,

Yep, the column_label is restricted to 128 characters, which could be the limiting factor for many data files and is the limitation for the files I posted.

What should be done?? text variables could be increased to 256 characters, but that might not be enough. A column lable of 2048 would seem reasonable with 100+ columns (which is not out of the question). Maybe having multiple text variables, $column_label_01, $column_label_02, etc. I can't think of an easy solution that isn't icky...

Ugh, Fortran. It's really starting to be the main problem here.

Yup, no arguments that the lack of a malloc is a major headache. I was thinking about this (among other things) during my 7 and a half hour drive up to NSLS today (traffic was awful!!!). Unless I am mistaken, the major purpose of $column_label is to have a quick way of how many columns were just read by read_data() and what their names are. That information is also reported by "show @group", but more reliably. I wouldn't discourage you from somehow solving the $column_label problem within the constraints of Fortran, but I think the real solution is for Athena (and anyone else's program that relies on read_data() to, uhh..., read the data) is to issue a call to "show @group" and parse its response. That will solve some of the problems with many-column data in Athena. It will allow Athena to actually display all the column in the column selection dialog and allow the user to import as many columns as desired or to add up as many columns as desired on the fly as the data is imported. However, two problems remain. 1. Even with Athena parsing "show @group" to construct the column selection dialog, Athena and the Athena user will still be limited by the fact the Ifeffit is written in Fortran. There are only so many (maybe less than 10) 58-column data files that you can import before using up all memory available to Ifeffit. If one wanted to process, say, 20 such files, one would probably have to use the trick of splitting them into groups, merging, then merging the merges. That not bad, but it's not ideal either. 2. Athena's column selection dialog is (in my wholly unbiased opinion ;-) very handy and easy to use for ion chamber data, i.e. for data with a small number of columns. It will not be so user friendly when you have to dig through a grid that is 58 columns wide to construct your mu(E) data. This brings me back to Carlo's comment that the user may be best served by preprocessing data before starting with Athena. In any case, I will get this fix into the next release of Athena. Between travel and an unusually high level of business at NRL, that may not happen until late October. B -- Bruce Ravel ----------------------------------- ravel@phys.washington.edu Code 6134, Building 3, Room 222 Naval Research Laboratory phone: (1) 202 767 5947 Washington DC 20375, USA fax: (1) 202 767 1697 NRL Synchrotron Radiation Consortium (NRL-SRC) Beamlines X11a, X11b, X23b National Synchrotron Light Source Brookhaven National Laboratory, Upton, NY 11973 My homepage: http://feff.phys.washington.edu/~ravel EXAFS software: http://feff.phys.washington.edu/~ravel/software/exafs/