On 07/04/2014 12:11 PM, Margaret Hinkle wrote:

Thanks for your help & your quick reply. I did not expect my altering one character in the code to fix the problem -- I had just hoped it would tell me if the Tip of the Day was indeed the culprit. I did as you suggested, and the problem is completely fixed. Thanks!

Per your comment regarding Issue 2 -- I have attached two files, one with the header removed (filename: '...NoHeader_exafsscan.dat'), and the other with the header left in with #s commenting it out (filename: '...Header_exafsscan.dat'). Note that these are transmission samples, with I0 in column 56, and I1 in column 57.

Margaret Anne, By the end of this email, I am going to thoroughly solve your problem. But, first, I am going to step up on my soapbox and pontificate. It's kinda what I do on this mailing list.... You started this by writing "I can only load data files into Athena if there are no headers of any sort". That is a demonstrably untrue statement, and easily so. Were it true, this mailing list would be either (a) filled with nothing but posts about people complaining that they cannot import their data, or (b) silent because no one would use software so completely fucked up and useless. A true statement might have been that you had examples of data files that could not be imported into Athena. But you did not include an example of the troublesome data file -- I had to ask for it. When you ask a question, you should try to put yourself in the shoes of the person answering. I am not in the same room as you, I am not looking at your computer, and I do not have the example that triggers the problem here in front of me. And I do not possess telepathy. When you ask a question, you have to give the person who wants to help a fighting chance of being able to help. All right. Enough of that. Let's solve your problem. Find the installation location of Demeter on your computer. If you do not know where that is, then open a command terminal and enter the following command: perl -e 'use Demeter; print $INC{"Demeter.pm"}, $/' The thing that gets printed is the folder containing the main Demeter module file. Everything else is beneath that folder. Download this file: https://s3.amazonaws.com/demeter4xas/SpecFileLongLine.pm and save it as Demeter/Plugins/SpecFileLongLine.pm where "Demeter/" is a folder inside the folder we identified above. Make sure that the file is called "SpecFileLongLines.pm" -- exactly as written there -- and that your web browser does not "help you out" by changing the extension of the file to ".txt" or any other such shenanigans. Now fire up Athena and select "Plugin Registry" from the main menu. This will display the view shown here: http://bruceravel.github.io/demeter/aug/other/plugin.html#athena%27spluginre... There should be an row that says "SpecFileLongLine". Click it on. You should now be able to import your files without needing to edit the header. Note that if you attempt to import one of your data files *without* enabling the "SpecFileLongLine" plugin, Athena will continue to crash. The mere presence of the SpecFileLongLine.pm file is not enough -- you *must* enable in the plugin registry. So, what was the problem? Well, Ifeffit -- which is the math and XAS engine underneath the hood of Athena -- is written in Fortran. Fortran does not have dynamic memory allocation. Thus there are some hardwired limits to things in Ifeffit, not all of which get handled gracefully when exceeded. The line in your data file causing the problem is the one that starts with "#L". It contains the column labels and is about 330 characters long. As one point, Ifeffit attempts to stuff those 330 characters in a variable configured to be 256 characters long. That is the cause and location of the error message you cited. The plugin does the simplest possible thing. It makes a copy of the data file, copying every line except the one beginning with #L. Athena then reads the copy. The original data is untouched, the copy is deleted once Athena reads it in. Athena is happy and so, I hope, are you! Let me know if you have any problems with the plugin. It worked for me on my linux computer. I suspect it'll work for you as well, but one never knows.... B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973 Homepage: http://xafs.org/BruceRavel Software: https://github.com/bruceravel

Hi, On Sat, Jul 5, 2014 at 2:06 PM, Bruce Ravel wrote:

On 07/04/2014 12:11 PM, Margaret Hinkle wrote:

...

Thanks for your help & your quick reply. I did not expect my altering one character in the code to fix the problem -- I had just hoped it would tell me if the Tip of the Day was indeed the culprit. I did as you suggested, and the problem is completely fixed. Thanks!

Per your comment regarding Issue 2 -- I have attached two files, one with the header removed (filename: '...NoHeader_exafsscan.dat'), and the other with the header left in with #s commenting it out (filename: '...Header_exafsscan.dat'). Note that these are transmission samples, with I0 in column 56, and I1 in column 57.

Margaret Anne,

By the end of this email, I am going to thoroughly solve your problem. But, first, I am going to step up on my soapbox and pontificate. It's kinda what I do on this mailing list....

You started this by writing "I can only load data files into Athena if there are no headers of any sort". That is a demonstrably untrue statement, and easily so. Were it true, this mailing list would be either (a) filled with nothing but posts about people complaining that they cannot import their data, or (b) silent because no one would use software so completely fucked up and useless.

A true statement might have been that you had examples of data files that could not be imported into Athena. But you did not include an example of the troublesome data file -- I had to ask for it.

When you ask a question, you should try to put yourself in the shoes of the person answering. I am not in the same room as you, I am not looking at your computer, and I do not have the example that triggers the problem here in front of me. And I do not possess telepathy. When you ask a question, you have to give the person who wants to help a fighting chance of being able to help.

Grouchy? Didn't Margaret Anne post an example file when asked? I believe she also tried to solve the Splash screen issue herself, and explained what she tried in pretty good detail... It may not have worked as well as she'd hoped but she definitely tried something plausible (admirable) and explained what she did (even more so). I would call that an exemplary report, myself.

All right. Enough of that. Let's solve your problem.

Find the installation location of Demeter on your computer. If you do not

know where that is, then open a command terminal and enter the following command:

perl -e 'use Demeter; print $INC{"Demeter.pm"}, $/'

The thing that gets printed is the folder containing the main Demeter module file. Everything else is beneath that folder.

Download this file:

https://s3.amazonaws.com/demeter4xas/SpecFileLongLine.pm

and save it as

Demeter/Plugins/SpecFileLongLine.pm

where "Demeter/" is a folder inside the folder we identified above.

Make sure that the file is called "SpecFileLongLines.pm" -- exactly as written there -- and that your web browser does not "help you out" by changing the extension of the file to ".txt" or any other such shenanigans.

Now fire up Athena and select "Plugin Registry" from the main menu. This will display the view shown here:

http://bruceravel.github.io/demeter/aug/other/plugin.html# athena%27spluginregistry

There should be an row that says "SpecFileLongLine". Click it on.

You should now be able to import your files without needing to edit the header.

Note that if you attempt to import one of your data files *without* enabling the "SpecFileLongLine" plugin, Athena will continue to crash. The mere presence of the SpecFileLongLine.pm file is not enough -- you *must* enable in the plugin registry.

So, what was the problem?

Well, Ifeffit -- which is the math and XAS engine underneath the hood of Athena -- is written in Fortran. Fortran does not have dynamic memory allocation. Thus there are some hardwired limits to things in Ifeffit, not all of which get handled gracefully when exceeded.

The line in your data file causing the problem is the one that starts with "#L". It contains the column labels and is about 330 characters long. As one point, Ifeffit attempts to stuff those 330 characters in a variable configured to be 256 characters long. That is the cause and location of the error message you cited.

Wait, what? What text string is configured to be only 256 characters? Something must be very wrong with the Macports installation - this file opens just fine for me, though I haven't tested it with the Macports version. The "read_data()" command in ifeffit uses a line length of 2048 characters, and should not crash on a longer line, but truncate lines to 2048 characters (silently, which is irritating when encountered, but not as bad as crashing). I've written and used thousands of data files with comment and data lines longer than 256 characters. I believe there is (and should be) nothing in the code for reading data that sets a limit as low as 256 characters (File names are allowed to be 512 characters!), and nothing that should cause a crash on larger lines. Of course it's fine to use a plugin for specific file types, and probably wise to not rely of ifeffit's read_data() for really big files (though, again, I regularly read files of 35 to 40 columns without a plugin). But this seems to me like its a Macports-only problem to me that should be fixed. --Matt

On Sun, Jul 6, 2014 at 5:24 PM, Bruce Ravel wrote:

Matt,

You're right. I was too grouchy. Margaret Anne, I apologize if I was too mean-spirited in my emails over the weekend.

On 07/06/2014 05:54 PM, Matt Newville wrote:

...

Wait, what? What text string is configured to be only 256 characters? Something must be very wrong with the Macports installation - this file opens just fine for me, though I haven't tested it with the Macports version.

The "read_data()" command in ifeffit uses a line length of 2048 characters, and should not crash on a longer line, but truncate lines to 2048 characters (silently, which is irritating when encountered, but not as bad as crashing). I've written and used thousands of data files with comment and data lines longer than 256 characters. I believe there is (and should be) nothing in the code for reading data that sets a limit as low as 256 characters (File names are allowed to be 512 characters!), and nothing that should cause a crash on larger lines.

Of course it's fine to use a plugin for specific file types, and probably wise to not rely of ifeffit's read_data() for really big files (though, again, I regularly read files of 35 to 40 columns without a plugin). But this seems to me like its a Macports-only problem to me that should be fixed.

I guess I explained it poorly, but it is not a MacPorts-only problem.

It /opens/ just fine. That's not the problem.

It's a problem with iff_show, which Athena uses to fetch the names of the column labels after ifeffit reads the data file. Here's what happens if you exercise iff_show by hand:

bruce@atropos [demeter] > ifeffit Ifeffit 1.2.11d Copyright (c) 2008 Matt Newville, Univ of Chicago command-line shell version 1.1 with GNU Readline Ifeffit> read_data (file="/home/bruce/Mn0214_7dMnO2_Header_exafsscan1.dat", group=a) Ifeffit> show @strings $plot_devices = /png /tpng /null /ps /vps /cps /vcps /xwindow /xserve $plot_device = /xserve $group = a $commentchar = # $&build = 1.2.11d Copyright (c) 2008 Matt Newville, Univ of Chicago $&install_dir = /usr/local/share/ifeffit $column_label = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 $column_label1 = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 $a_title_01 = S 1 exafsscan $a_title_02 = C exafs_edge_energy 6.539 $a_title_03 = C exafs_region -150 -20 26 1 0 0 0 $a_title_04 = C exafs_region -20 34.1959 108 1 0 0 0 $a_title_05 = C exafs_region 34.1959 972.683 260 1 0 1 1.4 $a_title_06 = D Wed Feb 26 22:19:02 2014 $a_title_07 = T 4.58834 (Seconds) $a_title_08 = G0 0 $a_title_09 = G1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 $a_title_10 = G3 0 0 0 0 0 0 0 0 0 $a_title_11 = G4 0 $a_title_12 = Q $a_title_13 = P0 17.707645 -0.3425 33.862069 -0.80317213 -18.366667 0.175013 -0.175013 8.99922 $a_title_14 = P1 8.49884 8.74903 -5.0037582 8.74903 1.7763568e-14 0 3.5002457 9.4148 $a_title_15 = P2 -44.9992 -23.95 -30.75 97.52 19.252 -11.58 8.4175 -2.25125 $a_title_16 = P3 -5.11125 1.4875 0.7325 1.5 85 0 -600 $a_title_17 = N 57

At line 436 of file iff_show.f Fortran runtime error: End of record bruce@atropos [demeter] >

Line 436 of iff_show is

write(messg,11) s(1:k), ' = ', t(1:j)

messg is defined just above that as character*256. I interpreted this as attempting to put a 330 character string in a character*256 variable. But, in truth, I did not examine the code all that closely.

I suppose I could have worked on fixing Ifeffit, but the plugin took way less time. And my time would be better spent making Demeter use Larch properly.

Or perhaps upgraded (see https://github.com/newville/ifeffit/commit/0b3512552dd08a85d4d69cbd32b4e525e... )? One of the problems of using a deprecated and poorly supported library, I suppose -- the Fortran code does have many shortcomings, but such crashes should definitely be addressed. The MacPorts (and other versions) may want to build with the 1.2.12 source code from github -- this isn't the only problem fixed in the past 6 years. --Matt

"It's kinda what I do on this mailing list." And I think I'm unsubscribing for that reason (although, like many others undoubtedly, I am probably professionally condemned to return eventually). In ten years of helping countless users as an expert on WIEN2k resp. FEFF, and encountering all the frustrations that come with mailing lists and novice users, I have not seen people treated the way they sometimes are on this mailing list. It goes against my values around helping users; it makes me uncomfortable; and I don't want to see it anymore. Much to your credit, you proceeded to type also several long and detailed paragraphs addressing the issue, which were helpful to the user. I would be delighted if you were willing to protect your time and energy (a valid need) in a friendlier way. Best regards, Kevin Jorissen PS still reachable through www.feffproject.org or at by e-mail for FEFF-related questions. On Sat, Jul 5, 2014 at 9:06 PM, Bruce Ravel wrote:

On 07/04/2014 12:11 PM, Margaret Hinkle wrote:

...

Thanks for your help & your quick reply. I did not expect my altering one character in the code to fix the problem -- I had just hoped it would tell me if the Tip of the Day was indeed the culprit. I did as you suggested, and the problem is completely fixed. Thanks!

Per your comment regarding Issue 2 -- I have attached two files, one with the header removed (filename: '...NoHeader_exafsscan.dat'), and the other with the header left in with #s commenting it out (filename: '...Header_exafsscan.dat'). Note that these are transmission samples, with I0 in column 56, and I1 in column 57.

Margaret Anne,

By the end of this email, I am going to thoroughly solve your problem. But, first, I am going to step up on my soapbox and pontificate. It's kinda what I do on this mailing list....

You started this by writing "I can only load data files into Athena if there are no headers of any sort". That is a demonstrably untrue statement, and easily so. Were it true, this mailing list would be either (a) filled with nothing but posts about people complaining that they cannot import their data, or (b) silent because no one would use software so completely fucked up and useless.

A true statement might have been that you had examples of data files that could not be imported into Athena. But you did not include an example of the troublesome data file -- I had to ask for it.

When you ask a question, you should try to put yourself in the shoes of the person answering. I am not in the same room as you, I am not looking at your computer, and I do not have the example that triggers the problem here in front of me. And I do not possess telepathy. When you ask a question, you have to give the person who wants to help a fighting chance of being able to help.

All right. Enough of that. Let's solve your problem.

Find the installation location of Demeter on your computer. If you do not know where that is, then open a command terminal and enter the following command:

perl -e 'use Demeter; print $INC{"Demeter.pm"}, $/'

The thing that gets printed is the folder containing the main Demeter module file. Everything else is beneath that folder.

Download this file:

https://s3.amazonaws.com/demeter4xas/SpecFileLongLine.pm

and save it as

Demeter/Plugins/SpecFileLongLine.pm

where "Demeter/" is a folder inside the folder we identified above.

Make sure that the file is called "SpecFileLongLines.pm" -- exactly as written there -- and that your web browser does not "help you out" by changing the extension of the file to ".txt" or any other such shenanigans.

Now fire up Athena and select "Plugin Registry" from the main menu. This will display the view shown here:

http://bruceravel.github.io/demeter/aug/other/plugin.html# athena%27spluginregistry

There should be an row that says "SpecFileLongLine". Click it on.

You should now be able to import your files without needing to edit the header.

Note that if you attempt to import one of your data files *without* enabling the "SpecFileLongLine" plugin, Athena will continue to crash. The mere presence of the SpecFileLongLine.pm file is not enough -- you *must* enable in the plugin registry.

So, what was the problem?

Well, Ifeffit -- which is the math and XAS engine underneath the hood of Athena -- is written in Fortran. Fortran does not have dynamic memory allocation. Thus there are some hardwired limits to things in Ifeffit, not all of which get handled gracefully when exceeded.

The line in your data file causing the problem is the one that starts with "#L". It contains the column labels and is about 330 characters long. As one point, Ifeffit attempts to stuff those 330 characters in a variable configured to be 256 characters long. That is the cause and location of the error message you cited.

The plugin does the simplest possible thing. It makes a copy of the data file, copying every line except the one beginning with #L. Athena then reads the copy. The original data is untouched, the copy is deleted once Athena reads it in. Athena is happy and so, I hope, are you!

Let me know if you have any problems with the plugin. It worked for me on my linux computer. I suspect it'll work for you as well, but one never knows....

B

-- Bruce Ravel ------------------------------------ bravel@bnl.gov

National Institute of Standards and Technology Synchrotron Science Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973

Homepage: http://xafs.org/BruceRavel Software: https://github.com/bruceravel _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit

HI Scott, On Sun, Jul 6, 2014 at 6:48 PM, Scott Calvin wrote:

Hi all,

This is probably as good a time as any to mention a miscommunication I've seen come up a couple of times on the list that may have caused unnecessary friction. It's a good time because it =isn't= what transpired in the current thread (which ended happily for all involved), so it hopefully won't get bogged down in particulars.

Sometimes I've seen someone post to the list that they've been having an intermittent, difficult to diagnose, or otherwise mysterious problem, asking if anyone else had seen something similar. That makes for a lousy bug report, but is, in my opinion, a reasonable and rational use of a list like this.

As a hypothetical, something like "is the latest distributed version of Hephaestus working for other people using MacPorts?" is a useful diagnostic question, and a terrible bug report. It does not require a response of any kind from Bruce, Matt, etc., and they should not feel compelled to give one. "Works for me" or "No, Hephaestus crashes" would be useful responses from other members of the community.

To put it another way, I think this list straddles the boundary between being part of the reporting system for software and a community board. It's important to recognize, I think, when each function is being utilized.

That's not to say that there aren't lousy bug reports that get posted here--there are. When someone says "Athena crashes every time I open it--please fix this bug," they are not being useful to anyone. But I am saying that not every report of problems is actually a bug report.

That said, I think this is a good list. Everyone who contributes to it is a volunteer, and there are interesting and helpful discussions that take place at a wide range of levels.

Mailing lists have the great advantage of being universally available, and of having an understood and relatively slow pace. That is, it always acceptable to wait a full day before responding to any other message, no exceptions. A corollary is that people familiar with such mailing lists should understand that it is NOT chatting, and that questions should be *posed* as opposed to carelessly thrown together. Of course, there will be vague and poorly-worded questions. There are better forums for actual bug reports, but lots of questions here are almost-but-not-quite-bug-reports. I think these should be fine on the mailing list. It's also completely OK to respond with "well I can't possibly answer your question without more details". Endlessly harping on people to "ask good questions" seems (empirically) to not be very fruitful. For me, I think the main issue is that the process is extremely asymmetric. If only Bruce (or me, or Sam, or only some one person) can answer broad categories of questions, that is inherently unfair. There are many questions asked on this list that that can be (and often are) answered by many people, say, general XAFS questions or about analysis strategies. These may get contentious, but not because of an inherent asymmetry. I think the best solution is to work on removing the asymmetry. It is just not sustainable for hundreds of people to rely on a large software base that is only developed and maintained by an individual. It is not only unfair to expect it, it is also a really bad strategy that you are taking -- "I know, I'll just rely on my research results on this software written by this one person". It's actually kind of insane. I don't know of an easy solution to this. Bruce and I have each asked for help many times, and both of us have re-designed software specifically to make it easier to have people contribute. The response has been poor. I won't speak for Bruce, but you have to understand that I view the problem as entirely yours, not mine. Why has this asymmetry persisted so long? P.S. If we're looking for ways to improve the list in the future, I'd

suggest considering the possibility of a more modern forum format. I am sure there are open-source platforms for such a thing. It would be wonderful to have an upvote type feature (Facebook's "like" is the most familiar example), more user-friendly threading, and the ability to delete and edit posts.

I'm familiar StackExchange -- is something like that what you have in mind? I could be persuaded that such a thing could be useful. Perhaps Kevin would find such a forum less offensive, which would be nice.

This seems like a great way for someone to contribute. --Matt