Unable to export flattened normalized XANES spectra from Athena 0.9.24
I am looking for some help with a possible error on my part or bug associated with exporting normalized mu(E) spectra from Athena 0.9.24 64-bit on Windows 10 Pro 64-bit. I have some Si XANES data with a challenging background shape that is well removed through careful normalization in Athena, as long as the "Flatten normalized data" option is checked. When I save the current group as norm(E), what is saved is the non-flattened normalized data. The checkbox for flattening the data has no effect on what is exported despite changing what is plotted. I have verified this by comparing the files exported the the flatten box checked or unchecked. I have tried this with multiple data files loaded in multiple formats (raw beamline data, simple two-column E and mu data, PRJ files) and the effect is always the same. It is also unaffected by the normalization ranges I select. No matter what options I select Athena 0.9.24 will only save non-flattened norm(E) data. This behavior is different from what I remember, and I have verified that in version 0.8 of Athena the exported norm(E) was the flattened data, provided that the flatten box was checked (of course). I have been unable to determine if I have any errors in the settings in version 0.9.24 but all parameters that I can find are identical in both 0.9.24 and 0.8. I have also uninstalled and reinstalled Athena 0.9.24 and the effect is still the same. Has anyone else noticed this problem? Is this a bug? Thank you, Jeff -- Jeffrey G. Catalano, Associate Professor Department of Earth and Planetary Sciences Washington University 1 Brookings Drive, Campus Box 1169 Saint Louis, MO 63130 USA http://aqgeochem.wustl.edu/
Here's what the manual has to say http://bruceravel.github.io/demeter/aug/output/column.html Looking at the code, however, it seems the document has swapped columns 2/3 and columns 4/5. I think what you are looking for is in column 4, not column 2. If that's right, I'll treat this as a documentation bug. If that is not the case, then I'll prod you for more information so I can understand the software bug. In either case, thanks for the report. Cheers, B On 12/08/2015 06:09 PM, Jeff Catalano wrote:
I am looking for some help with a possible error on my part or bug associated with exporting normalized mu(E) spectra from Athena 0.9.24 64-bit on Windows 10 Pro 64-bit. I have some Si XANES data with a challenging background shape that is well removed through careful normalization in Athena, as long as the "Flatten normalized data" option is checked. When I save the current group as norm(E), what is saved is the non-flattened normalized data. The checkbox for flattening the data has no effect on what is exported despite changing what is plotted. I have verified this by comparing the files exported the the flatten box checked or unchecked. I have tried this with multiple data files loaded in multiple formats (raw beamline data, simple two-column E and mu data, PRJ files) and the effect is always the same. It is also unaffected by the normalization ranges I select. No matter what options I select Athena 0.9.24 will only save non-flattened norm(E) data.
This behavior is different from what I remember, and I have verified that in version 0.8 of Athena the exported norm(E) was the flattened data, provided that the flatten box was checked (of course). I have been unable to determine if I have any errors in the settings in version 0.9.24 but all parameters that I can find are identical in both 0.9.24 and 0.8. I have also uninstalled and reinstalled Athena 0.9.24 and the effect is still the same. Has anyone else noticed this problem? Is this a bug?
Thank you, Jeff
-- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/
Hi Bruce, I appreciate the reply. It looks like what I want is the flattend column of data, which is not column 2 in the version 0.9.24 norm(E) files, it is column 3. I was unaware both flattened and unflatten normalized data were exported in the file. In Athena 0.8 if "Flatten normalized data" was selected then column 2 was this flattened data in a norm(E) file. I had no idea that the export format for norm(E) files changed in version 0.9. To make sure I have this correct, below are the column labels from norm(E) files exported from Athena 0.8.061 and Athena 0.9.24: 0.8.061: # energy norm bkg_norm der_norm 0.9.24: # e norm flat fbkg nder nsec I have plotting code that expects columns 1 and 2 and I was unaware that flattened data is now output separately (in column 3). In version 0.8 a flattened normalized spectra was outputted if the box was checked, and if it was unchecked then column 2 was the unflattened data, from what I can tell. Can you please confirm that I understand this correctly? I admit that I had not looked at the documentation before writing since I understood the Athena 0.8 file format and had not anticipated a format change in the new version. It clearly would have been informative to look at the current documentation! Of course, as you note that the columns listed in the documentation for norm(E) files are wrong, with the current output format being columns 1 4 2 3 6 7 from what is listed in the documentation (column 5 does not appear in my file). Should this be looked at? I actually would prefer the column order that is listed in the documentation. The file format is clearly not what you intended. Thank you, Jeff On 12/8/2015 9:15 PM, Bruce Ravel wrote:
Here's what the manual has to say
http://bruceravel.github.io/demeter/aug/output/column.html
Looking at the code, however, it seems the document has swapped columns 2/3 and columns 4/5. I think what you are looking for is in column 4, not column 2. If that's right, I'll treat this as a documentation bug. If that is not the case, then I'll prod you for more information so I can understand the software bug.
In either case, thanks for the report.
Cheers, B
On 12/08/2015 06:09 PM, Jeff Catalano wrote:
I am looking for some help with a possible error on my part or bug associated with exporting normalized mu(E) spectra from Athena 0.9.24 64-bit on Windows 10 Pro 64-bit. I have some Si XANES data with a challenging background shape that is well removed through careful normalization in Athena, as long as the "Flatten normalized data" option is checked. When I save the current group as norm(E), what is saved is the non-flattened normalized data. The checkbox for flattening the data has no effect on what is exported despite changing what is plotted. I have verified this by comparing the files exported the the flatten box checked or unchecked. I have tried this with multiple data files loaded in multiple formats (raw beamline data, simple two-column E and mu data, PRJ files) and the effect is always the same. It is also unaffected by the normalization ranges I select. No matter what options I select Athena 0.9.24 will only save non-flattened norm(E) data.
This behavior is different from what I remember, and I have verified that in version 0.8 of Athena the exported norm(E) was the flattened data, provided that the flatten box was checked (of course). I have been unable to determine if I have any errors in the settings in version 0.9.24 but all parameters that I can find are identical in both 0.9.24 and 0.8. I have also uninstalled and reinstalled Athena 0.9.24 and the effect is still the same. Has anyone else noticed this problem? Is this a bug?
Thank you, Jeff
-- Jeffrey G. Catalano, Associate Professor Department of Earth and Planetary Sciences Washington University 1 Brookings Drive, Campus Box 1169 Saint Louis, MO 63130 USA http://aqgeochem.wustl.edu/
On 12/09/2015 09:24 AM, Jeff Catalano wrote:
Hi Bruce,
I appreciate the reply. It looks like what I want is the flattend column of data, which is not column 2 in the version 0.9.24 norm(E) files, it is column 3. I was unaware both flattened and unflatten normalized data were exported in the file. In Athena 0.8 if "Flatten normalized data" was selected then column 2 was this flattened data in a norm(E) file. I had no idea that the export format for norm(E) files changed in version 0.9. To make sure I have this correct, below are the column labels from norm(E) files exported from Athena 0.8.061 and Athena 0.9.24:
You are correct. Some of the details about the program in its current form are different from the version of Athena for which development stopped over 8 years ago. It, of course, helps when the documentation is correct. So I fixed that this morning. The norm(E) output file is now described correctly at http://bruceravel.github.io/demeter/aug/output/column.html
0.8.061: # energy norm bkg_norm der_norm 0.9.24: # e norm flat fbkg nder nsec
That's surprising. The third column should be the normalized background function. I don't understand how the "nbkg" column is not written to the file. That might suggest some other bug or some problem with your data. I'd need to see a project file to understand how and why that's happening. Possibly the original data as well.
I have plotting code that expects columns 1 and 2 and I was unaware that flattened data is now output separately (in column 3). In version 0.8 a flattened normalized spectra was outputted if the box was checked, and if it was unchecked then column 2 was the unflattened data, from what I can tell. Can you please confirm that I understand this correctly?
I don't see how supporting your "plotting code" is my responsibility. The program works the way it works. Aside from the misstatement in the documentation which you pointed out -- and I am very grateful for that -- the document does a fair job of explaining the program.
I admit that I had not looked at the documentation before writing since I understood the Athena 0.8 file format and had not anticipated a format change in the new version. It clearly would have been informative to look at the current documentation!
<cough><cough> .....
Of course, as you note that the columns listed in the documentation for norm(E) files are wrong, with the current output format being columns 1 4 2 3 6 7 from what is listed in the documentation (column 5 does not appear in my file). Should this be looked at? I actually would prefer the column order that is listed in the documentation. The file format is clearly not what you intended.
As I said above, I would need an actionable bug report to understand why the "nbkg" column is missing in your case. That's a behavior that I've not seen before. Some hints on how to make an actionable bug report: http://bruceravel.github.com/demeter/pods/bugs.pod.html B
Thank you, Jeff
On 12/8/2015 9:15 PM, Bruce Ravel wrote:
Here's what the manual has to say
http://bruceravel.github.io/demeter/aug/output/column.html
Looking at the code, however, it seems the document has swapped columns 2/3 and columns 4/5. I think what you are looking for is in column 4, not column 2. If that's right, I'll treat this as a documentation bug. If that is not the case, then I'll prod you for more information so I can understand the software bug.
In either case, thanks for the report.
Cheers, B
On 12/08/2015 06:09 PM, Jeff Catalano wrote:
I am looking for some help with a possible error on my part or bug associated with exporting normalized mu(E) spectra from Athena 0.9.24 64-bit on Windows 10 Pro 64-bit. I have some Si XANES data with a challenging background shape that is well removed through careful normalization in Athena, as long as the "Flatten normalized data" option is checked. When I save the current group as norm(E), what is saved is the non-flattened normalized data. The checkbox for flattening the data has no effect on what is exported despite changing what is plotted. I have verified this by comparing the files exported the the flatten box checked or unchecked. I have tried this with multiple data files loaded in multiple formats (raw beamline data, simple two-column E and mu data, PRJ files) and the effect is always the same. It is also unaffected by the normalization ranges I select. No matter what options I select Athena 0.9.24 will only save non-flattened norm(E) data.
This behavior is different from what I remember, and I have verified that in version 0.8 of Athena the exported norm(E) was the flattened data, provided that the flatten box was checked (of course). I have been unable to determine if I have any errors in the settings in version 0.9.24 but all parameters that I can find are identical in both 0.9.24 and 0.8. I have also uninstalled and reinstalled Athena 0.9.24 and the effect is still the same. Has anyone else noticed this problem? Is this a bug?
Thank you, Jeff
-- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/
Hi Bruce, I realize that you stopped development on the prior version a long time ago, but I only made the switch to version 0.9 a couple of years ago, and thus was a long-time user of prior versions. I don't think users expect a file format [e.g., norm(E)] to change over time. I note this to make clear why I did not consider that the same data could be in a different column when saving norm(E) files in version 0.8 and 0.9. Users certainly should read documentation to understand how new features work (like multi-electron excitation removal), but if we have used a program for many years and are exporting the same file type as we always have then it is not instinctive to expect the format to have changed. You misunderstood my comment about my plotting code. I never asked you to support how I plot data using saved files. I only noted this because it is how I discovered the problem I was having. I will plot column 4 from now on (more on this below). I think I have identified the sequence of events to produce a missing nbkg column. The attached text file (cubes1_test.txt) is a simple two column energy vs. mu(E) data with no header. It is for Si XANES data collected at the CLS, and the Si content of this samples was low, hence the noise and poor pre-edge region. When reading this text data into Athena (1st column E, 2nd column mu), Athena sets this to datatype: XANES. See the attached Athena file, I did not change the default normalization settings for this example to keep things simple. If I export norm(E) from this Athena project (see attached cubes1_test_initial.nor), nbkg is missing. If I convert the data to mu(E) via Group > Change Data Type then the nbkg column appears (see attached cubes1_test_muE.nor). Changing data type back to XANES does not remove nbkg, it is still there (not attached, file looks the same as cubes1_test_initial.nor). The missing column seems to be associated with the importing process for the short energy range involved in my data. I appreciate you taking the time to look into this. As a long-time user of your software I am grateful for access to such a useful tool for data processing and analysis. Jeff On 12/9/2015 8:46 AM, Bruce Ravel wrote:
On 12/09/2015 09:24 AM, Jeff Catalano wrote:
Hi Bruce,
I appreciate the reply. It looks like what I want is the flattend column of data, which is not column 2 in the version 0.9.24 norm(E) files, it is column 3. I was unaware both flattened and unflatten normalized data were exported in the file. In Athena 0.8 if "Flatten normalized data" was selected then column 2 was this flattened data in a norm(E) file. I had no idea that the export format for norm(E) files changed in version 0.9. To make sure I have this correct, below are the column labels from norm(E) files exported from Athena 0.8.061 and Athena 0.9.24:
You are correct. Some of the details about the program in its current form are different from the version of Athena for which development stopped over 8 years ago.
It, of course, helps when the documentation is correct. So I fixed that this morning. The norm(E) output file is now described correctly at http://bruceravel.github.io/demeter/aug/output/column.html
0.8.061: # energy norm bkg_norm der_norm 0.9.24: # e norm flat fbkg nder nsec
That's surprising. The third column should be the normalized background function. I don't understand how the "nbkg" column is not written to the file. That might suggest some other bug or some problem with your data.
I'd need to see a project file to understand how and why that's happening. Possibly the original data as well.
I have plotting code that expects columns 1 and 2 and I was unaware that flattened data is now output separately (in column 3). In version 0.8 a flattened normalized spectra was outputted if the box was checked, and if it was unchecked then column 2 was the unflattened data, from what I can tell. Can you please confirm that I understand this correctly?
I don't see how supporting your "plotting code" is my responsibility. The program works the way it works. Aside from the misstatement in the documentation which you pointed out -- and I am very grateful for that -- the document does a fair job of explaining the program.
I admit that I had not looked at the documentation before writing since I understood the Athena 0.8 file format and had not anticipated a format change in the new version. It clearly would have been informative to look at the current documentation!
<cough><cough> .....
Of course, as you note that the columns listed in the documentation for norm(E) files are wrong, with the current output format being columns 1 4 2 3 6 7 from what is listed in the documentation (column 5 does not appear in my file). Should this be looked at? I actually would prefer the column order that is listed in the documentation. The file format is clearly not what you intended.
As I said above, I would need an actionable bug report to understand why the "nbkg" column is missing in your case. That's a behavior that I've not seen before.
Some hints on how to make an actionable bug report: http://bruceravel.github.com/demeter/pods/bugs.pod.html
B
Thank you, Jeff
On 12/8/2015 9:15 PM, Bruce Ravel wrote:
Here's what the manual has to say
http://bruceravel.github.io/demeter/aug/output/column.html
Looking at the code, however, it seems the document has swapped columns 2/3 and columns 4/5. I think what you are looking for is in column 4, not column 2. If that's right, I'll treat this as a documentation bug. If that is not the case, then I'll prod you for more information so I can understand the software bug.
In either case, thanks for the report.
Cheers, B
On 12/08/2015 06:09 PM, Jeff Catalano wrote:
I am looking for some help with a possible error on my part or bug associated with exporting normalized mu(E) spectra from Athena 0.9.24 64-bit on Windows 10 Pro 64-bit. I have some Si XANES data with a challenging background shape that is well removed through careful normalization in Athena, as long as the "Flatten normalized data" option is checked. When I save the current group as norm(E), what is saved is the non-flattened normalized data. The checkbox for flattening the data has no effect on what is exported despite changing what is plotted. I have verified this by comparing the files exported the the flatten box checked or unchecked. I have tried this with multiple data files loaded in multiple formats (raw beamline data, simple two-column E and mu data, PRJ files) and the effect is always the same. It is also unaffected by the normalization ranges I select. No matter what options I select Athena 0.9.24 will only save non-flattened norm(E) data.
This behavior is different from what I remember, and I have verified that in version 0.8 of Athena the exported norm(E) was the flattened data, provided that the flatten box was checked (of course). I have been unable to determine if I have any errors in the settings in version 0.9.24 but all parameters that I can find are identical in both 0.9.24 and 0.8. I have also uninstalled and reinstalled Athena 0.9.24 and the effect is still the same. Has anyone else noticed this problem? Is this a bug?
Thank you, Jeff
-- Jeffrey G. Catalano, Associate Professor Department of Earth and Planetary Sciences Washington University 1 Brookings Drive, Campus Box 1169 Saint Louis, MO 63130 USA http://aqgeochem.wustl.edu/
On 12/09/2015 11:49 AM, Jeff Catalano wrote:
I think I have identified the sequence of events to produce a missing nbkg column. The attached text file (cubes1_test.txt) is a simple two column energy vs. mu(E) data with no header. It is for Si XANES data collected at the CLS, and the Si content of this samples was low, hence the noise and poor pre-edge region. When reading this text data into Athena (1st column E, 2nd column mu), Athena sets this to datatype: XANES. See the attached Athena file, I did not change the default normalization settings for this example to keep things simple. If I export norm(E) from this Athena project (see attached cubes1_test_initial.nor), nbkg is missing. If I convert the data to mu(E) via Group > Change Data Type then the nbkg column appears (see attached cubes1_test_muE.nor). Changing data type back to XANES does not remove nbkg, it is still there (not attached, file looks the same as cubes1_test_initial.nor). The missing column seems to be associated with the importing process for the short energy range involved in my data.
Thanks, Jeff. I was making a mistake in handling the output arrays for data marked as being XANES data (http://bruceravel.github.io/demeter/aug/import/columns.html#datatypesandener...). I just pushed a fix to the repository which handles that in a sensible way. This fix will find its way into the next release of the Windows installer. Until then, you will find that the norm(E) files are missing the "normalized background" column and that the "flattened background" column is filled with weird numbers, and that the mu(E) files are missing the background column. The corrected behavior, which will be in the next Windows installer, puts all zeros in those columns. That assures that the column structure for mu(E) and norm(E) files will be the same for any data group, regardless of whether it is flagged as XANES data. Thanks for the bug report. Athena is much better thanks to useful bug reports from users. B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Science Group at NSLS-II Building 535A Upton NY, 11973 Homepage: http://bruceravel.github.io/home/ Software: https://github.com/bruceravel Demeter: http://bruceravel.github.io/demeter/
participants (2)
-
Bruce Ravel -
Jeff Catalano