Athena: Data cannot be imported by using plugin PFBL12C
Dear all, I tried to import a data of multi-element detector at SPring-8 by using plugin PFBL12C to Athena, but it was not successful. if the number of rows in the file is about 2000[success.dat], it can be successfully imported. But for the data over about 4000 rows[fail.dat], it seems that several data at first of both converted energy and intensity are substituted by 0. column6.png shows intensity of ch6 vs energy_attained and I0.png shows intensity of I0 vs energy_attained. you can see this behavior. As a result, intensity cannot divide by I0, and data cannot be imported. if someone knows approach to solve this, could you teach me. Best regards, Yuji Hiraoka
Hi Yuri, You are probably running into the memory limit. This has been discussed a number of times on this mailing list, e.g.: https://www.mail-archive.com/ifeffit@millenia.cars.aps.anl.gov/msg03931.html https://www.mail-archive.com/ifeffit@millenia.cars.aps.anl.gov/msg05673.html Bruce's answer in the 2nd link provides some suggestions. Note, if I delete 1500 rows or so from your fail file, I am able to import it. This seems to be similar to the success.dat file which also appears to be truncated. Using a column labelled '20' as Io seems to work -R. On 11/14/2017 12:59 AM, 平岡 裕治 wrote:
Dear all,
I tried to import a data of multi-element detector at SPring-8 by using plugin PFBL12C to Athena, but it was not successful. if the number of rows in the file is about 2000[success.dat], it can be successfully imported. But for the data over about 4000 rows[fail.dat], it seems that several data at first of both converted energy and intensity are substituted by 0. column6.png shows intensity of ch6 vs energy_attained and I0.png shows intensity of I0 vs energy_attained. you can see this behavior. As a result, intensity cannot divide by I0, and data cannot be imported.
if someone knows approach to solve this, could you teach me.
Best regards, Yuji Hiraoka
_______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
I have to admit I find this all sort of disappointing. First, the PFBL12C is a really poor way to represent EXAFS data. It uses a hard-to-parse and hard-to-interpret format and has many unlabeled columns. It appears to include this really odd notion that recording "angle" is somehow better than recording energy, and saves a ridiculuously over-sampled spectrum that needs smoothing because of the high noise per recorded point. It is, in short, a really poor way to represent EXAFS data. We've seen it for many years, we've written papers and published supported code for better representations of EXAFS data. Still, we get questions about using data recorded in such files. Data files this complex need a way to convert them into something useful. Second, the limitations you are seeing are due to using the outdated, no-longer-supported (for several years now actually) Ifeffit library. These data can be read and processed without problem with Larch, the replacement for Ifeffit. Stop using Ifeffit. Use Larch instead. Really. An example script to use larch for your `fail.dat` file is listed below. It is somewhat complicated because the raw data file is such a poor representation of EXAFS data. --Matt ##### larch script to read, smooth and process data in the horrible PFBL12C format # read raw data, extract angle, i0, sum of fluorescence datafile = 'fail.dat' rawdat = read_ascii(datafile) def degrees2eV(angle, dspace=1.63747): from scipy.constants import hbar, c, e hbarc = 1.e10*hbar*c/e return (pi*hbarc/dspace) / sin(angle*pi/180.0) #enddef rawdat.angle = rawdat.data[0, :] # angle in degrees. yikes! rawdat.energy = degrees2eV(rawdat.angle) rawdat.i0 = rawdat.data[42, :] rawdat.fluor = rawdat.data[3:42, :].sum(axis=0) # smooth data with Savitzky-Golay (data is ~evenly sampled in angle) smoothed = group(energy=rawdat.energy) smoothed.i0 = savitzky_golay(rawdat.i0, 5, 1) smoothed.fluor = savitzky_golay(rawdat.fluor, 5, 1) newplot(smoothed.energy, rawdat.fluor, win=3) plot(smoothed.energy, smoothed.fluor, zorder=50, linewidth=4, win=3) ### create "normal" XAFS energies, finely spaced over edge, ### but spaced evenly in k for the EXAFS e0 = 8980.0 # nominal Cu K edge emin = smoothed.energy.min() emax = smoothed.energy.max() - 5.0 epre1 = e0 - 20.0 epre2 = e0 + 20.0 kmin = etok(20.0) kmax = etok(emax-e0) def make_range(start, stop, step): npts = max(2, 1 + int(0.1 + abs(stop-start)/abs(step))) return linspace(start, stop, npts) #enddef en_new = concatenate((make_range(emin, epre1, 5), make_range(epre1+0.5, epre2-0.5, 0.5), e0 + ktoe(make_range(kmin, kmax, 0.05)))) # now interpolate smoothed data onto this energy grid dat = group(energy=en_new, filename=datafile) dat.i0 = interp(smoothed.energy, smoothed.i0, dat.energy) dat.fluor = interp(smoothed.energy, smoothed.fluor, dat.energy) dat.mu = dat.fluor / dat.i0 # run autobk autobk(dat, rbkg=0.8, kweight=1) # plot data plot_mu(dat, show_norm=True) plot_chik(dat, kweight=1, win=2) ###################################################
Dear Matt, Thanks for the reply and script. I will try to use Larch, instead of athena with Ifeffit, process the data by a proper method, and conduct a study for dataformat. Best regards, Yuji Hiraoka
I have to admit I find this all sort of disappointing.
First, the PFBL12C is a really poor way to represent EXAFS data. It uses a hard-to-parse and hard-to-interpret format and has many unlabeled columns. It appears to include this really odd notion that recording "angle" is somehow better than recording energy, and saves a ridiculuously over-sampled spectrum that needs smoothing because of the high noise per recorded point. It is, in short, a really poor way to represent EXAFS data. We've seen it for many years, we've written papers and published supported code for better representations of EXAFS data. Still, we get questions about using data recorded in such files. Data files this complex need a way to convert them into something useful.
Second, the limitations you are seeing are due to using the outdated, no-longer-supported (for several years now actually) Ifeffit library. These data can be read and processed without problem with Larch, the replacement for Ifeffit.
Stop using Ifeffit. Use Larch instead. Really.
An example script to use larch for your `fail.dat` file is listed below. It is somewhat complicated because the raw data file is such a poor representation of EXAFS data.
--Matt
##### larch script to read, smooth and process data in the horrible PFBL12C format # read raw data, extract angle, i0, sum of fluorescence datafile = 'fail.dat' rawdat = read_ascii(datafile)
def degrees2eV(angle, dspace=1.63747): from scipy.constants import hbar, c, e hbarc = 1.e10*hbar*c/e return (pi*hbarc/dspace) / sin(angle*pi/180.0) #enddef
rawdat.angle = rawdat.data[0, :] # angle in degrees. yikes! rawdat.energy = degrees2eV(rawdat.angle) rawdat.i0 = rawdat.data[42, :] rawdat.fluor = rawdat.data[3:42, :].sum(axis=0)
# smooth data with Savitzky-Golay (data is ~evenly sampled in angle) smoothed = group(energy=rawdat.energy) smoothed.i0 = savitzky_golay(rawdat.i0, 5, 1) smoothed.fluor = savitzky_golay(rawdat.fluor, 5, 1)
newplot(smoothed.energy, rawdat.fluor, win=3) plot(smoothed.energy, smoothed.fluor, zorder=50, linewidth=4, win=3)
### create "normal" XAFS energies, finely spaced over edge, ### but spaced evenly in k for the EXAFS
e0 = 8980.0 # nominal Cu K edge emin = smoothed.energy.min() emax = smoothed.energy.max() - 5.0 epre1 = e0 - 20.0 epre2 = e0 + 20.0 kmin = etok(20.0) kmax = etok(emax-e0)
def make_range(start, stop, step): npts = max(2, 1 + int(0.1 + abs(stop-start)/abs(step))) return linspace(start, stop, npts) #enddef
en_new = concatenate((make_range(emin, epre1, 5), make_range(epre1+0.5, epre2-0.5, 0.5), e0 + ktoe(make_range(kmin, kmax, 0.05))))
# now interpolate smoothed data onto this energy grid dat = group(energy=en_new, filename=datafile) dat.i0 = interp(smoothed.energy, smoothed.i0, dat.energy) dat.fluor = interp(smoothed.energy, smoothed.fluor, dat.energy) dat.mu = dat.fluor / dat.i0
# run autobk autobk(dat, rbkg=0.8, kweight=1)
# plot data plot_mu(dat, show_norm=True) plot_chik(dat, kweight=1, win=2)
###################################################
--------html-part included links------- http://dat.mu
-------------------------------text/plain------------------------------- _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
Dear Robert, Thanks for the reply and advice. I will try to use the larch, instead of athena with Ifeffit, and process the data by a proper method. Best regards, Yuji Hiraoka
Hi Yuri,
You are probably running into the memory limit. This has been discussed a number of times on this mailing list, e.g.:
https://www.mail-archive.com/ifeffit@millenia.cars.aps.anl.gov/msg03931.html
https://www.mail-archive.com/ifeffit@millenia.cars.aps.anl.gov/msg05673.html
Bruce's answer in the 2nd link provides some suggestions.
Note, if I delete 1500 rows or so from your fail file, I am able to import it. This seems to be similar to the success.dat file which also appears to be truncated.
Using a column labelled '20' as Io seems to work [img:daokbeheloophpeb.png]
-R.
On 11/14/2017 12:59 AM, 平岡 裕治 wrote: Dear all,
I tried to import a data of multi-element detector at SPring-8 by using plugin PFBL12C to Athena, but it was not successful. if the number of rows in the file is about 2000[success.dat], it can be successfully imported. But for the data over about 4000 rows[fail.dat], it seems that several data at first of both converted energy and intensity are substituted by 0. column6.png shows intensity of ch6 vs energy_attained and I0.png shows intensity of I0 vs energy_attained. you can see this behavior. As a result, intensity cannot divide by I0, and data cannot be imported.
if someone knows approach to solve this, could you teach me.
Best regards, Yuji Hiraoka
_______________________________________________ Ifeffit mailing list mailto:Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
-------------------------------text/plain------------------------------- _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
-------------------------------text/plain------------------------------- _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
participants (4)
-
Matt Newville -
Robert Gordon -
Yuji Hiraoka -
平岡 裕治