.. include:: _defs.rst

XDI: XAFS Data Interchange Format
------------------------------------


The `XAFS Data Interchange Format`_ or XDI format was first conceived by Bruce
Ravel, discussed `Q2XAFS_2011`_ meeting, and in the :cite:`XASDataFormats`
article from a working group formed at that meeting.  The format was then
further refined and presented at the Q2XAFS 2015 meeting (`XDI Q2XAFS2015
presentation`_), and then published by :cite:`XDI2016` in the proceedings of
the 2015 XAFS Conference.  Details of the format, including examples and code
for reading data, and `XDI metadata dictionary`_ in the XDI format are give at
the `XDI github repository`_.

Overview of the XDI format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The XDI standard represents a single XAS spectrum along with relevant metadata
in a text-based format with simple syntax for metadata describing the
measurement and a single data table that is meant to be easily read and
interpreted by both humans and software.  The members of the working groups
formed to discuss data formats for exchanging XAS data and the authors of the
XDI format and support software encourage using XDI as the "standard format"
for communicating XAS data whenever possible.

The XDI format is focused on representing the measured XAS spectrum
:math:`\mu(E)`, which will typically consist of a few 1-dimensional arrays of
numerical data that may include: ``energy``, the energy of the incident X-ray
beam, ``i0``, the measured intensity of the incident beam, ``itrans``, the
measured intensity of the transmitted beam, ``ifluor``, the measured intensity
of the fluorescent or emission beam, and ``irefer``, the measured intensity of
the beam after a reference sample often used as an energy calibration.  All of
the intensity values are left to be in unspecified units, and may not all be
available for all spectra.  In addition, the arrays ``mutrans``, ``mufluor``,
and ``murefer`` may contain the measurement of :math:`mu(E)` for the various
signals.  The energy will typically have units of eV or keV but can also be
derived by an ``angle`` array that represents the angle (in degrees) of the
monochromator as long as its d-spacing is available.  With these arrays
available, XDI gives ample flexibility to represent a single XAS spectrum.

Some limitations of XDI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

While XDI focuses on a single XAS spectrum of :math:`\mu(E)`, it is somewhat
limited for describing all variations of XAS data. Most obviously, it contains
only one spectrum, and cannot represent a series of related spectra.  It is
also not very specific about how to handle fluorescence or emission XAS data
measured with multi-element detectors, especially that use pulse-counting
electronics that may need dead-time corrections.  And while it can be used for
HERFD XAS data, it is not prescriptive about how to specify the energy window
selected by the HERFD analyzers, or the ranges of :math:`q` values for X-ray
Raman data.  XDI is also not really capable of describing multi-dimensional XAS
data, say measured in full-field mode or by scanning through a RIXS plane.

While some of these limitations are understandable for a text-based format
aimed at representing a single XAS measurement of "good data on a
well-characterized sample", it also means that XDI is not easily capable of
representing as-measured (or "Raw") data.  While many beamlines do use XDI-like
text files and even use its conventions for tagging metadata, it seems that
exact adherence to the XDI metadata tags is not universal (even at my own
beamline!).  Importantly, for "Raw" as-measured data files. it is not always
well-known whether data is being best measured in Transmission or Emission mode,
as many (it appears even "most") beamlines will measure all available channels
even when some of them contain poor or no measurements at all. For fluorescence
data measured with multi-element solid-state detectors, a single ``ifluor``
channel may not be available, as multiple channels would need to be corrected
for dead-time and summed.  While the "Raw" data may need some light processing
before identifying the proper arrays of data to communicate as "the spectrum".
While that processing may not be too complex, it likely requires specific
knowledge of the columns in the data table (for example which columns hold
InputCountRate and OutputCountRate for each detector element). That means that
while the derived ``ifluor`` or ``mufluor`` in the XDI specification is still
preferred, the full data table of the "Raw" data is still interesting and worth
preserving and transmitting.

Even with some beamlines collecting "Raw" XAS data into "XDI-like" files, many
of these regularly use multi-element fluorescence detectors and will record
many channels (up to 100 is not unusual) for even energy point in a typical XAS
scan.  While there are certainly advantages to plain-text files (more on this
below), a file with 100 columns is not easily described as "human readable".

That is, while the XDI format gives a simple and very good way to represent a
single :math:`\mu(E)` spectrum, it is less obvious how to best archive and
disseminate multiple raw XAS data, particularly those measured with
multi-element fluorescence detectors.  These two limitations do complicate the
presentation of XAS data for on-line databases, supplemental material and
downloadable archives of data.