[dfdl-wg] Usage scenarios

[Martin Westhead]

Hi Bob,

Here is my list of ways in which DFDL could be used - I have probably
missed some but here's enough to kick off with. If you need more details
on any of the real-world examples let me know.

Cheers,

Martin


Description
-----------

In QCD physics independent research groups from all over the world have
data which is always a 4d array of floating point values. However,
different groups have different standards for precision, dimension
order, byte order. It would be useful for them to have a simple,
canonical XML language for describing the format of a file. In the first
instance this need only be human readable.

Archiving
---------

Data needs to be stored, but the programs and systems for reading it
become obsolete. DFDL provides a valuable possibility of describing all
the details of a particular format so that even if there were no
programs able to read a format, the description (and the standard) would
provide sufficient information to access archived data. (There are lots
of examples of this type they might include atmospheric measurement).

A sophistication of this is that archived data may need to be
transformed as it is moved to up to data physical media (changes of
precision) etc. It would be nice if DFDL could (a) annotate these
changes (b) (perhaps) be used to ensure that the changes did not result
in data loss.

Format abstraction
------------------

At the simplest level the QCD physicists (described above) would like to
be able to have a single API that would allow them to read any described
piece of data, and carry out all the transformations required to ensure
that they get the correct array in memory.

I have examples of potential users who are just interested in describing
byte-order in a standard way.

At the next level we would like to supply a high level DFDL description
that captures a standard view of the data, and have generic DFDL logic
that can transform an existing DFDL-described format into this generic
view. This is one of the primary motivations for "layers" in the
standard. It is a very powerful feature but it introduces scoping
issues: What transformations can DFDL not describe? (also what
transformations can DFDL not describe efficiently).


Generic data access
-------------------

A DFDL library should provide the ability to interrogate a data
description and read all aspects of the data into memory. An example of
a generic tool is a browser that will allow arbitrary DFDL-described
data to be displayed in some sensible human-readable form. This case
requires the standard to specify an API for reading and interrogating
the data. The favoured suggestion for this is to extend DOM/SAX to allow
the reading of data fields directly into in-memory types (float, int,
char etc.)


Data queries
------------

The DFDL description implies an associated XML document. This document
can be queried using XPath/XQuery to extract pieces of data.

[Note: If the data comes back as an XML-XPath result then this process
is straight forward. With BinX we tried to return the data in a similar
format to the one it is represented in with an accompanying description.
We found a number of issues arose in this case that may or may not also
arise for DFDL].

Data annotations
----------------

The same XPath/XQuery expressions that can be used to query a document
can provide external (format independent) annotations. For example NASA
stores photographic images of hurricanes. A scientist can identify a
blob of pixels that correspond to the hurricane in an image. They could
like to store this annotation is such a way that the will be preserved
through future transformations (e.g. new image format, or different
pixel depth, or compression level). Note the point here is that a byte
offset into the image data cannot do this.

XML without the tags
--------------------

There are groups who would like to use DFDL as a sort of cheap data
compression technique. An example here is particle physics collision
data. This is stored as a set of sparse (hence variable sized) trees of
results. The data is richly structured trees and they would like to
access it and talk about it as if it were in XML but they don't want to
(cannot afford to) represent it using XML markup or use conventional XML
tools to parse it.

The idea is that such a group would design a new binary format that
could be described in XML and then they would work with the implied XML
data. Note: naturally these folks do not want to access their floating
point values as strings so they would want the sort of DOM extensions
that we alluded to earlier. For this same reason things like Binary XML
do not solve their problem.

Another example comes from the astronomy community has recently moved
from a long-standing binary data format (FITS) to an XML version
(VOTable). FITS was very rich in metadata but also included binary
images and large tables of observational data representations. VOTable
is great for capturing the metadata in a standard way but leads to
excessive bloat for images and large tables. The community has ended up
with a complicated compromise in which they allow raw binary data in at
the bottom of the XML file. A DFDL-described format could provide a
cleaner solution.