[dfdl-wg] Defuddle Questions and Pull-Parsing Thoughts

[Jim Myers]

Tom,
The short answer (if my understanding is correct) is that we started 
basically as you describe but Tara has been implementing on-demand 
reading of data : Only structure is read up front to create empty 
parsing classes that get compiled. When you connect that structure to 
a data source(s), nothing happens to start. When you ask for an 
element, you either read or skip (by knowing/calculating their 
length) everything required to find that element's data (which may 
not simply be all elements above it in the schema) and return the 
value. I don't think we have a mechanism to free the memory of that 
element if you are now done with it, though I think we could.  We've 
at least thought about StAX but mostly at the beginning of creating 
defuddle when it was very new, so part of our decision on JaxMe was 
relative maturity. So - I think we can do better with the current 
architecture than a straight fill-the-structure model, but for truly 
large data, while we might be able to scale, starting directly from a 
streaming approach might be better or at least more natural 
(presumably fits the model of the surrounding program better).

  Jim

At 06:54 AM 6/2/2006, Tom Sugden wrote:
>Hi all,
>
>Apologies for missing the telcon this week due to other work
>pressures. I haven't made much progress with any implementation, but
>have been taking a look at the Defuddle code. I have some questions
>that someone may be able to answer, and a few thoughts for discussion.
>
>The current Defuddle implementation is based upon JAXME, the Java/XML
>binding implementation. Presumably JAXME is used to generate an object
>model representation of the data format described by the DFDL schema.
>And then, I think the underlying data stream would be marshaled into
>an instance of that object model. Is this understanding correct?
>
>If my understanding is correct, I'm concerned that this approach may
>not be suitable for large data streams, since the entire object model
>instance would probably have to be assembled and stored in memory,
>like a DOM tree. Has anybody considered using a streamed pull-parsing
>approach instead, based upon or similar to StAX (Streaming API for
>XML)?
>
>I was thinking along the lines of parsing the DFDL schema into DOM or
>some other internal representation. Then pull-parsing the data stream,
>producing a sequence of StAX-like events corresponding to the data in
>the stream and its structure. During the pull-parsing, the context
>would need to be maintained and the conversion algorithm used for
>transforming parts of the data stream into values of the correct type.
>These values would then be wrapped in corresponding event objects.
>
>If this approach was viable, then these StAX-like APIs could be used
>to implement higher-level applications or APIs. For instance, it would
>be straight-forward to produce an XML serialization of any data
>described by a DFDL schema. One could also imagine binding any data
>described by a DFDL schema to auto-generated Java beans, or to a DOM
>object, when desirable. The process may even be reversible, so that
>data could be written back to a data stream as well as being read from
>one.
>
>I haven't thought this through very deeply yet and my understandings
>of the issues are still quite naive, so I will be very interested to
>hear any comments. Sorry if this avenue has already been explored, or
>I've misunderstood the mechanics of Defuddle or JAXME.
>
>Cheers,
>Tom

James D. Myers
Associate Director, Cyberenvironments and Technologies, NCSA
1205 W. Clark St, MC-257
Urbana, IL 61801
217-244-1934
jimmyers at ncsa.uiuc.edu