[DFDL-WG] Action 174: Making DFDL implementations easier - closed

Tue Jun 12 11:59:10 EDT 2012

In the WG call today for action 174 we decided to make both 'endOfParent' 
and the expression language optional, giving:

Feature 
Detection 
Text representation for types other than String 
dfdl:representation="text" for Number, Calendar or Boolean types 
Delimiters 
dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or 
dfdl:lengthKind="delimited" 
BCD calendars 
dfdl:binaryCalendarRep="bcd"   
Multiple schemas 
xs:include or xs:import in xsd 
Named Formats 
dfdl:defineFormat or dfdl:ref 
Choices 
xs:choice in xsd ** 
Arrays where size not known in advance 
dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' ** 
Expressions 
Use of an expression in any property value
End of parent
dfdl:lengthKind ="endOfParent"

Errata will be added.

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848

From:   Mike Beckerle <mbeckerle.dfdl at gmail.com>
To:     Suman Kalia <kalia at ca.ibm.com>
Cc:     dfdl-wg at ogf.org, dfdl-wg-bounces at ogf.org, Steve 
Hanson/UK/IBM at IBMGB
Date:   28/05/2012 02:07
Subject:        Re: [DFDL-WG] Action 174: Making DFDL implementations 
easier

I tend to agree that really restricted path expressions seem easier to me 
than variables both conceptually and implementation wise.
On May 27, 2012 8:34 PM, "Suman Kalia" <kalia at ca.ibm.com> wrote:
I wonder if variables will make the implementation simpler;  set  only 
once semantics,  scoping and  namespace rules etc will be equally 
difficult to implement.   With XPath tools and runtime support freely and 
easily available, I might be  inclined towards using expressions instead 
of variables..  It would be worth checking with other implementers and get 
their opinion.. 

Suman Kalia 
IBM Canada Lab 
WMB Toolkit Architect and Development Lead 
Tel: 905-413-3923 T/L 313-3923 
Email: kalia at ca.ibm.com 

For info on Message broker 
http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.html 

From:        Steve Hanson <smh at uk.ibm.com> 
To:        dfdl-wg at ogf.org, 
Date:        05/25/2012 12:41 PM 
Subject:        Re: [DFDL-WG] Action 174: Making DFDL implementations 
easier 
Sent by:        dfdl-wg-bounces at ogf.org 

Interesting, but there's a problem - variables are currently an optional 
feature!  You could swap variables for paths, which is what I think Mike 
proposed in an earlier mail? 

exprSubset = / | / exprList
atom = . | .. | identifier 
exprList = atom | atom / exprList 

That's essentially equivalent to what MRM provides for its 'repeat 
reference' and 'length reference' properties. 

It doesn't handle things like an array count being 0 based, where you need 
the ability to add 1 to the result. And you can't do a simple comparison, 
which would make it easy to add choices and discriminators to an 
implementation.  The trouble is that once you add in operators and 
literals you've pulled in a lot of the language. 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 

From:        Tim Kimber/UK/IBM 
To:        Mike Beckerle <mbeckerle.dfdl at gmail.com> 
Cc:        dfdl-wg at ogf.org, dfdl-wg-bounces at ogf.org, Steve 
Hanson/UK/IBM at IBMGB 
Date:        25/05/2012 14:29 
Subject:        Re: [DFDL-WG] Action 174: Making DFDL implementations 
easier 

My current thinking is: 
- Minimum expression language is a simple variable reference. e.g. 
 {$myLength} or {$myParentStructureLength}.   
- That would require the ability to declare DFDL variables, set them, read 
them and put them into and out of scope. And restore their value after 
backtracking if backtracking is supported in the implementation. 
This would make all the XPath functions optional, and would also make it 
unnecessary to use an XPath query engine. 

regards,

Tim Kimber, Common Transformation Team,
Hursley, UK
Internet:  kimbert at uk.ibm.com
Tel. 01962-816742  
Internal tel. 246742

From:        Mike Beckerle <mbeckerle.dfdl at gmail.com> 
To:        Steve Hanson/UK/IBM at IBMGB 
Cc:        Tim Kimber/UK/IBM at IBMGB, dfdl-wg at ogf.org, 
dfdl-wg-bounces at ogf.org 
Date:        25/05/2012 13:58 
Subject:        Re: [DFDL-WG] Action 174: Making DFDL implementations 
easier 

But the expression language is highly restricted in the subset.

On Fri, May 25, 2012 at 7:59 AM, Steve Hanson <smh at uk.ibm.com> wrote: 
Tim 

I tend to agree with endOfParent as optional. 

Expression language was kept in the core to handle occursCountKind 
'expression' which is also in the core.  The examples of binary data we 
have seen from NSA and ESA both have occurs counts in the data, in the 
same way as COBOL does. When you have untagged binary data, it's typically 
the way to provide an array size. It was felt that dropping this from the 
core did left us with too little capability.   

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 

From:        Tim Kimber/UK/IBM 
To:        Steve Hanson/UK/IBM at IBMGB 
Cc:        dfdl-wg at ogf.org, dfdl-wg-bounces at ogf.org 
Date:        25/05/2012 10:07 
Subject:        Re: [DFDL-WG] Action 174: Making DFDL implementations 
easier 

1) I would make endOfParent an optional feature - there are not many 
formats that require it. 

3) There are many formats that do not require the expression language - it 
is only required when a property value or an assert/discriminator needs to 
query already-parsed data. On that basis, I think the entire expression 
language feature should be optional. 

regards,

Tim Kimber, Common Transformation Team,
Hursley, UK
Internet:  kimbert at uk.ibm.com
Tel. 01962-816742  
Internal tel. 246742

From:        Steve Hanson/UK/IBM at IBMGB 
To:        dfdl-wg at ogf.org 
Date:        25/05/2012 00:08 
Subject:        [DFDL-WG] Action 174: Making DFDL implementations easier 
Sent by:        dfdl-wg-bounces at ogf.org 

Agreed on list, just need to answer questions 1) and 3) below. 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 
----- Forwarded by Steve Hanson/UK/IBM on 23/05/2012 18:21 ----- 

From:        Steve Hanson/UK/IBM 
To:        dfdl-wg at ogf.org 
Date:        15/05/2012 09:55 
Subject:        Making DFDL implementations easier 

Please see below for a proposal to make an additional set of DFDL features 
optional.  The goal is to make it considerably easier to create a minimal 
conforming DFDL processor for binary data. 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 
----- Forwarded by Steve Hanson/UK/IBM on 11/05/2012 12:50 ----- 
Feature 
Detection 
Text representation for types other than String 
dfdl:representation="text" for Number, Calendar or Boolean types 
Delimiters 
dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or 
dfdl:lengthKind="delimited" 
BCD calendars 
dfdl:binaryCalendarRep="bcd"   
Multiple schemas 
xs:include or xs:import in xsd 
Named Formats 
dfdl:defineFormat or dfdl:ref 
Choices 
xs:choice in xsd ** 
Arrays where size not known in advance 
dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' ** 
Advanced expressions 
Advanced features of the DFDL expression language (tbd)

** Including one of these features mean that speculative parsing is 
needed. 

Remaining questions: 

1) What about lengthKind 'endOfParent' ? 
2) Is leaving out choices too restrictive? 
3) Expression language subset 

The result is that a minimal conformant DFDL implementation just needs to 
support the following annotations and properties, and does not need 
speculative parsing. 

dfdl:element 
dfdl:sequence 
dfdl:format 

byteOrder 
encoding 
utf16width 
alignment 
alignmentUnits (bytes) 
fillByte 
leadingSkip 
trailingSkip 
lengthKind (explicit, implicit) 
length 
lengthUnits (bytes, characters) 
representation (binary) 
textPadKind 
textTrimKind 
textStringJustification 
textStringPadCharacter 
truncateSpecifiedLengthString 
decimalSigned 
binaryNumberRep 
binaryVirtualDecimalPoint 
binaryFloatRep (ieee) 
binaryBooleanTrueRep 
binaryBooleanFalseRep 
binaryCalendarRep (binarySeconds, binaryMilliseconds) 
binaryCalendarEpoch 
sequenceKind (ordered) 
occursCountKind (fixed, expression) 
occursCount 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

--
dfdl-wg mailing list
dfdl-wg at ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU 

--
dfdl-wg mailing list
dfdl-wg at ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg 

-- 
Mike Beckerle | OGF DFDL WG Co-Chair 
Tel:  781-330-0412

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
--
 dfdl-wg mailing list
 dfdl-wg at ogf.org
 https://www.ogf.org/mailman/listinfo/dfdl-wg 

--
 dfdl-wg mailing list
 dfdl-wg at ogf.org
 https://www.ogf.org/mailman/listinfo/dfdl-wg

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/dfdl-wg/attachments/20120612/ca15d9b2/attachment-0001.html>