[DFDL-WG] Minutes from 2007-08-08 Call
Simon Parker
simon.parker at polarlake.com
Wed Aug 15 10:14:07 CDT 2007
Thanks for the explanation, Mike.
This helpful principle is expressed on page 105:
"a parser for any construct (simple or complex) consumes its own
delimiters and only its own delimiters"
Separators belong to the file, terminators belong to the record.
The lenient record-per-line text file can be viewed in several ways,
such as:
file with prefix separators and optional terminator = [record,
{newline, record}], [newline];
file with suffix separators and optional terminator = [{record,
newline}, record], [newline];
file with terminated records last optional = {terminated record},
[record];
terminated record = record, newline;
record = characters except newline;
newline = CR, LF;
Is there anything to choose between these interpretations? Perhaps it's
not our business to worry about it anyway.
I did indeed mean that the property is redundant, and was advocating the
smallest possible language. I still favour a small language, but I now
accept that it needs to be supported by a rich library of convenient
secondary properties in which controlled redundancy is acceptable. We
don't need to talk of eliminating such constructs providing we can find
a good way to express this language/library division.
Simon
________________________________
From: Mike Beckerle [mailto:beckerle at us.ibm.com]
Sent: 14 August 2007 14:24
To: dfdl-wg at ogf.org; Simon Parker
Subject: Re: [DFDL-WG] Minutes from 2007-08-08 Call
I forgot to clarify Simon's question on sp165.
This was the 'finalTerminatorCanBeMissing" property.
We considered the comment that this might be unnecessary.
Use case: file of text format. Each "record" in the file is
terminated by a CRLF so sez the user. At the top level this file
contains an array of these records.
The file might or might not have a CRLF at the end of the file
because human beings might have edited the file with a text editor, and
either inserted or neglected to insert this final CRLF.
We want the file format to be legal with or without the final
CRLF; however, all prior CRLFs in the file must be present.
So how to express this:
1) CRLF is a terminator of the record
2) CRLF is an occursSeparator of the enclosing array, records
have no terminator. We enclose the array in a sequence group where the
array is followed by a hidden "optional" (minOccurs=0 max=1) element of
fixed="CRLF" string value.
Choice (1) requires that we have finalTerminatorCanBeMissing
Choice (2) is just modeling the behavior that is required
directly via hidden elements. This is tantamount to saying that this
keyword is not worth having because there is a way to model it already.
This is true of many keywords. If we deem this one too obscure, then we
need to revisit many others. (Leading/Trailing Skip Bytes is a good
example. Trivially represented by a hidden element). What are our
criteria for inclusion? Up until now our criteria have been to include
things that existing systems already have found a need for. However,
existing systems don't have hidden field capability.
Note that this same missing final terminator issue can come up
not only with End-of-data, but with any bounded size structure.
E.g., suppose we say that an array has occursUnits="bytes" and
occursPath="874". Then it is 874 bytes long. The array elements can be
terminated by a particular data. E.g., semicolon. For the same reasons
as the CRLF example above, we want to be able to tolerate a missing
final semicolon before the end of the 874 bytes. In effect the
byte-length-limit creates an implicit "end-of-data" for a sub-stream
consisting of just those bytes.
Conclusion: finalTerminatorCanBeMissing seems to be useful
enough and comes up often enough that I think the keyword is worthwhile.
Implication: we should create a list of keywords or enumerated
values for properties that we think are in the grey area where perhaps
we want to drop them. Here's some candidates: byteOrderMarkPolicy,
leading/trailingSkipBytes. Both these can be modeled readily as hidden
elements. There are probably others.
Mike Beckerle
STSM, Architect, Scalable Computing
IBM Software Group
Information Platform and Solutions
Westborough, MA 01581
direct: voice and FAX 508-599-7148
assistant: Pam Riordan
priordan at us.ibm.com
508-599-7046
Mike Beckerle/Worcester/IBM
08/14/2007 08:40 AM
To
"Simon Parker" <simon.parker at polarlake.com>
cc
dfdl-wg at ogf.org
Subject
Re: [DFDL-WG] Minutes from 2007-08-08 CallLink
<Notes://D01ML259/85256FDB00077D54/DABA975B9FB113EB852564B5001283EA/BD9C
FD7CA73D7AFD852573360052302A>
In conjunction with the annotated document these notes are
clear, except for 'sp165'. Perhaps someone will recapitulate the
discussion briefly at Wednesday's conference. I think only three
annotations remain:
sp167 Absent and missing (expanded discussion on the wiki
already)
This will be a major topic on a call.
sp172 separatorType="infix"
I'm happy to drop this strange stuff about separatorType=prefix
or postfix and just say separator means infix. However, I would note
that at least two major integration products (IBM WebSphere
Transformation Extender - formerly Mercator, and Microsoft Biztalk, have
this concept, so we may end up putting it back in. Presumably MS copied
the earlier Mercator style, or both got it from common requirements in
some EDI standard.
sp173 defaultWhenMissing (expanded discussion on the wiki
already)
Same topic as sp167 above. Will have a call topic to discuss.
I've added another contribution to the wiki discussion on
'require'.
This seems to be at resolution I think, which is that we can
express this using assertions. The general style of using DFDL to
describe what fixed-data syntactic constructs look like is a good one.
However, I've amended the Wiki thread on this with a further
issue for group consideration. See bottom of page:
https://forge.gridforum.org/sf/wiki/do/viewPage/projects.dfdl-wg/wiki/Re
quire?_message=1187096164776
The 'length and occurs' proposal is an improvement, though I
still have reservations to discuss; likewise the 'opaque data' proposal.
For a call, this week or soon. I will send out an agenda.
Mike Beckerle
STSM, Architect, Scalable Computing
IBM Software Group
Information Platform and Solutions
Westborough, MA 01581
direct: voice and FAX 508-599-7148
assistant: Pam Riordan
priordan at us.ibm.com
508-599-7046
"Simon Parker" <simon.parker at polarlake.com>
Sent by: dfdl-wg-bounces at ogf.org
08/13/2007 10:56 AM
To
<dfdl-wg at ogf.org>
cc
Subject
Re: [DFDL-WG] Minutes from 2007-08-08 Call
In conjunction with the annotated document these notes are
clear, except for 'sp165'. Perhaps someone will recapitulate the
discussion briefly at Wednesday's conference. I think only three
annotations remain:
sp167 Absent and missing (expanded discussion on the wiki
already)
sp172 separatorType="infix"
sp173 defaultWhenMissing (expanded discussion on the wiki
already)
I've added another contribution to the wiki discussion on
'require'.
The 'length and occurs' proposal is an improvement, though I
still have reservations to discuss; likewise the 'opaque data' proposal.
Regards,
Simon
________________________________
From: dfdl-wg-bounces at ogf.org [mailto:dfdl-wg-bounces at ogf.org]
On Behalf Of Mike Beckerle
Sent: 08 August 2007 18:00
To: dfdl-wg at ogf.org
Subject: [DFDL-WG] Minutes from 2007-08-08 Call
MikeB, Geoff Judd, Alan Powell attended.
Continued through SP's comments.
sp37 - got it.
sp45 - agree. This whole part to be rewritten.
sp115 - ok. strict and "lax" as enums. No built-in default - we
never use defaults in the processor itself. Only in the predefined
formats.
sp118 - ok
sp123 - Proposal to simplify length, lengthKind, lengthUnits,
and also occursKind, occursPath, occursPathUnits needed. (along the
lines of byteCount, itemCount, length='delimited' enum, etc.)
sp154 - Need specific proposal to eliminate hexBinary and use
what for opaque (consider also string with encoding='bytes'. ) Or
introduce a dfdl:byteString type or dfdl:opaque type. (derived type -
just a standard name).
sp158 - see sp123
sp165 - needed to have composition property for enclosing groups
and or end-of-data. Regexp doesn't fix this.
Mike Beckerle
STSM, Architect, Scalable Computing
IBM Software Group
Information Platform and Solutions
Westborough, MA 01581
direct: voice and FAX 508-599-7148
assistant: Pam Riordan
priordan at us.ibm.com
508-599-7046
--
dfdl-wg mailing list
dfdl-wg at ogf.org
http://www.ogf.org/mailman/listinfo/dfdl-wg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.ogf.org/pipermail/dfdl-wg/attachments/20070815/b83dc3c8/attachment.htm
More information about the dfdl-wg
mailing list