[DFDL-WG] Action 301: Versioning mechanism for DFDL v2.0 / experimental features
Mike Beckerle
mbeckerle.dfdl at gmail.com
Sat Apr 27 23:19:59 EDT 2019
I like this proposal.
I propose that we include it in DFDL 1.0 (hopefully the last thing we add!)
We'll need to create an erratum for it.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology |
www.tresys.com
Please note: Contributions to the DFDL Workgroup's email discussions are
subject to the OGF Intellectual Property Policy
<http://www.ogf.org/About/abt_policies.php>
On Fri, Apr 5, 2019 at 12:42 PM Steve Hanson <smh at uk.ibm.com> wrote:
> OK so after discussion on WG call the proposal is:
>
> - New namespace "*http://www.ogf.org/dfdl/dfdl-1.0/experimental*
> <http://www.ogf.org/dfdl/dfdl-1.0/experimental>" with default prefix
> "dfdlx"
> - Must be bound on the root of the schema only
> - Attribute form allowed - dfdlx:myNewProp
> - Element form allowed - <dfdl:property name="dfdlx:myNewProp">...
> - Short form allowed - dfdlx:myNewProp
> - Scoping rules apply if makes sense for property
> - A property is needed to enable any experimental feature (presence of the
> namespace is not sufficient)
> - An experience document will result from each experimental feature and
> will guide any inclusion in DFDL 2.0.
>
> Regards
>
> Steve Hanson
>
> IBM Hybrid Integration, Hursley, UK
> Architect, *IBM DFDL*
> <http://www.ibm.com/developerworks/library/se-dfdl/index.html>
> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/>
> *smh at uk.ibm.com* <smh at uk.ibm.com>
> tel:+44-1962-815848
> mob:+44-7717-378890
> Note: I work Tuesday to Friday
>
>
>
> From: Mike Beckerle <mbeckerle.dfdl at gmail.com>
> To: Steve Hanson <smh at uk.ibm.com>
> Cc: Bradd Kadlecik <braddk at us.ibm.com>, DFDL-WG <dfdl-wg at ogf.org>,
> Michele Zundo <michele.zundo at esa.int>
> Date: 04/04/2019 20:58
> Subject: Re: [DFDL-WG] Action 301: Versioning mechanism for DFDL
> v2.0 / experimental features
> ------------------------------
>
>
>
>
> I think we have to allow scoping rules to apply to new experimental
> properties if they are regular format properties. Some proposed extensions
> would be much too hard to use without scoping being available.
>
> But some new proposed extensions aren't the kind of thing one puts in
> scope anyway, so like existing non-scoped properties, new extension
> properties could be scoped or not-scoped depending on their nature.
>
> I think any new experimental property is going to also need a default
> value for it, for backward compatibility as you suggest.
>
> Since these are in namespace dfdlx, we don't have to worry about turning
> OFF this default. Once a property is accepted into the DFDL spec formally,
> then implementations probably should have switches for turning on/off
> defaults for it, as for new properties that have been added since the
> earliest implementation were already in existence.
>
> There are other things needed as extensions that aren't properties
> however. Ex: we've had requests for %LSP; (and +, * variants) character
> class entity. This is like %WSP; without the NLs., i.e., intra-line
> whitespace characters.
> I expect we want some property-like way to enable/disable use of such an
> extension feature. (With the default being to disallow.)
>
> ...mikeb
>
>
> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology |
> *www.tresys.com* <http://www.tresys.com>
> Please note: Contributions to the DFDL Workgroup's email discussions are
> subject to the *OGF Intellectual Property Policy*
> <http://www.ogf.org/About/abt_policies.php>
>
>
>
> On Thu, Apr 4, 2019 at 10:59 AM Steve Hanson <*smh at uk.ibm.com*
> <smh at uk.ibm.com>> wrote:
> These are just experimental properties, we don't have to roll out the full
> property syntax treatment, we just need to do the minimum to allow their
> use. I propose the following:
>
> - New namespace "*http://www.ogf.org/dfdl/dfdl-1.0/experimental*
> <http://www.ogf.org/dfdl/dfdl-1.0/experimental>" with default prefix
> "dfdlx"
> - Must be bound on the root of the schema only
> - Attribute form not allowed
> - Element form not allowed
> - Short form allowed via dfdlx:myNewProp
> - Scoping rules do not apply
>
> There is a problem if we allow scoping. Scoping means that you can not be
> silent about the value of a property when it is needed by an object. But by
> definition we do not know the full set of experimental properties, they
> will change over time, so schemas using the experimental namespace would
> throw new SDEs out of nowhere if a processor started to support a new
> experimental feature.
>
> We could allow scoping if we changed the rules for experimental properties
> so they have a built-in default (we have been forced to do that when errata
> have added new properties to the core 1.0 spec - but there have only been
> one or two). If so the prefixed property would appear in attribute form on
> DFDL annotation elements, ie, dfdlx:myNewProp.
>
> If we really need to allow element form then that could be done via a new
> dfdlx:property annotation element.
>
> Regards
>
> Steve Hanson
> IBM Hybrid Integration, Hursley, UK
> Architect, *IBM DFDL*
> <http://www.ibm.com/developerworks/library/se-dfdl/index.html>
> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/>
> *smh at uk.ibm.com* <smh at uk.ibm.com>
> tel:+44-1962-815848
> mob:+44-7717-378890
> Note: I work Tuesday to Friday
>
>
>
> From: Mike Beckerle <*mbeckerle.dfdl at gmail.com*
> <mbeckerle.dfdl at gmail.com>>
> To: Steve Hanson <*smh at uk.ibm.com* <smh at uk.ibm.com>>
> Cc: DFDL-WG <*dfdl-wg at ogf.org* <dfdl-wg at ogf.org>>, Bradd Kadlecik <
> *braddk at us.ibm.com* <braddk at us.ibm.com>>
> Date: 01/02/2019 14:34
> Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL
> v2.0 features/changes
> ------------------------------
>
>
>
> This seems reasonable to me.
>
> A few questions and suggestions.
>
> How are these properties expressed in long form annotations or
> property-element form annotations?
>
> I.e., <dfdl:format ..../> normally the attributes that short form have
> "dfdl:" would not have any namespace. Would the extension properties that
> show up inside dfdl:format carry their namespace there also? As in
> <dfdl:format tddt:pointerEnable="true"/> ?
>
> We also would need a way to make ultra long form properties work, e.g.,
>
> <dfdl:element>
> <dfdl:property name="tddt:pointerEnable">true</dfdl:property>
> </dfdl:element>
>
> It's odd at first in any XSD-embedded language for a name attribute to
> take a QName as value. In this case I think it is proper. Really the name
> attribute isn't creating a name here, it's referencing a name.
>
> Assuming all the above notational issues are resolvable,.....
>
> Having a agreed upon URI for extension properties makes sense, because we
> can check if it is bound, and can validate short-form properties in that
> namespace.
>
> We need some way of looking at the schema at the top, and saying "this has
> extensions".
>
> There is a precedent for requiring a namespace binding to be at the root
> of a schema file, which is the feature where we allow include/import of
> non-DFDL XSD schemas that define other annotation languages. Such
> annotation languages can use attributes for example, though DFDL schemas
> cannot. When there is no binding of the dfdl namespace on the root of a
> schema, we don't include it for schema compilation purposes, though it is
> included for schema validation purposes.
>
> Given that precedent, we could require a binding of this extension URI at
> the root of the schema, rather than just allowing it anywhere.
>
>
> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology |
> *www.tresys.com* <http://www.tresys.com>
> Please note: Contributions to the DFDL Workgroup's email discussions are
> subject to the *OGF Intellectual Property Policy*
> <http://www.ogf.org/About/abt_policies.php>
>
>
>
> On Fri, Feb 1, 2019 at 4:34 AM Steve Hanson <*smh at uk.ibm.com*
> <smh at uk.ibm.com>> wrote:
> Mike
>
> I'd like to make some progress on this if possible? The IBM z/TPF team
> are a heavy user of IBM DFDL. They are interested in adding offset support
> and also support for pointers (more on the latter on next WG call).
>
> A problem with using DFDLVersion="2" is that it is pre-judging what is
> going to be in 2.0. I would prefer that new features appeared in DFDL 1.0
> schemas as 'experimental' with their DFDL properties in a separate
> namespace. They would switch to the DFDL namespace only in a 2.0 schema.
> This gives us flexibility around property naming and behaviour - we might
> decide to change things when formalising 2.0.
>
> For example this is what z/TPF have done for pointers:
>
> <xs:element name="myString" type="xs:string" dfdl:lengthKind="explicit"
> dfdl:length="../len" tddt:indirectKind="pointer" tddt:indirectLength="4" />
>
>
> It means that when a user migrates to DFDL 2.0 they would need to change
> their schemas but I think that is preferable to the current proposal where
> 2.0 bleeds into 1.0.
>
> Regards
>
> Steve Hanson
> IBM Hybrid Integration, Hursley, UK
> Architect, *IBM DFDL*
> <http://www.ibm.com/developerworks/library/se-dfdl/index.html>
> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/>
> *smh at uk.ibm.com* <smh at uk.ibm.com>
> tel:+44-1962-815848
> mob:+44-7717-378890
> Note: I work Tuesday to Friday
>
>
>
> From: Mike Beckerle <*mbeckerle.dfdl at gmail.com*
> <mbeckerle.dfdl at gmail.com>>
> To: *dfdl-wg at ogf.org* <dfdl-wg at ogf.org>
> Date: 02/02/2018 22:07
> Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL
> v2.0 features/changes
> Sent by: "dfdl-wg" <*dfdl-wg-bounces at ogf.org*
> <dfdl-wg-bounces at ogf.org>>
> ------------------------------
>
>
>
> So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add
> <dfdl:format DFDLVersion="2"/> to it at top level in order to express
> version when DFDL v2.0 features are added to it.
>
> I think the alternative would be a separate annotation element
> <dfdl:version value="2"/> that is explicitly for this purpose, is allowed
> only at top level, etc.
> That would eliminate the complexity of getting a DFDLVersion by way of a
> dfdl:ref that includes a named format. It could be required to just be at
> lexical top level of every schema document that uses v2.0 features.
>
> The reason not to change namespaces, even though that is what we thought
> we should do originally, is based on various guidance such as
> *http://www.xml.com/pub/a/2004/07/21/design.html *
> <http://www.xml.com/pub/a/2004/07/21/design.html>
>
> These comments mostly apply to versioning of individual DFDL schemas, not
> of the schema-defining language itself.
>
> The quintessential example of a language version changing is XMLSchema
> itself which now has version 1.1.
> This still uses the same namespace as XMLSchema 1.0.
>
>
> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology |
> *www.tresys.com* <http://www.tresys.com>
> Please note: Contributions to the DFDL Workgroup's email discussions are
> subject to the *OGF Intellectual Property Policy*
> <http://www.ogf.org/About/abt_policies.php>
>
>
> On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <*smh at uk.ibm.com*
> <smh at uk.ibm.com>> wrote:
> Mike,
>
> Your scheme as proposed has a couple of problems.
> a) It is possible to create a schema that does not use dfdl:format at all.
> Unlikely but possible. With such a schema there is no way to indicate
> version.
> b) What are the version implications for annotations that have nothing to
> do with dfdl:format? Asserts, discriminators, variables.
>
> Most of the language you use refers to versions of *schemas*. That's not
> the same as version of a dfdl:format. Any reason you didn't just change the
> DFDL namespace URL to be *http://www.ogf.org/dfdl/dfdl-2.0/*
> <http://www.ogf.org/dfdl/dfdl-2.0/>, which must be declared before any
> DFDL feature can be used? I think that was always the intent of the DFDL
> namespace.
>
> The key piece of work is validating any proposal against the scoping rules
> in section 8 of the spec.
>
> Regards
>
> Steve Hanson
> IBM Hybrid Integration, Hursley, UK
> Architect, *IBM DFDL*
> <http://www.ibm.com/developerworks/library/se-dfdl/index.html>
> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/>
> *smh at uk.ibm.com* <smh at uk.ibm.com>
> tel:*+44-1962-815848* <+44%201962%20815848>
> mob:*+44-7717-378890* <+44%207717%20378890>
> Note: I work Tuesday to Friday
>
>
>
> From: Mike Beckerle <*mbeckerle.dfdl at gmail.com*
> <mbeckerle.dfdl at gmail.com>>
> To: *dfdl-wg at ogf.org* <dfdl-wg at ogf.org>
> Date: 23/01/2018 16:03
> Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0
> features/changes
> Sent by: "dfdl-wg" <*dfdl-wg-bounces at ogf.org*
> <dfdl-wg-bounces at ogf.org>>
> ------------------------------
>
>
>
>
> There are some features now under discussion which will most likely not be
> part of the DFDL v1.0 standard, but a subsequent version 2.0 of the
> standard.
>
> Implementations may have these features prior to the existence of that
> draft version of the DFDL standard; hence, a mechanism for indicating the
> support, and requirements that implies, on DFDL schemas is needed in order
> for these v2.0 features to be introduced.
>
> The purpose of the versioning mechanism is to provide access to DFDL v2.0
> features, while maintaining also backward compatibility with DFDL v1.0
> schemas.
>
> For example, new DFDL format properties may be added to the DFDL v2.0
> version; however, those properties are not required to be specified unless
> the version of the schema is specifically set to indicate it is using
> version 2.0 features. Also, deprecated features should only generate
> deprecation warnings if the user is requesting version 2.0 of DFDL, and
> should be silent otherwise.
>
> Proposal:
>
> New attribute on the dfdl:format annotation: DFDLVersion with valid values
> "1", "2". All values are reserved for future use.
>
> The DFDLVersion, if unspecified, defaults to "1" which means the
> implementation is of the DFDL v1.0 specification.
>
> If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL
> may be enforced on the schema, and schema definition errors will result if
> version 2.0 requirements are not met. DFDL Version 1.0 features that are
> deprecated will generate warnings.
>
> Example:
>
> <dfdl:format DFDLVersion="2" .../>
>
> Example:
>
> <dfdl:defineFormat name="myV2Format">
> <dfdl:format DFDLVersion="2" .../>
> </dfdl:defineFormat>
>
> The DFDLVersion is not a format property and is not scoped. It can be
> added to a named format definition which is then referenced in the usual
> manner for named format definitions.
>
> Backward Compatibility Requirement
>
> DFDL Implementations that provide version 2.0 functionality must also
> provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends
> version 1.0.
>
> Mixed Version Schemas
>
> If a schema consists of multiple schema documents, and their respective
> dfdl:format annotations do not all contain the same value for the
> DFDLVersion attribute, then the schema is said to be "mixed version". This
> is allowed.
>
> A named dfdl:format carrying DFDLVersion "2" may not be referenced from a
> DFDL Schema document having DFDLVersion "1".
>
> The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or
> lacking the DFDLVersion attribute may be referenced from a DFDL Schema
> having DFDLVersion="2".
>
> There are really 3 ways that DFDL version 2.0 can differ from version 1.0,
> and these are that new features are added, old features are deprecated, or
> old features are removed. This latter is quite undesirable, but will be
> considered a possibility for purpose of discussion this versioning system.
>
> These will commonly be in combination such as if an old feature is
> deprecated with the replacement feature being added.
>
> These will now be discussed separately, with the emphasis on how mixed
> version schemas must work.
>
> New Features Added
>
> Each new required DFDL property introduced for version 2.0 will be defined
> along with what is called its "version 1.0 default value". When a mixed
> version schema exists, the DFDL v1.0 parts of it behave as if any required
> DFDLVersion 2 properties have their version 1.0 default value defined, at
> top level of the schema (as if defined with that default value in each
> DFDLVersion 1 schema document's dfdl:format annotation).
>
> Features Removed
>
> Incompatibilities are to be avoided wherever possible, and may not be
> needed at all. However, there is still the possibility that some feature
> will want to be entirely removed and replaced by something better, and in
> that case the versioning scheme needs to be able to support this.
>
> In that situation, the schema definition error for version 2 is only
> generated when a schema has DFDLVersion 2 specified (in dfdl:format
> explicitly, or by reference to a named format carrying or referencing
> DFDLVersion 2). Mixed schemas that contain schema documents entirely using
> version 1 of DFDL do not generate this schema definition error.
>
> Features deprecated
>
> Deprecation warnings are generated only when a DFDLVersion="2" attribute
> is present or is incorporated by reference (to a named format). A version 1
> schema that is incorporated with a version 2 DFDL schema but is not
> modified in any way, does not generate deprecation warnings about version
> 1.0 features that have been deprecated in version 2.0.
>
> (Most likely deprecations would be revising property names or value enums
> for clarity.)
>
> Implementations may provide mechanisms for suppressing warnings. These are
> implementation dependent.
>
> Version 1.0 Existing Deprecations and Compatibility
>
> During the process of standardization for DFDL v1.0 a number of properties
> names were changed. DFDL version 1.0 implementations accept both the old
> and new property names.
> (Example: dfdl:separatorSuppressionPolicy, which replaced an older
> property name.) DFDL schemas that specify DFDLVersion="2" will not accept
> the older property names, and it is a schema definition error if the old
> (pre DFDL v1.0) property names are used.
>
> Properly constructed mixed version schemas can still depend on the older
> property names in the version 1.0 parts of the schema.
>
> ------------------------------
>
>
> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology |
> *www.tresys.com* <http://www.tresys.com>
> Please note: Contributions to the DFDL Workgroup's email discussions are
> subject to the *OGF Intellectual Property Policy*
> <http://www.ogf.org/About/abt_policies.php>
> --
> dfdl-wg mailing list
> *dfdl-wg at ogf.org* <dfdl-wg at ogf.org>
> *https://www.ogf.org/mailman/listinfo/dfdl-wg*
> <https://www.ogf.org/mailman/listinfo/dfdl-wg>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> --
> dfdl-wg mailing list
> *dfdl-wg at ogf.org* <dfdl-wg at ogf.org>
> *https://www.ogf.org/mailman/listinfo/dfdl-wg*
> <https://www.ogf.org/mailman/listinfo/dfdl-wg>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/dfdl-wg/attachments/20190427/ac83d1d8/attachment-0001.html>
More information about the dfdl-wg
mailing list