[DFDL-WG] Action 204: Establish strict versus lax behaviour for ICU calendar patterns

[Steve Hanson]

Agreed on call to add in these descriptions, minus the footnotes.
Errata will be raised to add EEEEEE and eeeeee.
There are several bugs in ICU, all of which should ideally be documented 
in the release notes for a DFDL implementation. The broken EEEEE behaviour 
and the ICU4C v ICU4J differences both come under this.

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848



From:   Mike Beckerle <mbeckerle.dfdl at gmail.com>
To:     Steve Hanson/UK/IBM at IBMGB, 
Cc:     "dfdl-wg at ogf.org" <dfdl-wg at ogf.org>
Date:   14/08/2013 14:23
Subject:        Re: [DFDL-WG] Action 204: Establish strict versus lax 
behaviour for ICU calendar patterns



This is helpful.

Given where we are, let's just put this in as doc of what strict and lax 
mean. 

I'm in favor of adding the variations of EEEE... and eeee... which are 
supported by ICU. This is upward compatible, and will avoid need for a 
special check to exclude them.

The broken EEEEE form is just a bug - I'd say this is just a release note 
item for products providing DFDL, unless ICU fixes it 'real soon now'. 


On Wed, Aug 14, 2013 at 8:58 AM, Steve Hanson <smh at uk.ibm.com> wrote:
For the subset of ICU symbols that DFDL supports, here is what ICU claim: 

1) Lenient parsing behaviour when in 'strict' mode: 
a) case insensitive matching for text fields 
b) MMM, MMMM, MMMMM all accept either short or long form of Month 
c) E, EE, EEE, EEEE, EEEEE **, EEEEEE *** all accept either abbreviated, 
full, narrow and short forms of Day of Week 
d) accept truncated leftmost numeric field (eg, pattern "HHmmss" allows 
"123456" (12:34:56) and "23456" (2:34:56) but not "3456") 

2) Additional lenient parsing behaviour when in 'lax' mode: 
a) values outside valid ranges are normalized (eg, "March 32 1996" is 
treated as "April 1 1996") 
b) ignoring a trailing dot after a non-numeric field 
c) leading and trailing whitespace in the data but not in the pattern is 
accepted **** 
d) whitespace in the pattern can be missing in the data 
e) partial matching on literal strings (eg, data "20130621d" allowed for 
pattern "yyyyMMdd'date' " **** 
** Bug found when testing this - EEEEE 'narrow' form completely broken - 
ICU ticket raised. 
*** EEEEEE and eeeeee are new and support a 2 char version of 'short' form 
- eg Tu or Mo. Not currently allowed by DFDL, we should consider allowing 
it. 
**** Only currently in ICU4C. ICU4J will be changed to match ICU4C. 
Note: IBM is in discussion with ICU to provide a 'really strict' mode 
(name tbd) which has no leniency at all. We need to decide whether to 
reflect all three variants in the dfdl:calendarCheckPolicy, or whether to 
remap our 'strict' to the new 'really strict' mode when it appears. Given 
where we are I think is a DFDL 2.0 item. 
Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

--
  dfdl-wg mailing list
  dfdl-wg at ogf.org
  https://www.ogf.org/mailman/listinfo/dfdl-wg



-- 
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | 
www.tresys.com


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/dfdl-wg/attachments/20130814/6f904edd/attachment.html>