[DFDL-WG] S2G use of DFDL - to be discussed on 8th May DFDL-WG call

Tue May 8 09:24:46 EDT 2012

Hi Michele

We are looking forward to the discussion on S2G and DFDL on today's 
DFDL-WG call.

So that other folk on the call can see what we have been discussing over 
the past few weeks, I've listed the full set of issues in one place:

The original 4 issues which I noted are reproduced here...

1. DFDL does not have defaults for its properties. You must not use 
defaults, instead use dfdl:format to apply values across all the xsd. 
2. You are using dfdl:encoding and dfdl:byteOrder on complex elements to 
control the encoding and byte order of child elements. This is not how 
DFDL works, you need to use dfdl:format. 
3. The namespace for DFDL in the xsds is incorrect - it is missing a 
trailing slash - "http://www.ogf.org/dfdl/dfdl-1.0/" 
4. There is no such property value as dfdl:lengthKind="expression" - it is 
just "explicit" (DFDL knows it is an expression because of {..} ) 

(Your responses to these are further down the attached e-mail thread.)

Some other issues I have noted...

5. Section 3.1 in the pdf. The DFDL infoset shown is not a correct 
infoset, as it contains DFDL property values instead of data values.

6. Section 5. dmx:representation. Better if the name was not 
'representation' as that is too close to dfdl:representation. Suggest 
dmx:displayRep.

7. Section 5.2. Time code representations. Are any of these sufficiently 
common to justify including in the DFDL specification? 

8. What does this DMX property do?  dmx:concealable="true"

9. Attribute form of DFDL & DMX properties are mixed together in an 
illegal way. Take the following example.

        <xs:simpleType name="TypeCADUSyncMarker">
                <xs:annotation>
                        <xs:appinfo source="
http://www.ogf.org/dfdl/dfdl-1.0">
                                <dmx:simpleType dfdl:lengthKind="explicit" 
dfdl:lengthUnits="bytes" dfdl:length="4" dmx:representation="Hexadecimal"
/>
                        </xs:appinfo>
                </xs:annotation>
                <xs:restriction base="xs:int"/>
        </xs:simpleType>

a) The 'source' attribute says this is a DFDL appinfo annotation, yet the 
content is a dmx:simpleType element.
b) The only place DFDL short form properties can appear is directly on a 
schema object.

I would expect to see the following, which has separate appinfos for DFDL 
and DMX. Note that the dmx: is not needed when encoded on the appinfo 
element.

        <xs:simpleType name="TypeCADUSyncMarker">
                <xs:annotation>
                        <xs:appinfo source="
http://www.ogf.org/dfdl/dfdl-1.0">
                                <dfdl:simpleType lengthKind="explicit" 
lengthUnits="bytes" length="4"/>
                        </xs:appinfo>
                        <xs:appinfo source="<S2G URL>">
                                <dmx:simpleType 
representation="Hexadecimal"/>
                        </xs:appinfo>
                </xs:annotation>
                <xs:restriction base="xs:int"/>
        </xs:simpleType>

Or carry DMX properties on the schema object in short form:

        <xs:simpleType name="TypeCADUSyncMarker" 
dmx:representation="Hexadecimal">
                <xs:annotation>
                        <xs:appinfo source="
http://www.ogf.org/dfdl/dfdl-1.0">
                                <dfdl:simpleType lengthKind="explicit" 
lengthUnits="bytes" length="4"/>
                        </xs:appinfo>
                </xs:annotation>
                <xs:restriction base="xs:int"/>
        </xs:simpleType>

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848

From:   Steve Hanson/UK/IBM
To:     Michele Zundo <michele.zundo at esa.int>
Date:   16/04/2012 10:22
Subject:        Re: [DFDL-WG] DFDL based application released

Hi Michele

The invite I sent was a repeating invite, so you should have been added to 
all future DFDL calls. That obviously didn't work, so how about we choose 
Tues 8th May?  I will write up a more formal review.

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848

From:   Michele Zundo <michele.zundo at esa.int>
To:     Steve Hanson/UK/IBM at IBMGB
Date:   15/04/2012 11:10
Subject:        Re: [DFDL-WG] DFDL based application released

Dear Steve,

I got reply to your comments from our contractor. We might discuss this at 
the next call (I can try to involve the developer) or prior to 
that (not next week as I'm fully booked).
btw I'm very happy that you are giving such high quality detailed review, 
hopefully I can find enough resources on our side to pursue them as 
necessary.

Michele

Point 1:
 is a documentation error. 
the "VERY old" DFDL standard comment seems to come from a documentation 
error in S2G-DME-TEC-TNO014-1B.pdf. Table 2 should refer to [RD.1] Data 
Format Description Language (DFDL) v1.0 Specification (January 31,2011), 
as this is the DFDL standard used to support S2G development (this is also 
the latest version of the document available at http://www.ogf.org/dfdl/).

Point 2:
A brief analysis of the implementation suggests that implementing 
dfdl:format could be done with reasonable effort, This considers only the 
use of dfdl:format to specify global level properties, as implementing the 
ability to define new formats (through dfdl:defineFormat) to be referred 
by other dfdl:elements would increase complexity and thus implementation 
effort. We should understand if doing only this would be sufficient 
(taking into account we are not implementing a full DFDL arising engine) 
i.e. from current:

<?xml version="1.0" encoding="UTF-8"?> 
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dmx="
http://www.deimos.com.pt/dmx/dmx-1.0"  xmlns:dfdl="
http://www.ogf.org/dfdl/dfdl-1.0"> 

    <xs:include schemaLocation="EarthCARES-BandTMCADUTypes.xsd"/> 

    <xs:element name="CADU" type="TypeCADU" dfdl:encoding="utf-8" 
dfdl:byteOrder="bigEndian"/> 

</xs:schema>

to the following (added a new annotation at xs:schema level, containing 
the encoding and byteOrder properties; and removed the properties from the 
CADU element)

<?xml version="1.0" encoding="UTF-8"?> 
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dmx="
http://www.deimos.com.pt/dmx/dmx-1.0"  xmlns:dfdl="
http://www.ogf.org/dfdl/dfdl-1.0/"> 

    <xs:annotation> 
        <xs:appinfo source=”http://www.ogf.org/dfdl/”> 
            <dfdl:format encoding="utf-8" byteOrder="bigEndian"/> 
        </xs:appinfo> 
    </xs:annotation>

    <xs:include schemaLocation="EarthCARES-BandTMCADUTypes.xsd"/> 

    <xs:element name="CADU" type="TypeCADU"/> 

</xs:schema>

Point 3:
not a  problem the / can be added already now with no impact to S2G 
implementation.

Point 4:
DFDL parsing  implementation S2G is simplified for the application needs 
and not fully generic.
In the case of dfdl:lengthKind="expression", it was a way to circumvent 
the "naive" DFDL expressions implementation setup for S2G. At some point I 
considered allowing dfdl:lengthKind="explicit" for expression-like 
lengths, but given the available time it was not feasible.
In terms of schemas, the only change would be to replace expression by 
explicit. In terms of implementation, a quick-fix is still possible by 
adding a few hard-coded evaluation rules. We instead would prefer to 
design a full evaluation engine, in order to improve the coverage of the 
DFDL expressions language which however needs much more effort.

Regards

Michele

PS I have only received one invitation for the 10th April (in the past). 
When are the next one sent ?

On Apr 11, 2012, at 12:09 , Steve Hanson wrote:

Hi Michele 

That's interesting. The Proposed Recommendation version was Jan 2011 
(draft 45). The version used by your contractor was (according to 
http://eop-cfi.esa.int/S2G/DOCS/S2G-DME-TEC-TNO014-1B.pdf) dated 
12/03/2010 (draft 40).  The differences between the drafts 40 & 45 are not 
that great, so it should not have made much difference. It explains issue 
3, but not issues 1, 2, 4.   

That's all for now, I look forward to hearing from you when you are back 
from vacation. 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 

From:        Michele Zundo <michele.zundo at esa.int> 
To:        Steve Hanson/UK/IBM at IBMGB 
Date:        10/04/2012 18:40 
Subject:        Re: [DFDL-WG] DFDL based application released 

Steven, 

I will be happy to discuss the changes and take them into account in our 
next contract as much as possible. 

When we wrote the SOW for the activity (around beginning of 2011 so not so 
long ago after all) I assume that was the version available on ogf web 
site. Take into account I am not the developer who coded but I will 
forward your comments to my contractor to see how to proceed. Take also 
into account that we can ingest evolution of DFDL only when we have a 
contract ongoing. 
I will explain what we can (and cannot do ) at the call. 

Regards 

Inviato da iPhone 

Il giorno 10 Apr 2012, alle ore 18:51, Steve Hanson <smh at uk.ibm.com> ha 
scritto:

Hi Michele 

I have looked at your documentation and example xsds.  We will discuss in 
more detail on the call, but my main observation is that the version of 
the DFDL specification you were using is VERY old. The DFDL language has 
changed in some important ways since that draft, so the way you use DFDL 
in S2G is not correct.   

The key issues are: 
1. DFDL does not have defaults for its properties. You must not use 
defaults, instead use dfdl:format to apply values across all the xsd. 
2. You are using dfdl:encoding and dfdl:byteOrder on complex elements to 
control the encoding and byte order of child elements. This is not how 
DFDL works, you need to use dfdl:format. 
3. The namespace for DFDL in the xsds is incorrect - it is missing a 
trailing slash - "http://www.ogf.org/dfdl/dfdl-1.0/" 
4. There is no such property value as dfdl:lengthKind="expression" - it is 
just "explicit" (we know it is an expression because of {..} ) 

Please can you add a disclaimer to the S2G documentation as soon as 
possible, to clarify that S2G is based on a draft version of DFDL from 
2010 and not the OGF Proposed Recommendation? 

On the call we can discuss ways to make your implementation more 
conformant with the OGF PR version of the spec. 

Thanks

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 

From:        Michele Zundo <michele.zundo at esa.int> 
To:        Steve Hanson/UK/IBM at IBMGB 
Date:        10/04/2012 12:19 
Subject:        Re: [DFDL-WG] DFDL based application released 

That is great. I will confim when I'm back at work next week with my 
agenda in front.

Inviato da iPhone 

Il giorno 10 Apr 2012, alle ore 11:29, Steve Hanson <smh at uk.ibm.com> ha 
scritto:

Hi Michele 

When I said "join a DFDL WG call" I meant just dialling into one of our 
weekly calls, and explaining to the WG what you have created.   

If you want to subsequently participate in the WG then that's great, but 
that's a separate issue and entirely up to you. 

I have added you as an invitee to the DFDL WG weekly calls.  Please accept 
those and choose one that suits you (other than Tues April 24th & May 
1st). 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 

From:        Michele Zundo <michele.zundo at esa.int> 
To:        Steve Hanson/UK/IBM at IBMGB 
Date:        06/04/2012 14:08 
Subject:        Re: [DFDL-WG] DFDL based application released 

I would prefer having a phone chat first to understand the scope of the wg 
activities to see if/how I can contribute. 
Arranging meeting at such short notice with me being away for next week or 
so  is bit problematic. 

Anyway when and where is the next meeting ?

Inviato da iPhone 

Il giorno 6 Apr 2012, alle ore 14:20, Steve Hanson <smh at uk.ibm.com> ha 
scritto:

Hi Michele 

Are you back on Wed 18th April itself?  If so we could move the DFDL WG 
call that week to Wed. I have a meeting clash so that would help me too. 
If that does not work then we are looking at Tues May 8th, because I am in 
the USA on Tues April 24th & May 1st. 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 

From:        Michele Zundo <michele.zundo at esa.int> 
To:        Steve Hanson/UK/IBM at IBMGB 
Cc:        "dfdl-wg at ogf.org" <dfdl-wg at ogf.org> 
Date:        06/04/2012 11:12 
Subject:        Re: [DFDL-WG] DFDL based application released 

Dear Steve, 

I would be happy to participate taking into account I have quite limited 
availability of time since this il not my main work line. 

This week I'm on Easter holiday but we could talk after I'm back ( after 
18 Apr) to make some planning. Shall we phone then ?

Inviato da iPhone 

Il giorno 5 Apr 2012, alle ore 19:26, Steve Hanson <smh at uk.ibm.com> ha 
scritto:

Hi Michele 

Thanks for letting the DFDL WG know about your DFDL based application. 
It's great that DFDL is being adopted in applications.   

Would you like to join a DFDL WG call in the near future, for a discussion 
on what you have created, and for us to feedback our findings? 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh at uk.ibm.com
tel:+44-1962-815848 

From:        Michele Zundo <michele.zundo at esa.int> 
To:        dfdl-wg at ogf.org 
Date:        05/04/2012 13:43 
Subject:        [DFDL-WG] DFDL based application released 
Sent by:        dfdl-wg-bounces at ogf.org 

Dear all, 

you might be interested to know that yesterday we have freely released an 
application for visualizing binary satellite data (compliant to CCSDS 
standard) 
which based on DFDL. 

The satellite data binary structure for a number of ESA Earth Observation 
Mission 
has been described using DFDL schemas and a java parser implemented which 
allow the application to access all different data fields. 
The binary DFDL schemas can be also edited by the user to describe custom 
type 
of satellite data (i.e. changing the structure) without changing the 
application. 

We are considering in future to release the java Read API stand alone for 
everybody to 
use and build their own applications. 

The release message with pointers to URL to download the application is 
attached 
here below: 

***** 

Dear all, 

please be informed today we have released the (Space-to-Ground) S2G Data 
Viewer which 
can be freely downloaded from : 

http://eop-cfi.esa.int/S2G/s2g.html 

S2G data Viewer is a modern, multi platform, extendable application to 
inspect CCSDS data (CADU, TF/VCDU and SP) 
and example screenshot attached below. 

The application is currently configured to support the following ESA Earth 
Observation Mission 
(including the custom Source Packet Data Header): 

Sentinel-1 
Sentinel-2 
Sentinel-3 
Swarm 
EarthCARE 

and it can be extended by the user to support different type of data or 
new missions. 

I invite you to download and use it to support your AIV activities and, if 
necessary, to send any 
feedback to s2g at eopp.esa.int 

****** 

----------------------------------------- 
Mr. Michele Zundo 

Head of Ground System Definition and Verification Office 
EOP-PEP 
European Space Agency, ESTEC 
e-mail: michele.zundo at esa.int 

This message and any attachments are intended for the use of the addressee 
or addressees only. The unauthorised disclosure, use, dissemination or 
copying (either in whole or in part) of its content is not permitted. If 
you received this message in error, please notify the sender and delete it 
from your system. Emails can be altered and their integrity cannot be 
guaranteed by the sender.

Please consider the environment before printing this email.
--
dfdl-wg mailing list
dfdl-wg at ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg 