[DFDL-WG] My understanding of binary encodings in DFDL

Thu Apr 27 22:01:31 EDT 2017

Mike

That's great, many thanks! I was not expecting such elaborate feedback!

The schema generator is a great idea! I'll work on that.

Most sincerely,

Roger

--------------------------------------------------------------
Roger Champagne, P.Eng., Ph. D.

Professeur agrégé, Département de génie logiciel et des TI
Associate Professor, Department of Software and IT Engineering
École de technologie supérieure
1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada

Bureau / Office : A-4499

Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / Fax
: +1 514 396 8405 <(514)%20396-8405>

Web: http://etsmtl.ca/Professeurs/rchampagne

Le 27 avr. 2017 12:21 PM, "Mike Beckerle" <mbeckerle.dfdl at gmail.com> a
écrit :

> Roger,
>
> This is an interesting format.
>
> I have found that DFDL newbies frequently have very difficult formats to
> deal with. They end up on the cutting edge of DFDL language features.
> Basically, one of the first things people have to explore is whether DFDL
> can really handle their data problems. So hard cases first.
>
> But anyway, to your use case....
>
> The first way I started to model this is as a vector of bits that are the
> physical representation, and a computed value (dfdl:inputValueCalc).
>
> Something like this:
>
> ...assume the schema has dfdl:lengthUnits='bits'
>
> <dfdl:defineVariable name="Range173" type="xs:double">0.4</dfdl:def
> ineVariable>
>
> <sequence>
>    <element name="bits" minOccurs="12" maxOccurs="12">
>       <sequence>
>          <element name="bit" type="xs:unsignedInt" dfdl:length="1"
>               dfdl:outputValueCalc="{ .... omit for now. ...}" />
>       </sequence>
>   </element>
>   <element name="value" type="xs:double"
>      dfdl:inputValueCalc="{
>           ( ../bits[1]/bit * $Range173 / 2.0) +
>           ( ../bits[2]/bit * $Range173 / 4.0) +
>            ...
>            ( ../bits[12]/bit * ... you get the idea...range divided by the
> right number here...)
>        }" />
> </sequence>
>
> This is doable, but really isn't that great a way to model this. Lots of
> overhead. It has the advantage of being able to give the bits any weighting
> factors at all. Note that I omitted the dfdl:outputValueCalc needed for
> unparsing the format above. That's also doable, but a little painful.
>
> But the right way is to model this as an unsigned integer of length 12
> (bits), and then convert to a double float and multiply by the right scale
> factor.
>
> For your message example labeled 173 the value of the double is
>
> ( ( x * sign) / 4096.0) * 0.4
>
> where sign is -1 or 1 depending on the sign bit value.
>
> That formula is easily inverted (for the unparse direction), so you end up
> with this schema having raw integer and value floating point elements.
>
> <sequence>
>   <element name="raw" type="xs:unsignedInt" dfdl:length="12"
>       dfdl:outputValueCalc="{ (../value / 0.4) * 4096.0 * (if (../value lt
> 0.0) -1.0 else 1.0) }"/>
>   <element name="value" type="xs:double"
>      dfdl:inputValueCalc="{ ((../raw * (if (../sign eq 1) -1.0 else 1.0) )
> / 4096.0) * 0.4) }"/>
> </sequence>
>
> You'd also need a dfdl:outputValueCalc on the sign element, which is
> elsewhere in the data, so
> that it takes on 1 or 0 based on the sign of the value element when
> unparsing.
>
> This is not as generic as you would like. You need one of these for every
> range e.g., the 0.4 above, and for every bits length (12 above), but the
> scheme is clear, and at runtime this would be very efficient.
>
> If you have many of these combinations of different lengths and ranges,
> then I'd whip up a little DFDL schema generator that just creates all the
> combinations from a common template.
>
> I hope that helps.
>
> -Mike Beckerle
>
>
> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology |
> www.tresys.com
> Please note: Contributions to the DFDL Workgroup's email discussions are
> subject to the OGF Intellectual Property Policy
> <http://www.ogf.org/About/abt_policies.php>
>
>
> On Tue, Apr 25, 2017 at 9:11 PM, Champagne, Roger <
> roger.champagne at etsmtl.net> wrote:
>
>> Steve, Mike,
>>
>> I just shared a spreadsheet illustrating the encoding/decoding of two
>> different ARINC429 messages, you should have received the link through
>> Google.
>>
>> The data portion of both messages is 12 bits long, but have different
>> maximum values or ranges. Both messages admit positive and negative values.
>>
>> Encoding of floats in ARINC429 follows a few simple rules:
>>
>>    - a maximum value and number of bits are established for the data;
>>    - the most significant bit represents one half of the maximum value;
>>    - the value of each successive bit is one half of the previous;
>>    - negative values are represented in two's complement. The sign is
>>    contained elsewhere in the 32 bit word and is not included in the data
>>    shown. I am showing the binary values for both positive and negative values
>>    for each of the two messages. I produced the twos complement values by
>>    hand, having not figured out how to do that automatically in a spreadsheet
>>    (bitwise binary addition).
>>
>> In an ideal world, we would like to be able to specify in DFDL the
>> maximum value (float) and number of bits for each message, and the encoding
>> and decoding would be performed the same way for all messages, based on
>> these two parameters.
>>
>> If anything is unclear, please let me know. I most appreciate your help
>> with this.
>>
>> Sincerely,
>>
>> Roger
>>
>>
>> --------------------------------------------------------------
>> Roger Champagne, P.Eng., Ph. D.
>> Professeur agrégé, Département de génie logiciel et des TI
>> Associate Professor, Department of Software and IT Engineering
>>
>> École de technologie supérieure
>> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada
>> Bureau / Office : A-4499
>> Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur /
>> Fax : +1 514 396 8405 <(514)%20396-8405>
>> Web: http://etsmtl.ca/Professeurs/rchampagne
>>
>> On 25 April 2017 at 05:46, Steve Hanson <smh at uk.ibm.com> wrote:
>>
>>> Roger
>>>
>>> Please can you supply a couple of concrete examples - for each, the
>>> ARINC 429 message, what the bits mean, and what you want to appear in the
>>> infoset.
>>>
>>> Regards
>>>
>>> Steve Hanson
>>>
>>> IBM Hybrid Integration, Hursley, UK
>>> Architect, *IBM DFDL*
>>> <http://www.ibm.com/developerworks/library/se-dfdl/index.html>
>>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/>
>>> *smh at uk.ibm.com* <smh at uk.ibm.com>
>>> tel:+44-1962-815848 <+44%201962%20815848>
>>> mob:+44-7717-378890 <+44%207717%20378890>
>>>
>>>
>>>
>>> From:        "Champagne, Roger" <roger.champagne at etsmtl.net>
>>> To:        Steve Hanson/UK/IBM at IBMGB
>>> Cc:        smhdfdl at gmail.com, mbeckerle.dfdl at gmail.com
>>> Date:        23/04/2017 15:50
>>> Subject:        Re: My understanding of binary encodings in DFDL
>>> ------------------------------
>>>
>>>
>>>
>>> Steve,
>>>
>>> Thanks for the swift reply.
>>>
>>> I am a DFDL newbie, so not familiar with the concepts you propose. As
>>> there are hundreds of ARINC 429 messages, and each has its own range of
>>> values and number of bits, I am guessing this is a bit complicated.
>>>
>>> Ideally, we would be able to specify the number of bits, plus one of
>>> either range of values or value of most/least significant bit, and that
>>> would be it. If this is not directly supported but can be done through
>>> other means such as expression language, that would be great.
>>>
>>> Otherwise, this is kind of a show stopper for us. We should have caught
>>> this sooner.
>>>
>>> Sincerely,
>>>
>>> Roger
>>>
>>> --------------------------------------------------------------
>>> Roger Champagne, P.Eng., Ph. D.
>>>
>>> Professeur agrégé, Département de génie logiciel et des TI
>>> Associate Professor, Department of Software and IT Engineering
>>> École de technologie supérieure
>>> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada
>>>
>>> Bureau / Office : A-4499
>>>
>>> Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur /
>>> Fax : +1 514 396 8405 <(514)%20396-8405>
>>>
>>> Web: *http://etsmtl.ca/Professeurs/rchampagne*
>>> <http://etsmtl.ca/Professeurs/rchampagne>
>>>
>>>
>>> Le 23 avr. 2017 3:13 AM, "Steve Hanson" <*smh at uk.ibm.com*
>>> <smh at uk.ibm.com>> a écrit :
>>> Hi Michael
>>>
>>> Thanks for reaching out.  Direct contact is fine but better to post to
>>> the DFDL mailing list *dfdl-wg at ogf.org* <dfdl-wg at ogf.org>.
>>>
>>> DFDL supports binary integers of arbitrary length, but not binary
>>> floats.
>>>
>>> Could you treat the mantissa and exponent as integers inside a hidden
>>> group, then use inputValueCalc to fabricate a float using the DFDL
>>> expression language?
>>>
>>> Michael has more experience with calculated values and may be able to
>>> offer more advice.
>>>
>>> Regards
>>>
>>> Steve Hanson
>>> IBM Hybrid Integration, Hursley, UK
>>> Architect, *IBM DFDL*
>>> <http://www.ibm.com/developerworks/library/se-dfdl/index.html>
>>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/>
>>> *smh at uk.ibm.com* <smh at uk.ibm.com>
>>> tel:*+44-1962-815848* <+44%201962%20815848>
>>> mob:*+44-7717-378890* <+44%207717%20378890>
>>>
>>>
>>>
>>> From:        "Champagne, Roger" <*roger.champagne at etsmtl.net*
>>> <roger.champagne at etsmtl.net>>
>>> To:        *mbeckerle.dfdl at gmail.com* <mbeckerle.dfdl at gmail.com>,
>>> *smhdfdl at gmail.com* <smhdfdl at gmail.com>
>>> Date:        22/04/2017 22:24
>>> Subject:        My understanding of binary encodings in DFDL
>>> ------------------------------
>>>
>>>
>>>
>>> Michael, Steve,
>>>
>>> I am not sure this is the proper way to proceed, but I figured I would
>>> give it a try. Please redirect me to the proper channel if relevant.
>>>
>>> I am working on an avionics project where we are trying to describe
>>> ARINC429 messages in DFDL. After a couple of months of a student working on
>>> this and requesting my assistance, I realize DFDL is perhaps not a good
>>> match for our needs.
>>>
>>> My understanding, based on section 13.8 of the specification, is that
>>> for floating-point values represented in binary, only two encodings are
>>> supported in DFDL: IEEE 754 and IBM 390 HEX. In both cases, the data must
>>> be exactly 32 bits (4 bytes) long.
>>>
>>> * Question 1*: Is this assumption exact?
>>>
>>> The reason I ask is that ARINC429 messages typically consist in a single
>>> 32 bit word. The message data is a subset of the 32 bits, and has variable
>>> length (up to 18 bits), based on the nature of the message. The way the
>>> data is encoded is different for each message, based of the range of values
>>> for that message and the number of bits it uses. We are try to represent
>>> these messages in DFDL.
>>>
>>> * Question 2: *If my assumption is correct, are there any plans to
>>> support in DFDL binary encodings of arbitrary length, based on the
>>> knowledge of number of bits and range of admissible values?
>>>
>>> I can supply more details if required, I tried to keep this initial
>>> message short.
>>>
>>> Thanks for any light you can shed on this matter.
>>>
>>> Most sincerely,
>>>
>>> Roger
>>>
>>> --------------------------------------------------------------
>>> Roger Champagne, P.Eng., Ph. D.
>>> Professeur agrégé, Département de génie logiciel et des TI
>>> Associate Professor, Department of Software and IT Engineering
>>>
>>> École de technologie supérieure
>>> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada
>>> Bureau / Office : A-4499
>>> Téléphone / Phone : *+1 514 396 8825* <(514)%20396-8825> -- Télécopieur
>>> / Fax : *+1 514 396 8405* <(514)%20396-8405>
>>> Web: *http://etsmtl.ca/Professeurs/rchampagne*
>>> <http://etsmtl.ca/Professeurs/rchampagne>
>>>
>>> Unless stated otherwise above:
>>> IBM United Kingdom Limited - Registered in England and Wales with number
>>> 741598.
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>>> 3AU
>>>
>>> Unless stated otherwise above:
>>> IBM United Kingdom Limited - Registered in England and Wales with number
>>> 741598.
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>>> 3AU
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/dfdl-wg/attachments/20170427/275abaed/attachment-0001.html>