- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] My understanding of binary encodings in DFDL
by Champagne, Roger 28 Apr '17

28 Apr '17

Mike That's great, many thanks! I was not expecting such elaborate feedback! The schema generator is a great idea! I'll work on that. Most sincerely, Roger -------------------------------------------------------------- Roger Champagne, P.Eng., Ph. D. Professeur agrégé, Département de génie logiciel et des TI Associate Professor, Department of Software and IT Engineering École de technologie supérieure 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada Bureau / Office : A-4499 Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / Fax : +1 514 396 8405 <(514)%20396-8405> Web: http://etsmtl.ca/Professeurs/rchampagne Le 27 avr. 2017 12:21 PM, "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> a écrit : > Roger, > > This is an interesting format. > > I have found that DFDL newbies frequently have very difficult formats to > deal with. They end up on the cutting edge of DFDL language features. > Basically, one of the first things people have to explore is whether DFDL > can really handle their data problems. So hard cases first. > > But anyway, to your use case.... > > The first way I started to model this is as a vector of bits that are the > physical representation, and a computed value (dfdl:inputValueCalc). > > Something like this: > > ...assume the schema has dfdl:lengthUnits='bits' > > <dfdl:defineVariable name="Range173" type="xs:double">0.4</dfdl:def > ineVariable> > > <sequence> > <element name="bits" minOccurs="12" maxOccurs="12"> > <sequence> > <element name="bit" type="xs:unsignedInt" dfdl:length="1" > dfdl:outputValueCalc="{ .... omit for now. ...}" /> > </sequence> > </element> > <element name="value" type="xs:double" > dfdl:inputValueCalc="{ > ( ../bits[1]/bit * $Range173 / 2.0) + > ( ../bits[2]/bit * $Range173 / 4.0) + > ... > ( ../bits[12]/bit * ... you get the idea...range divided by the > right number here...) > }" /> > </sequence> > > This is doable, but really isn't that great a way to model this. Lots of > overhead. It has the advantage of being able to give the bits any weighting > factors at all. Note that I omitted the dfdl:outputValueCalc needed for > unparsing the format above. That's also doable, but a little painful. > > But the right way is to model this as an unsigned integer of length 12 > (bits), and then convert to a double float and multiply by the right scale > factor. > > For your message example labeled 173 the value of the double is > > ( ( x * sign) / 4096.0) * 0.4 > > where sign is -1 or 1 depending on the sign bit value. > > That formula is easily inverted (for the unparse direction), so you end up > with this schema having raw integer and value floating point elements. > > <sequence> > <element name="raw" type="xs:unsignedInt" dfdl:length="12" > dfdl:outputValueCalc="{ (../value / 0.4) * 4096.0 * (if (../value lt > 0.0) -1.0 else 1.0) }"/> > <element name="value" type="xs:double" > dfdl:inputValueCalc="{ ((../raw * (if (../sign eq 1) -1.0 else 1.0) ) > / 4096.0) * 0.4) }"/> > </sequence> > > You'd also need a dfdl:outputValueCalc on the sign element, which is > elsewhere in the data, so > that it takes on 1 or 0 based on the sign of the value element when > unparsing. > > This is not as generic as you would like. You need one of these for every > range e.g., the 0.4 above, and for every bits length (12 above), but the > scheme is clear, and at runtime this would be very efficient. > > If you have many of these combinations of different lengths and ranges, > then I'd whip up a little DFDL schema generator that just creates all the > combinations from a common template. > > I hope that helps. > > -Mike Beckerle > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > www.tresys.com > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the OGF Intellectual Property Policy > <http://www.ogf.org/About/abt_policies.php> > > > On Tue, Apr 25, 2017 at 9:11 PM, Champagne, Roger < > roger.champagne(a)etsmtl.net> wrote: > >> Steve, Mike, >> >> I just shared a spreadsheet illustrating the encoding/decoding of two >> different ARINC429 messages, you should have received the link through >> Google. >> >> The data portion of both messages is 12 bits long, but have different >> maximum values or ranges. Both messages admit positive and negative values. >> >> Encoding of floats in ARINC429 follows a few simple rules: >> >> - a maximum value and number of bits are established for the data; >> - the most significant bit represents one half of the maximum value; >> - the value of each successive bit is one half of the previous; >> - negative values are represented in two's complement. The sign is >> contained elsewhere in the 32 bit word and is not included in the data >> shown. I am showing the binary values for both positive and negative values >> for each of the two messages. I produced the twos complement values by >> hand, having not figured out how to do that automatically in a spreadsheet >> (bitwise binary addition). >> >> In an ideal world, we would like to be able to specify in DFDL the >> maximum value (float) and number of bits for each message, and the encoding >> and decoding would be performed the same way for all messages, based on >> these two parameters. >> >> If anything is unclear, please let me know. I most appreciate your help >> with this. >> >> Sincerely, >> >> Roger >> >> >> -------------------------------------------------------------- >> Roger Champagne, P.Eng., Ph. D. >> Professeur agrégé, Département de génie logiciel et des TI >> Associate Professor, Department of Software and IT Engineering >> >> École de technologie supérieure >> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >> Bureau / Office : A-4499 >> Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / >> Fax : +1 514 396 8405 <(514)%20396-8405> >> Web: http://etsmtl.ca/Professeurs/rchampagne >> >> On 25 April 2017 at 05:46, Steve Hanson <smh(a)uk.ibm.com> wrote: >> >>> Roger >>> >>> Please can you supply a couple of concrete examples - for each, the >>> ARINC 429 message, what the bits mean, and what you want to appear in the >>> infoset. >>> >>> Regards >>> >>> Steve Hanson >>> >>> IBM Hybrid Integration, Hursley, UK >>> Architect, *IBM DFDL* >>> <http://www.ibm.com/developerworks/library/se-dfdl/index.html> >>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >>> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >>> tel:+44-1962-815848 <+44%201962%20815848> >>> mob:+44-7717-378890 <+44%207717%20378890> >>> >>> >>> >>> From: "Champagne, Roger" <roger.champagne(a)etsmtl.net> >>> To: Steve Hanson/UK/IBM@IBMGB >>> Cc: smhdfdl(a)gmail.com, mbeckerle.dfdl(a)gmail.com >>> Date: 23/04/2017 15:50 >>> Subject: Re: My understanding of binary encodings in DFDL >>> ------------------------------ >>> >>> >>> >>> Steve, >>> >>> Thanks for the swift reply. >>> >>> I am a DFDL newbie, so not familiar with the concepts you propose. As >>> there are hundreds of ARINC 429 messages, and each has its own range of >>> values and number of bits, I am guessing this is a bit complicated. >>> >>> Ideally, we would be able to specify the number of bits, plus one of >>> either range of values or value of most/least significant bit, and that >>> would be it. If this is not directly supported but can be done through >>> other means such as expression language, that would be great. >>> >>> Otherwise, this is kind of a show stopper for us. We should have caught >>> this sooner. >>> >>> Sincerely, >>> >>> Roger >>> >>> -------------------------------------------------------------- >>> Roger Champagne, P.Eng., Ph. D. >>> >>> Professeur agrégé, Département de génie logiciel et des TI >>> Associate Professor, Department of Software and IT Engineering >>> École de technologie supérieure >>> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >>> >>> Bureau / Office : A-4499 >>> >>> Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / >>> Fax : +1 514 396 8405 <(514)%20396-8405> >>> >>> Web: *http://etsmtl.ca/Professeurs/rchampagne* >>> <http://etsmtl.ca/Professeurs/rchampagne> >>> >>> >>> Le 23 avr. 2017 3:13 AM, "Steve Hanson" <*smh(a)uk.ibm.com* >>> <smh(a)uk.ibm.com>> a écrit : >>> Hi Michael >>> >>> Thanks for reaching out. Direct contact is fine but better to post to >>> the DFDL mailing list *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>. >>> >>> DFDL supports binary integers of arbitrary length, but not binary >>> floats. >>> >>> Could you treat the mantissa and exponent as integers inside a hidden >>> group, then use inputValueCalc to fabricate a float using the DFDL >>> expression language? >>> >>> Michael has more experience with calculated values and may be able to >>> offer more advice. >>> >>> Regards >>> >>> Steve Hanson >>> IBM Hybrid Integration, Hursley, UK >>> Architect, *IBM DFDL* >>> <http://www.ibm.com/developerworks/library/se-dfdl/index.html> >>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >>> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >>> tel:*+44-1962-815848* <+44%201962%20815848> >>> mob:*+44-7717-378890* <+44%207717%20378890> >>> >>> >>> >>> From: "Champagne, Roger" <*roger.champagne(a)etsmtl.net* >>> <roger.champagne(a)etsmtl.net>> >>> To: *mbeckerle.dfdl(a)gmail.com* <mbeckerle.dfdl(a)gmail.com>, >>> *smhdfdl(a)gmail.com* <smhdfdl(a)gmail.com> >>> Date: 22/04/2017 22:24 >>> Subject: My understanding of binary encodings in DFDL >>> ------------------------------ >>> >>> >>> >>> Michael, Steve, >>> >>> I am not sure this is the proper way to proceed, but I figured I would >>> give it a try. Please redirect me to the proper channel if relevant. >>> >>> I am working on an avionics project where we are trying to describe >>> ARINC429 messages in DFDL. After a couple of months of a student working on >>> this and requesting my assistance, I realize DFDL is perhaps not a good >>> match for our needs. >>> >>> My understanding, based on section 13.8 of the specification, is that >>> for floating-point values represented in binary, only two encodings are >>> supported in DFDL: IEEE 754 and IBM 390 HEX. In both cases, the data must >>> be exactly 32 bits (4 bytes) long. >>> >>> * Question 1*: Is this assumption exact? >>> >>> The reason I ask is that ARINC429 messages typically consist in a single >>> 32 bit word. The message data is a subset of the 32 bits, and has variable >>> length (up to 18 bits), based on the nature of the message. The way the >>> data is encoded is different for each message, based of the range of values >>> for that message and the number of bits it uses. We are try to represent >>> these messages in DFDL. >>> >>> * Question 2: *If my assumption is correct, are there any plans to >>> support in DFDL binary encodings of arbitrary length, based on the >>> knowledge of number of bits and range of admissible values? >>> >>> I can supply more details if required, I tried to keep this initial >>> message short. >>> >>> Thanks for any light you can shed on this matter. >>> >>> Most sincerely, >>> >>> Roger >>> >>> -------------------------------------------------------------- >>> Roger Champagne, P.Eng., Ph. D. >>> Professeur agrégé, Département de génie logiciel et des TI >>> Associate Professor, Department of Software and IT Engineering >>> >>> École de technologie supérieure >>> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >>> Bureau / Office : A-4499 >>> Téléphone / Phone : *+1 514 396 8825* <(514)%20396-8825> -- Télécopieur >>> / Fax : *+1 514 396 8405* <(514)%20396-8405> >>> Web: *http://etsmtl.ca/Professeurs/rchampagne* >>> <http://etsmtl.ca/Professeurs/rchampagne> >>> >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >>> 3AU >>> >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >>> 3AU >>> >>> >> >

1 0

Re: [DFDL-WG] My understanding of binary encodings in DFDL
by Mike Beckerle 27 Apr '17

27 Apr '17

Roger, This is an interesting format. I have found that DFDL newbies frequently have very difficult formats to deal with. They end up on the cutting edge of DFDL language features. Basically, one of the first things people have to explore is whether DFDL can really handle their data problems. So hard cases first. But anyway, to your use case.... The first way I started to model this is as a vector of bits that are the physical representation, and a computed value (dfdl:inputValueCalc). Something like this: ...assume the schema has dfdl:lengthUnits='bits' <dfdl:defineVariable name="Range173" type="xs:double">0.4</dfdl: defineVariable> <sequence> <element name="bits" minOccurs="12" maxOccurs="12"> <sequence> <element name="bit" type="xs:unsignedInt" dfdl:length="1" dfdl:outputValueCalc="{ .... omit for now. ...}" /> </sequence> </element> <element name="value" type="xs:double" dfdl:inputValueCalc="{ ( ../bits[1]/bit * $Range173 / 2.0) + ( ../bits[2]/bit * $Range173 / 4.0) + ... ( ../bits[12]/bit * ... you get the idea...range divided by the right number here...) }" /> </sequence> This is doable, but really isn't that great a way to model this. Lots of overhead. It has the advantage of being able to give the bits any weighting factors at all. Note that I omitted the dfdl:outputValueCalc needed for unparsing the format above. That's also doable, but a little painful. But the right way is to model this as an unsigned integer of length 12 (bits), and then convert to a double float and multiply by the right scale factor. For your message example labeled 173 the value of the double is ( ( x * sign) / 4096.0) * 0.4 where sign is -1 or 1 depending on the sign bit value. That formula is easily inverted (for the unparse direction), so you end up with this schema having raw integer and value floating point elements. <sequence> <element name="raw" type="xs:unsignedInt" dfdl:length="12" dfdl:outputValueCalc="{ (../value / 0.4) * 4096.0 * (if (../value lt 0.0) -1.0 else 1.0) }"/> <element name="value" type="xs:double" dfdl:inputValueCalc="{ ((../raw * (if (../sign eq 1) -1.0 else 1.0) ) / 4096.0) * 0.4) }"/> </sequence> You'd also need a dfdl:outputValueCalc on the sign element, which is elsewhere in the data, so that it takes on 1 or 0 based on the sign of the value element when unparsing. This is not as generic as you would like. You need one of these for every range e.g., the 0.4 above, and for every bits length (12 above), but the scheme is clear, and at runtime this would be very efficient. If you have many of these combinations of different lengths and ranges, then I'd whip up a little DFDL schema generator that just creates all the combinations from a common template. I hope that helps. -Mike Beckerle Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Apr 25, 2017 at 9:11 PM, Champagne, Roger < roger.champagne(a)etsmtl.net> wrote: > Steve, Mike, > > I just shared a spreadsheet illustrating the encoding/decoding of two > different ARINC429 messages, you should have received the link through > Google. > > The data portion of both messages is 12 bits long, but have different > maximum values or ranges. Both messages admit positive and negative values. > > Encoding of floats in ARINC429 follows a few simple rules: > > - a maximum value and number of bits are established for the data; > - the most significant bit represents one half of the maximum value; > - the value of each successive bit is one half of the previous; > - negative values are represented in two's complement. The sign is > contained elsewhere in the 32 bit word and is not included in the data > shown. I am showing the binary values for both positive and negative values > for each of the two messages. I produced the twos complement values by > hand, having not figured out how to do that automatically in a spreadsheet > (bitwise binary addition). > > In an ideal world, we would like to be able to specify in DFDL the maximum > value (float) and number of bits for each message, and the encoding and > decoding would be performed the same way for all messages, based on these > two parameters. > > If anything is unclear, please let me know. I most appreciate your help > with this. > > Sincerely, > > Roger > > > -------------------------------------------------------------- > Roger Champagne, P.Eng., Ph. D. > Professeur agrégé, Département de génie logiciel et des TI > Associate Professor, Department of Software and IT Engineering > > École de technologie supérieure > 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada > Bureau / Office : A-4499 > Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / > Fax : +1 514 396 8405 <(514)%20396-8405> > Web: http://etsmtl.ca/Professeurs/rchampagne > > On 25 April 2017 at 05:46, Steve Hanson <smh(a)uk.ibm.com> wrote: > >> Roger >> >> Please can you supply a couple of concrete examples - for each, the ARINC >> 429 message, what the bits mean, and what you want to appear in the infoset. >> >> Regards >> >> Steve Hanson >> >> IBM Hybrid Integration, Hursley, UK >> Architect, *IBM DFDL* >> <http://www.ibm.com/developerworks/library/se-dfdl/index.html> >> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >> tel:+44-1962-815848 <+44%201962%20815848> >> mob:+44-7717-378890 <+44%207717%20378890> >> >> >> >> From: "Champagne, Roger" <roger.champagne(a)etsmtl.net> >> To: Steve Hanson/UK/IBM@IBMGB >> Cc: smhdfdl(a)gmail.com, mbeckerle.dfdl(a)gmail.com >> Date: 23/04/2017 15:50 >> Subject: Re: My understanding of binary encodings in DFDL >> ------------------------------ >> >> >> >> Steve, >> >> Thanks for the swift reply. >> >> I am a DFDL newbie, so not familiar with the concepts you propose. As >> there are hundreds of ARINC 429 messages, and each has its own range of >> values and number of bits, I am guessing this is a bit complicated. >> >> Ideally, we would be able to specify the number of bits, plus one of >> either range of values or value of most/least significant bit, and that >> would be it. If this is not directly supported but can be done through >> other means such as expression language, that would be great. >> >> Otherwise, this is kind of a show stopper for us. We should have caught >> this sooner. >> >> Sincerely, >> >> Roger >> >> -------------------------------------------------------------- >> Roger Champagne, P.Eng., Ph. D. >> >> Professeur agrégé, Département de génie logiciel et des TI >> Associate Professor, Department of Software and IT Engineering >> École de technologie supérieure >> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >> >> Bureau / Office : A-4499 >> >> Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / >> Fax : +1 514 396 8405 <(514)%20396-8405> >> >> Web: *http://etsmtl.ca/Professeurs/rchampagne* >> <http://etsmtl.ca/Professeurs/rchampagne> >> >> >> Le 23 avr. 2017 3:13 AM, "Steve Hanson" <*smh(a)uk.ibm.com* >> <smh(a)uk.ibm.com>> a écrit : >> Hi Michael >> >> Thanks for reaching out. Direct contact is fine but better to post to >> the DFDL mailing list *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>. >> >> DFDL supports binary integers of arbitrary length, but not binary floats. >> >> Could you treat the mantissa and exponent as integers inside a hidden >> group, then use inputValueCalc to fabricate a float using the DFDL >> expression language? >> >> Michael has more experience with calculated values and may be able to >> offer more advice. >> >> Regards >> >> Steve Hanson >> IBM Hybrid Integration, Hursley, UK >> Architect, *IBM DFDL* >> <http://www.ibm.com/developerworks/library/se-dfdl/index.html> >> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >> tel:*+44-1962-815848* <+44%201962%20815848> >> mob:*+44-7717-378890* <+44%207717%20378890> >> >> >> >> From: "Champagne, Roger" <*roger.champagne(a)etsmtl.net* >> <roger.champagne(a)etsmtl.net>> >> To: *mbeckerle.dfdl(a)gmail.com* <mbeckerle.dfdl(a)gmail.com>, >> *smhdfdl(a)gmail.com* <smhdfdl(a)gmail.com> >> Date: 22/04/2017 22:24 >> Subject: My understanding of binary encodings in DFDL >> ------------------------------ >> >> >> >> Michael, Steve, >> >> I am not sure this is the proper way to proceed, but I figured I would >> give it a try. Please redirect me to the proper channel if relevant. >> >> I am working on an avionics project where we are trying to describe >> ARINC429 messages in DFDL. After a couple of months of a student working on >> this and requesting my assistance, I realize DFDL is perhaps not a good >> match for our needs. >> >> My understanding, based on section 13.8 of the specification, is that for >> floating-point values represented in binary, only two encodings are >> supported in DFDL: IEEE 754 and IBM 390 HEX. In both cases, the data must >> be exactly 32 bits (4 bytes) long. >> >> * Question 1*: Is this assumption exact? >> >> The reason I ask is that ARINC429 messages typically consist in a single >> 32 bit word. The message data is a subset of the 32 bits, and has variable >> length (up to 18 bits), based on the nature of the message. The way the >> data is encoded is different for each message, based of the range of values >> for that message and the number of bits it uses. We are try to represent >> these messages in DFDL. >> >> * Question 2: *If my assumption is correct, are there any plans to >> support in DFDL binary encodings of arbitrary length, based on the >> knowledge of number of bits and range of admissible values? >> >> I can supply more details if required, I tried to keep this initial >> message short. >> >> Thanks for any light you can shed on this matter. >> >> Most sincerely, >> >> Roger >> >> -------------------------------------------------------------- >> Roger Champagne, P.Eng., Ph. D. >> Professeur agrégé, Département de génie logiciel et des TI >> Associate Professor, Department of Software and IT Engineering >> >> École de technologie supérieure >> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >> Bureau / Office : A-4499 >> Téléphone / Phone : *+1 514 396 8825* <(514)%20396-8825> -- Télécopieur >> / Fax : *+1 514 396 8405* <(514)%20396-8405> >> Web: *http://etsmtl.ca/Professeurs/rchampagne* >> <http://etsmtl.ca/Professeurs/rchampagne> >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >

1 0

OGF DFDL WG Call Minutes 2017-04-18
by Steve Hanson 19 Apr '17

19 Apr '17

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13552?download= Next call May 2nd Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 18 Apr '17

18 Apr '17

1 0

OGF DFDL WG Call Agenda 2017-04-18
by Steve Hanson 18 Apr '17

18 Apr '17

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13552?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL WG Call Minutes 2017-04-04
by Steve Hanson 04 Apr '17

04 Apr '17

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13551?download= Next call April 18th Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding
by Steve Hanson 04 Apr '17

04 Apr '17

The intent of the last sentence of erratum 3.9 is that the parser simply assumes the content of the complex element is valid text in the stated encoding, and reads the specified number of characters. If a decode error occurs, that's too bad, your model stated otherwise. So, no point in making this optional, it's straightforward to implement. Action 291 closed. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 04/04/2017 11:56 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Some light on action 291 - see the last sentence of this extract from the original errata document (experience doc 1): 3.9. Section 12.3.5, 7.3.1, 7.3.2. The spec originally allows lengthKind ‘pattern’ to be used when the representation of the current element, or of a child element, is binary, but imposes restrictions on the encoding that can be in force. Clarify that the encoding property must be defined for the element (else schema definition error), and that a decoding processing error is possible if the match of the regex encounters data that does not decode in that encoding, dependent on the setting of encodingErrorPolicy. Remove section 12.3.5.1. Same clarifications needed for testKind ”pattern” property for asserts and discriminators. For consistency, the restriction that a complex element of specified length and lengthUnits ‘characters’ must have children that are all text and that have the same encoding as the complex element, is dropped So that explains how IBM DFDL's error message CTDV1524E came about, it was policing a restriction in the original GFD.174 spec, a restriction which no longer exists. IBM DFDL has not yet implemented the erratum. It wasn't an extra IBM restriction. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 14/09/2016 08:44 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Actions 290 and 291 raised to investigate further - see minutes. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/09/2016 13:14 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Mike I am assuming that the processing for utf-16 'fixed' or 'variable' is entirely handled by ICU so there should be no coding overhead. IBM DFDL works ok for dfdl:lengthKind='explicit' for an element of complex type with dfdl:lengthUnits='characters' and dfdl:encoding="utf-8". But there are conditions the content of the complex type must satisfy otherwise an SDE results, such as: CTDV1524E : For a complex element, when 'lengthKind' is 'explicit' or 'prefixed', and 'lengthUnits' is characters, all simple child elements must have text representation, 'lengthUnits' set to 'characters' and the same encoding. So we insist that the properties of the children are consistent with the properties of the parent. If you recall, IBM DFDL does all these kinds of validation checks in a pre-processing phase. That seems a pretty sensible rule but I am not sure if the rule appears in the spec as such - I just had a quick look but didn't spot anything. So I guess I don't see a need for these things to be optional features? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 10/08/2016 18:57 Subject: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Given the limited set of required encodings for a conforming DFDL processor, I believe dfdl:utf16Width='variable' should be an optional feature. That's just consistency with what is optional already. But it is also quite hard to implement. There are other situations that are very hard to implement, probably never used by real users, yet which are non optional in the spec: I would suggest that dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding like utf-8 is very problematic to implement. I am pretty sure IBM DFDL has no implementation of this per email threads, and I know I don't want to implement this in Daffodil even though we're trying to be very comprehensive in the implementation eventually. I think implementations should be free to just not implement this. These sorts of cases often exist just because we're trying to preserve some orthogonality of composition in the language. So it's possible to do quite a few things that probably aren't ever needed by anyone, that reflect ill-defined data formats, etc. I'd rather not document a bunch of "non-conformances" for Daffodil or other implementations for these sorts of things. I'd like to say we don't implement them, but they're optional, and so that's allowed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2017-04-04
by Steve Hanson 04 Apr '17

04 Apr '17

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13551?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Confirmed: OGF DFDL Working Group call (4 Apr 15:00 GDT in IBM: Hursley DE3S09)
by Steve Hanson 04 Apr '17

04 Apr '17

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Proposed feature: lookup tables via simple type unions
by Mike Beckerle 04 Apr '17

04 Apr '17

A write up of the proposal, which we are prototyping in Daffodil, is here: https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Enumerations+a… This is needed by a number of data formats we are working with where there are large enumerations having as many as 2000 members. Often these enumerations are a mixture where single values correspond to some enumerated strings, and ranges of values correspond to others. Using expressions to translate representation integers into strings is infeasible, as no constant-time case-statement-like construct is available in DFDL. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1