- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] clarification needed - choice with both dispatch and discriminator
by Andrew Edwards 27 Jan '17

27 Jan '17

1 0

clarification needed - choice with both dispatch and discriminator
by Mike Beckerle 27 Jan '17

27 Jan '17

Section 15.1.2 says you can have both dfdl:choiceBranchKey and a discriminator on a choice branch. But it doesn't clarify what the meaning of having both is. My assumption is that if you have both, the choice is resolved by the choiceDispatchKey/choiceBranchKey, so any discriminator would be about some surrounding point of uncertainty outside the choice. I wanted to confirm this is the intention. The alternative semantics would be that the discriminator is redundant, and so if it evaluates to true that means nothing at all since the choice branch has already been discriminated by the choiceDispatchKey/choiceBranchKey. If the discriminator evaluates to false, or errors when it evaluates, that would be a parse error that would cause the entire choice to fail. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 2

suggest: need hexBinary with lengthUnits 'bits' with length not a multiple of 8.
by Mike Beckerle 27 Jan '17

27 Jan '17

We have users who have binary blobs the size of which is given in bits, and these blobs are not a multiple of 8 long. Today the DFDL spec doesn't allow hexBinary to have lengthUnits 'bits'. I am wondering if this restriction should be lifted. XSD constrains hexBinary to always have an even number of Hex digits, so we would have to do the same. So for an example, a 17 bit long hexBinary containing all 1 bits would be FFFF80 Erratum 5.15 extends the types that are allowed to have length in bits to include packed calendars. So there is precedent for opening this restriction up if need arises. I claim we need to (a) allow length units bits for all types (b) restrict the length to have to be 32-bits or 64-bits only, for types xs:float and xs:double when representation 'binary' (c) restrict packed decimal to have lengths be a multiple of 4 bits (when specified in units 'bits') All other restrictions should be lifted as those restrictions just cause problems in some formats. For example 12.3.7.2.5 Specifies that binary calendars must be 4 bytes or 8 bytes exactly, and cannot be specified in units 'bits'. This is just a mistake in DFDL. I have even seen binary calendars with 33 bits length. (seconds since 1-1-1970 representation aka binarySeconds) That additional bit extends the end time substantially. These restrictions were put into DFDL because our experience of many bit-granularity formats was limited. What we've found is that there are plenty of data formats where the notion of a "byte" is simply absent. Nothing uses multiples of 8 bits for anything, and nothing is measured in those units. It's always measured in bits. Even for things like float and double, which have impliicit lengths of 4 and 8 bytes respectively, many specifications will express those as 32 bits or 64 bits. Having to divide by 8 just makes the DFDL schema awkward. Similarly in these formats strings are given length in bits. 448 bits worth of 7-bit packed ascii characters is 64 characters, occupying 56 bytes, but the spec uses 448. These changes are all backward-compatible. They make legal property settings that previously had no meaning and caused SDEs. Discussion? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

OGF DFDL WG Call 2017-01-24 - cancelled
by Steve Hanson 24 Jan '17

24 Jan '17

Steve not available so cancelling. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 24 Jan '17

24 Jan '17

1 0

OGF DFDL WG Call Minutes 2017-10-01
by Steve Hanson 16 Jan '17

16 Jan '17

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13548?download= Next call Jan 24th. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Announcement: Daffodil now runs NACHA DFDL Schema
by Mike Beckerle 12 Jan '17

12 Jan '17

(I posted this to the news items on the OGF DFDL Workgroup pages, but here it is for those following dfdl-wg also) Tresys Technology is pleased to announce that the Daffodil <https://exchange10.columbia.tresys.com/owa/redir.aspx?C=abb4b9a398b24e08ac9…> open source DFDL implementation is now able to run the NACHA Electronic Payment messages DFDL schema <https://exchange10.columbia.tresys.com/owa/redir.aspx?C=abb4b9a398b24e08ac9…>. This demonstrates an increase in interoperability for Daffodil with DFDL schemas created and tested on the IBM DFDL implementation <https://exchange10.columbia.tresys.com/owa/redir.aspx?C=abb4b9a398b24e08ac9…> . The version of Daffodil that has this functionality is still pre-release, but the forthcoming Daffodil 2.0.0 release will incorporate this capability. (Daffodil 2.0.0 development snapshots newer than 2017-01-11 can also be used.) The NACHA Electronic Payment messages DFDL schema <https://exchange10.columbia.tresys.com/owa/redir.aspx?C=abb4b9a398b24e08ac9…> was originally created and tested by IBM in 2014. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Announcement: NITF DFDL Schema
by Steve Lawrence 12 Jan '17

12 Jan '17

Tresys Technology is pleased to announce the release of a DFDL schema to describe MIL-STD-2500C: National Imagery Transmission Format, more commonly known as NITF. The schema and test files are available for download on the DFDLSchemas github page [1]. The version of Daffodil that has the functionality to parse and unparse NITF files using this schema is still pre-release, but the forthcoming Daffodil 2.0.0 release will incorporate the capability. (Daffodil 2.0.0 development snapshots newer than 2017-01-12 can also be used.) [1] https://github.com/DFDLSchemas/NITF

1 0

Re: [DFDL-WG] spec bug dfdl:choiceLength says 'bytes' should use dfdl:lengthUnits
by Steve Hanson 10 Jan '17

10 Jan '17

https://redmine.ogf.org/issues/325 created Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 16/12/2016 09:17 Subject: Re: [DFDL-WG] spec bug dfdl:choiceLength says 'bytes' should use dfdl:lengthUnits Mike dfdl:lengthUnits is not currently used by dfdl:choice. If by erratum we changed this, existing schemas could break as they would suddenly pick up the value from scope. We have a couple of alternatives: 1) A new property dfdl:choiceLengthUnits with enums 'bytes' and 'bits' (not seen a use case for 'characters'). For compatibility, an implementation would need to default the value to 'bytes' if not set locally or in scope, rather than give an error. ** 2) New enum for dfdl:choiceLengthKind 'explicitBits' which causes the dfdl:choiceLength to be interpreted as a bit count. Avoids the default pitfall, but is architecturally inconsistent with lengthKind/length/lengthUnits. ** IBM DFDL had to do a similar thing when dfdl:encodingErrorPolicy was added to avoid breaking existing schemas. Alternative is to bump the DFDL version number. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 15/12/2016 16:04 Subject: [DFDL-WG] spec bug dfdl:choiceLength says 'bytes' should use dfdl:lengthUnits Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> for dfdl:choiceLength, the description says "Specifies the length of the choice in bytes..." clearly this should use dfdl:lengthUnits. We have tons of cases where byte granularity is not sufficient, and this is hard to work around. You have to use dfdl:choiceLengthKind='implicit' and then carefully pad the branches to the same length in bits. It's painful. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] choiceBranchKey - suggest change to List of DFDL String Literals ?
by Steve Hanson 10 Jan '17

10 Jan '17

https://redmine.ogf.org/issues/326 created Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/12/2016 18:15 Subject: Re: [DFDL-WG] choiceBranchKey - suggest change to List of DFDL String Literals ? Yes it gets ugly and inefficient when you start having a large 'if' statement. An example of an IBM-schema with direct dispatch choice is SWIFT, which has dozens of messages but they all fall into 7 patterns so the expression 'if' statement has 7 branches. Not seen anything more complex than that so far. A List of DFDL String Literals is a possibility. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/12/2016 17:34 Subject: Re: [DFDL-WG] choiceBranchKey - suggest change to List of DFDL String Literals ? Yes this situation is that the expressions are big, because there are 100's of branches, and so the constants are remote from where the branch is expressed. And it ends up being quite inefficient. What one wants is a table based lookup. Not an expression like if (../key eq "11 or ../key eq "22") then "11" else if (../key eq "AA" or ../key eq "BB" then "AA" ... so on for dozens of cases. and so on. That's just a linear chain again, so equivalent to using discriminators on the branches. An interesting point expressed to us is that users would like the DFDL schema to not only work to parse and unparse the file, but for it to serve as a declarative expression of the data format - as the machine-readable documentation of the format ultimately. While a big expression will work, it's not as declarative as listing the possible dispatch keys for each branch right on that branch. A big expression really fails to capture the essence of the format in a manner that enables efficient implementation. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Dec 13, 2016 at 11:01 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Hi Mike If I recall, choiceBranchKey was a single DFDL String Literal because the value to be matched against it is computed by a DFDL expression (from choiceDispatchKey), which can do things like lower casing strings or other manipulations. Are you seeing use cases where multiple keys per branch are making the expression too complex? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 12/12/2016 19:48 Subject: [DFDL-WG] choiceBranchKey - suggest change to List of DFDL String Literals ? Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Our first experience with requiring and implementing choiceBranchKey in Daffodil and we're already dealing with the fact that many times the same branch has multiple different keys all of which indicate that it should be selected. Right now, dfdl:choiceBranchKey is a DFDL String Literal, so values like dfdl:choiceBranchKey=" " (that's two spaces) are legal and will work. If it was to be a whitespace separated value, then those would have to use DFDL Entities to express the whitespace. I know I have seen COBOL data where there were multiple tags that denote the same structure. I am curious what others experience with choiceBranchKey is, and how this issue was handled. I believe from prior emails that IBM has already implemented this in its DFDL implementation. There is an erratum specifying that choiceBranchKey comparison is supposed to be case-sensitive (for performance reasons), but nothing heretofore about it using DFDL Entities to represent whitespace. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0