January 2017 - dfdl-wg - lists.ogf.org

Re: [DFDL-WG] spec bug dfdl:choiceLength says 'bytes' should use dfdl:lengthUnits
by Steve Hanson 10 Jan '17

10 Jan '17

https://redmine.ogf.org/issues/325 created Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 16/12/2016 09:17 Subject: Re: [DFDL-WG] spec bug dfdl:choiceLength says 'bytes' should use dfdl:lengthUnits Mike dfdl:lengthUnits is not currently used by dfdl:choice. If by erratum we changed this, existing schemas could break as they would suddenly pick up the value from scope. We have a couple of alternatives: 1) A new property dfdl:choiceLengthUnits with enums 'bytes' and 'bits' (not seen a use case for 'characters'). For compatibility, an implementation would need to default the value to 'bytes' if not set locally or in scope, rather than give an error. ** 2) New enum for dfdl:choiceLengthKind 'explicitBits' which causes the dfdl:choiceLength to be interpreted as a bit count. Avoids the default pitfall, but is architecturally inconsistent with lengthKind/length/lengthUnits. ** IBM DFDL had to do a similar thing when dfdl:encodingErrorPolicy was added to avoid breaking existing schemas. Alternative is to bump the DFDL version number. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 15/12/2016 16:04 Subject: [DFDL-WG] spec bug dfdl:choiceLength says 'bytes' should use dfdl:lengthUnits Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> for dfdl:choiceLength, the description says "Specifies the length of the choice in bytes..." clearly this should use dfdl:lengthUnits. We have tons of cases where byte granularity is not sufficient, and this is hard to work around. You have to use dfdl:choiceLengthKind='implicit' and then carefully pad the branches to the same length in bits. It's painful. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] choiceBranchKey - suggest change to List of DFDL String Literals ?
by Steve Hanson 10 Jan '17

10 Jan '17

https://redmine.ogf.org/issues/326 created Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/12/2016 18:15 Subject: Re: [DFDL-WG] choiceBranchKey - suggest change to List of DFDL String Literals ? Yes it gets ugly and inefficient when you start having a large 'if' statement. An example of an IBM-schema with direct dispatch choice is SWIFT, which has dozens of messages but they all fall into 7 patterns so the expression 'if' statement has 7 branches. Not seen anything more complex than that so far. A List of DFDL String Literals is a possibility. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/12/2016 17:34 Subject: Re: [DFDL-WG] choiceBranchKey - suggest change to List of DFDL String Literals ? Yes this situation is that the expressions are big, because there are 100's of branches, and so the constants are remote from where the branch is expressed. And it ends up being quite inefficient. What one wants is a table based lookup. Not an expression like if (../key eq "11 or ../key eq "22") then "11" else if (../key eq "AA" or ../key eq "BB" then "AA" ... so on for dozens of cases. and so on. That's just a linear chain again, so equivalent to using discriminators on the branches. An interesting point expressed to us is that users would like the DFDL schema to not only work to parse and unparse the file, but for it to serve as a declarative expression of the data format - as the machine-readable documentation of the format ultimately. While a big expression will work, it's not as declarative as listing the possible dispatch keys for each branch right on that branch. A big expression really fails to capture the essence of the format in a manner that enables efficient implementation. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Dec 13, 2016 at 11:01 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Hi Mike If I recall, choiceBranchKey was a single DFDL String Literal because the value to be matched against it is computed by a DFDL expression (from choiceDispatchKey), which can do things like lower casing strings or other manipulations. Are you seeing use cases where multiple keys per branch are making the expression too complex? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 12/12/2016 19:48 Subject: [DFDL-WG] choiceBranchKey - suggest change to List of DFDL String Literals ? Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Our first experience with requiring and implementing choiceBranchKey in Daffodil and we're already dealing with the fact that many times the same branch has multiple different keys all of which indicate that it should be selected. Right now, dfdl:choiceBranchKey is a DFDL String Literal, so values like dfdl:choiceBranchKey=" " (that's two spaces) are legal and will work. If it was to be a whitespace separated value, then those would have to use DFDL Entities to express the whitespace. I know I have seen COBOL data where there were multiple tags that denote the same structure. I am curious what others experience with choiceBranchKey is, and how this issue was handled. I believe from prior emails that IBM has already implemented this in its DFDL implementation. There is an erratum specifying that choiceBranchKey comparison is supposed to be case-sensitive (for performance reasons), but nothing heretofore about it using DFDL Entities to represent whitespace. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2017-01-10
by Steve Hanson 09 Jan '17

09 Jan '17

Happy New Year. Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13548?download= Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 09 Jan '17

09 Jan '17

Confirmed: OGF DFDL Working Group call 10/01/2017 - (Repeats) Location: IBM: Hursley DE3S09 Steve Hanson has confirmed this meeting Repeats This entry repeats Required: mbeckerle(a)tresys.com, mbeckerle.dfdl(a)gmail.com Optional: Alex Wood1/UK/IBM@IBMGB, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, jorge.marizan(a)gmail.com, Mark Frost/UK/IBM FYI: Calls for 2017. OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 Netherlands Toll-Free 0-800-363-6036 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg DFDL Getting Started: http://www.ibm.com/developerworks/library/se-dfdl/ DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0