April 2020 - dfdl-wg - lists.ogf.org

DFDL Spec 1.0.5 draft r22
by Mike Beckerle 30 Apr '20

30 Apr '20

I pushed r22 up to redmine. This has all review comments by M. Westhead thusfar incorporated. I pushed up an HTML version which has change tracking so the text changes are all visible in gory detail (which is pretty bad given that big sections moved around) I pushed up the MS-Word master. I pushed up a PDF with "simple markup" of changes. All of these have some artifacts of change tracking in them - things that won't look right until the changes get accepted. NOTE: The Errata-related changes were all accepted for post revision r12, so this r22 spin is really about the edit-driven changes from M.Westhead review, and improving the flow and organization based upon that. If you want a DFDL spec that still has all the same sections and section ordering as the prior official draft, but with the errata applied (with change tracking) then you want revision r12, which is also on redmine. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 30 Apr '20

30 Apr '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

what do you think of this for Section 22 Property Precedence
by Mike Beckerle 30 Apr '20

30 Apr '20

I just revamped the first section in the attached MS-Word nested table structure. This is tedious to assemble, but I think does a better job than just the indented trees particularly given the fact that we ran out of indent levels. I'm curious whether this is better/clearer. There's also a few comment bubbles where I discovered things I didn't understand or were surprising to me. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Re: [DFDL-WG] DFDL pointer & offset proposal
by Steve Hanson 30 Apr '20

30 Apr '20

309 Create example scenarios to illustrate offset & pointer requirements (Bradd) 5/4/19: Daffodil have a draft proposal for offset support, TPF have experimental implementation for pointer support. Need examples to show the requirement, especially unparsing. 2/5: Bradd supplied an example of pointers. On parsing the pointer is used as an absolute address to a piece of accessible memory, and the element is parsed from that location. On unparsing memory is allocated and unparsing of the element occurs into that location and the pointer set to the location (memory allocation is implementation-defined). Note the pointer value does *not* appear in the infoset. Looks like a useful and workable addition to DFDL. Could solve the parsing requirements for TIFF image files. Bradd also has extension for offset, which is like pointer but uses relative location instead of absolute. Both are examples of indirection. A further example could be specifying a file to read. Contrast this with what DFDL has used the term 'offset' for in the past, namely as an alternative property to alignment/skip which allows the parser/unparser to jump directly to a point in the current buffer. These are orthogonal concepts. Noted that parsing of ZIP files may need both. Secure implementations may need to disallow use of pointers and/or offsets unless they can guarantee to fill everywhere with the fill byte. Implementations should also be deterministic. Agreed that recursion not needed to implement this. Bradd mentioned a further concept 'overflows', an example being an array unparsed into a linked list. Pointers proposal needs to be written up as an experimental feature. 31/5: Bradd to write up pointers proposal as an experimental feature. ... 11/7: No update 8/8: Bradd aiming to get this written up for next time. Also needs issue tracker raising. 29/8: Bradd unable to make the call. 17/10: Written up for review and sent to WG but not as an experimental feature document. Mike also noted http://www.binarydom.com/sdk/doc/bddl.shtml. Mike has reviewed and commented on the write-up, Steve needs to do the same, then send back to Bradd. Main discussion was around unparsing, eg, buffering implications, whether to try and format exactly or canonically. In parallel, Bradd to create an experimental feature document (see table below). 12/12: Bradd sent an updated document. WG will review for next meeting. There was some discussion about the use in the document of the term 'empty' and whether that really meant 'missing'. This led to an in-depth discussion about the different use cases for default values, it is likely that DFDL 2.0 will introduce support for some of these, specifically: - Item exists in Infoset with default value, so unparse empty rep (the mirror of parsing, as practised by GPB) - Item exists in data with default value, so remove from Infoset post-validation (the mirror of unparsing, and a requirement from z/TPF who have a post-parse option to do this) 9/1/20: No progress, still needs reviewing. 16/4: Steve & Mike to review latest document dated 2019-12-12 for next call. 30/4: Spent some time discussing Steve's review comments. Conclusion is that the feature is useful and a serious candidate for DFDL 2.0. The properties seem to be the minimum needed to handle the concepts and known use cases. As this is currently an experimental feature we don't have to get it 100% precise now, and can impose restrictions that z/TPF users would be ok with (for example, no initiators or terminators allowed; binary indirection types only). Important though that the properties and their application is driven by the grammar, so next step is for Bradd to see how the grammar is affected. It would be nice if all the behaviour could be handled at the same point in the grammar as 'prefixLength' but that might not be possible. Property name 'indirectionEmptyValue' probably needs a better name, eg 'indirectionUnusedRep'.... other suggestions welcome. Or perhaps the dfdl:fillByte of the indirectionType could be used? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: "Bradd Kadlecik" <braddk(a)us.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 30/04/2020 15:46 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal Comments for call today (yellow) Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: "Bradd Kadlecik" <braddk(a)us.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 12/12/2019 15:42 Subject: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Updated with comments: (See attached file: DFDL_Indirection_v2.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States Mike Beckerle ---10/16/2019 05:43:52 PM--- I added some comments to your original document. Attached. Mike Beckerle | OGF DFDL Workgroup Co-Ch From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Bradd Kadlecik <braddk(a)us.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 10/16/2019 05:43 PM Subject: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I added some comments to your original document. Attached. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Sep 25, 2019 at 12:55 PM Bradd Kadlecik <braddk(a)us.ibm.com> wrote: Here's what I've put together regarding the pointer & offset proposal for the next meeting's review. (See attached file: DFDL_Indirection.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States Steve Hanson---04/05/2019 12:42:34 PM---Regards From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int> Date: 04/05/2019 12:42 PM Subject: Re: Latest OGF DFDL WG Call Minutes No Action 309 Create example scenarios to illustrate offset & pointer requirements (Bradd) 5/4/19: Daffodil have a draft proposal for offset support, TPF have experimental implementation for pointer support. Need examples to show the requirement, especially unparsing. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Steve Hanson---05/04/2019 10:29:24---I can see a difference between offsets and pointers. If I follow an offset and parse an element x t From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int> Date: 05/04/2019 10:29 Subject: Re: Latest OGF DFDL WG Call Minutes I can see a difference between offsets and pointers. If I follow an offset and parse an element x then I won't automatically jump back to where I was - the next element y I parse will continue from location offset + length x unless I use offset again to jump back to the original location. If I follow a pointer and parse an element x, then the next element y I parse will continue from original location + length(pointer). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Steve Hanson---04/04/2019 09:42:36---This is a proposal from the Daffodil team for offset support, needed for formats like TIFF and if we From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int> Date: 04/04/2019 09:42 Subject: Re: Latest OGF DFDL WG Call Minutes This is a proposal from the Daffodil team for offset support, needed for formats like TIFF and if we ever want to be able to handle zip files. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74687382 I think this proposal can implement your requirement - the dfdl:offset property can be an expression that refers to another element (your pointer element), which can be hidden so as not to appear in the infoset. I think what you are proposing is a more convenient way of handling offsets that are defined dynamically in the data, as opposed to defined statically with fixed values (though as I said below I need unparsing explained). But I may be mis-understanding your use cases. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Bradd Kadlecik---03/04/2019 16:50:13---Ok, I'll work on putting a scenario together for various pointer setups (array, complex element, str From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int> Date: 03/04/2019 16:50 Subject: Re: Latest OGF DFDL WG Call Minutes Ok, I'll work on putting a scenario together for various pointer setups (array, complex element, string) and show how it looks for both the JSON and the binary. I'm currently at a conference this week but will be returning late Thursday so expect to be available for the call Friday. I presented the TPF specific pointer implementation at the conference this week and there are some that will be trying to use it soon. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States Steve Hanson---04/03/2019 10:29:25 AM---Hi Bradd I have a few questions please ... inline below. From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 04/03/2019 10:29 AM Subject: Re: Latest OGF DFDL WG Call Minutes Hi Bradd I have a few questions please ... inline below. I'm still going to need a real worked example, starting with some actual data and its schema, what it appears like after paring in the infoset, and how unparsing lays it back out again. Are you able to make the rescheduled call this Friday? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Bradd Kadlecik---27/02/2019 22:22:48---Regarding proposal for offsets and pointers: The following are the properties to be defined: From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 27/02/2019 22:22 Subject: Re: Latest OGF DFDL WG Call Minutes Regarding proposal for offsets and pointers: The following are the properties to be defined: SMH: I was expecting to see these properties only on dfdl:element, especially as you say '...the element contents... ? indirectKind Enum Valid values 'pointer', 'offset' (there is also a thought of objectId or refId for handling BSON but not at this time) Specifies the type of indirection used to access the element contents in the data stream. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I am missing the distinction between offset and pointer. Is one relative to current position and the other relative to start of bitstream? SMH: In earlier DFDL proposals for offset support, we had used the term to refer to a property to be used to establish position of the current element instead of assuming the current element followed straight after the previous one. It would allow sparse modelling of fixed structures. The offset could be relative to start of bitstream or some other point. I don't think that's what you mean when you say 'offset' so I will refer to your new concept as 'pointer'. SMH: Assuming that indirectKind is a normal DFDL property, it can be in scope. It would therefore need to have an enum 'None' which would be the default used in most schemas. indirectLength Non-negative Integer or DFDL expression Specifies the length of the indirection in units according to the indirectUnits property. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group indirectUnits Enum Valid values 'bytes','bits' Specifies the units to be used for reading or writing the indirection according to indirectLength. The default value is 'bytes'. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I think a better approach is to provide a property dfdl:indirectType, instead of indirectLength/indirectUnits, which refers to a simple type (not element) that carries its own lengthKind, length & lengthUnits properties. Similar idea to dfdl:prefixLengthType. That allows a lot of flexibility on how the pointer can appear. offsetBase non-empty string containing an absolute or relative XPath expression for the base element. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group The proposal would be to have the contents of the indirection be after the LeadingAlignment and before the TrailingAlignment. This would mean the aligment and skip factors apply to the indirection values in the data stream instead of the the contents of the indirection. SMH: Agree. This also then means in an array element, each element has its own indirection value (pointer or offset) and the alignment,skip factors then apply to each of these indirection values. SMH: Do you mean '...each occurrence...' ? It would be thought that the indirection values apply only to the data stream and not the infoset. During parse when the infoset is populated from the data stream, the indirection values are replaced by the contents. During unparse, the indirection values don't exist in the infoset and are created during the writing to/creation of the data stream. SMH: I agree that the indirection should be a purely physical thing, but I am not clear how the value is filled in when unparsing. Where does the value come from? outputValueCalc? Or maybe it's not needed when unparsing, and the data is always contiguous? For pointers, a null pointer creates the scenario of either nil representation or empty representation depending on whether or not nillable is defined as true. Unless default values (or 0 occurrence) are defined for all underlying content, then this is a processing error. During unparse, the only scenario in which a null pointer would be created is for a nil representation. SMH: This needs more thought. The nil & default properties apply to the contents of the indirection, not to the pointer. If you want to give a nil semantic to the pointer value itself, then that would require a new enum for dfdl:nilKind. I don't see why a pointer value 0 can't be treated like any other indirection value. A missing pointer is an error - it must be present - there is no way to control optionality because minOccurs/maxOccurs apply to the contents. (Alternatively, if you want the concepts of nil, default, occurs to apply to the indirect value, then dfdl:indirectType could point at an element instead of a simple type - but that seems way too over engineered). Examples: The following is the definition for the address of a null-terminated string in which the string address may be NULL as indicated by a nillable value of true: <xs:element name="myString" type="xs:string" dfdl:lengthKind="delimited" dfdl:encoding="UTF-8" dfdl:terminator="%NUL;" dfdl:indirectKind="pointer" dfdl:indirectLength="8" dfdl:indirectUnits="bytes" nillable="true" /> The following is the definition for an array of three 4 byte addresses of a complex element defined by ns0:myStruct: <xs:element name="myArray" type="ns0:myStruct" dfdl:lengthKind="implicit" dfdl:indirectKind="pointer" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" minOccurs="3" maxOccurs="3" dfdl:occursCountKind="fixed" /> The following is the definition for a 4 byte offset to a 100 byte hexBinary value from the start of the parent element definition: <xs:element name="myData" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="100" dfdl:lengthUnits="bytes" dfdl:indirectKind="offset" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" dfdl:offsetBase=".." /> SMH: I don't see how unparsing works. What provides the value? The proposal would also allow for the following optional item but I don't currently see a need for this: dfdl:offsetKind with values "startToStart" or "endToStart" - indicates if the offset is from the start of the base element or the end of the base element. I tried getting this out before my vacation so it might take a little bit to respond for issues. Thank you for your time. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States Steve Hanson---02/07/2019 12:32:26 PM---Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/newsRegards Ste From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 02/07/2019 12:32 PM Subject: Latest OGF DFDL WG Call Minutes Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg[attachment "DFDL_Indirection-mikeb-comments.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL pointer & offset proposal
by Bradd Kadlecik 30 Apr '20

30 Apr '20

Here's what I've put together regarding the pointer & offset proposal for the next meeting's review. (See attached file: DFDL_Indirection.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 04/05/2019 12:42 PM Subject: Re: Latest OGF DFDL WG Call Minutes |------+------------------------------------------------------------------| | No |Action | |------+------------------------------------------------------------------| | 309 |Create example scenarios to illustrate offset & pointer | | |requirements (Bradd) | | |5/4/19: Daffodil have a draft proposal for offset support, TPF | | |have experimental implementation for pointer support. Need | | |examples to show the requirement, especially unparsing. | |------+------------------------------------------------------------------| Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 05/04/2019 10:29 Subject: Re: Latest OGF DFDL WG Call Minutes I can see a difference between offsets and pointers. If I follow an offset and parse an element x then I won't automatically jump back to where I was - the next element y I parse will continue from location offset + length x unless I use offset again to jump back to the original location. If I follow a pointer and parse an element x, then the next element y I parse will continue from original location + length(pointer). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 04/04/2019 09:42 Subject: Re: Latest OGF DFDL WG Call Minutes This is a proposal from the Daffodil team for offset support, needed for formats like TIFF and if we ever want to be able to handle zip files. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74687382 I think this proposal can implement your requirement - the dfdl:offset property can be an expression that refers to another element (your pointer element), which can be hidden so as not to appear in the infoset. I think what you are proposing is a more convenient way of handling offsets that are defined dynamically in the data, as opposed to defined statically with fixed values (though as I said below I need unparsing explained). But I may be mis-understanding your use cases. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 03/04/2019 16:50 Subject: Re: Latest OGF DFDL WG Call Minutes Ok, I'll work on putting a scenario together for various pointer setups (array, complex element, string) and show how it looks for both the JSON and the binary. I'm currently at a conference this week but will be returning late Thursday so expect to be available for the call Friday. I presented the TPF specific pointer implementation at the conference this week and there are some that will be trying to use it soon. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 04/03/2019 10:29 AM Subject: Re: Latest OGF DFDL WG Call Minutes Hi Bradd I have a few questions please ... inline below. I'm still going to need a real worked example, starting with some actual data and its schema, what it appears like after paring in the infoset, and how unparsing lays it back out again. Are you able to make the rescheduled call this Friday? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 27/02/2019 22:22 Subject: Re: Latest OGF DFDL WG Call Minutes Regarding proposal for offsets and pointers: The following are the properties to be defined: SMH: I was expecting to see these properties only on dfdl:element, especially as you say '...the element contents... ? indirectKind Enum Valid values 'pointer', 'offset' (there is also a thought of objectId or refId for handling BSON but not at this time) Specifies the type of indirection used to access the element contents in the data stream. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I am missing the distinction between offset and pointer. Is one relative to current position and the other relative to start of bitstream? SMH: In earlier DFDL proposals for offset support, we had used the term to refer to a property to be used to establish position of the current element instead of assuming the current element followed straight after the previous one. It would allow sparse modelling of fixed structures. The offset could be relative to start of bitstream or some other point. I don't think that's what you mean when you say 'offset' so I will refer to your new concept as 'pointer'. SMH: Assuming that indirectKind is a normal DFDL property, it can be in scope. It would therefore need to have an enum 'None' which would be the default used in most schemas. indirectLength Non-negative Integer or DFDL expression Specifies the length of the indirection in units according to the indirectUnits property. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group indirectUnits Enum Valid values 'bytes','bits' Specifies the units to be used for reading or writing the indirection according to indirectLength. The default value is 'bytes'. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I think a better approach is to provide a property dfdl:indirectType, instead of indirectLength/indirectUnits, which refers to a simple type (not element) that carries its own lengthKind, length & lengthUnits properties. Similar idea to dfdl:prefixLengthType. That allows a lot of flexibility on how the pointer can appear. offsetBase non-empty string containing an absolute or relative XPath expression for the base element. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group The proposal would be to have the contents of the indirection be after the LeadingAlignment and before the TrailingAlignment. This would mean the aligment and skip factors apply to the indirection values in the data stream instead of the the contents of the indirection. SMH: Agree. This also then means in an array element, each element has its own indirection value (pointer or offset) and the alignment,skip factors then apply to each of these indirection values. SMH: Do you mean '...each occurrence...' ? It would be thought that the indirection values apply only to the data stream and not the infoset. During parse when the infoset is populated from the data stream, the indirection values are replaced by the contents. During unparse, the indirection values don't exist in the infoset and are created during the writing to/creation of the data stream. SMH: I agree that the indirection should be a purely physical thing, but I am not clear how the value is filled in when unparsing. Where does the value come from? outputValueCalc? Or maybe it's not needed when unparsing, and the data is always contiguous? For pointers, a null pointer creates the scenario of either nil representation or empty representation depending on whether or not nillable is defined as true. Unless default values (or 0 occurrence) are defined for all underlying content, then this is a processing error. During unparse, the only scenario in which a null pointer would be created is for a nil representation. SMH: This needs more thought. The nil & default properties apply to the contents of the indirection, not to the pointer. If you want to give a nil semantic to the pointer value itself, then that would require a new enum for dfdl:nilKind. I don't see why a pointer value 0 can't be treated like any other indirection value. A missing pointer is an error - it must be present - there is no way to control optionality because minOccurs/maxOccurs apply to the contents. (Alternatively, if you want the concepts of nil, default, occurs to apply to the indirect value, then dfdl:indirectType could point at an element instead of a simple type - but that seems way too over engineered). Examples: The following is the definition for the address of a null-terminated string in which the string address may be NULL as indicated by a nillable value of true: <xs:element name="myString" type="xs:string" dfdl:lengthKind="delimited" dfdl:encoding="UTF-8" dfdl:terminator="%NUL;" dfdl:indirectKind="pointer" dfdl:indirectLength="8" dfdl:indirectUnits="bytes" nillable="true" /> The following is the definition for an array of three 4 byte addresses of a complex element defined by ns0:myStruct: <xs:element name="myArray" type="ns0:myStruct" dfdl:lengthKind="implicit" dfdl:indirectKind="pointer" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" minOccurs="3" maxOccurs="3" dfdl:occursCountKind="fixed" /> The following is the definition for a 4 byte offset to a 100 byte hexBinary value from the start of the parent element definition: <xs:element name="myData" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="100" dfdl:lengthUnits="bytes" dfdl:indirectKind="offset" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" dfdl:offsetBase=".." /> SMH: I don't see how unparsing works. What provides the value? The proposal would also allow for the following optional item but I don't currently see a need for this: dfdl:offsetKind with values "startToStart" or "endToStart" - indicates if the offset is from the start of the base element or the end of the base element. I tried getting this out before my vacation so it might take a little bit to respond for issues. Thank you for your time. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 02/07/2019 12:32 PM Subject: Latest OGF DFDL WG Call Minutes Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 3

Agenda for OGF DFDL WG Call
by Steve Hanson 30 Apr '20

30 Apr '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13611?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group Call
by Steve Hanson 30 Apr '20

30 Apr '20

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Clarification on UTF-16 and UTF-32 encoding byte order
by Steve Hanson 29 Apr '20

29 Apr '20

Tested on Linux & Windows, and the behaviour is 1. assumes big-endian Which matches what the spec said post-erratum for when there was no BOM (the sentences I highlighted in blue below). So I think we have Plan D - allow UTF-16 & UTF-32 without byte order specifiers and always treat as BE. Anything else is not compatible with IBM DFDL. If there is a BOM in the data then it must be modelled, otherwise it will end up being treated as part of a value or delimiter. I've attached a zip that does this, which I created back in 2018 as proof we could drop the BOM support. The BOM element could be wrapped in a hidden group, can't do that with IBM DFDL. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 28/04/2020 19:18 Subject: Re: [EXTERNAL] Re: [DFDL-WG] Clarification on UTF-16 and UTF-32 encoding byte order Plan A is appealing, but IBM DFDL allows UTF-16 / 32 unadorned, and always has done. Problem is I can't remember what that does. IBM DFDL never implemented erratum 3.7 so there's four possibilities: 1. assumes big-endian 2. assumes little-endian 3. platform-dependent (no!) 4. uses dfdl:byteOrder (as per GFD.174) I need to do some tests to ascertain the behaviour. I'll try and do that by next WG call. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 28/04/2020 16:00 Subject: [EXTERNAL] Re: [DFDL-WG] Clarification on UTF-16 and UTF-32 encoding byte order So the important use case is this: The data is dfdl:representation='text', the encoding is utf-16 but a BOM tells us the byte order - but there is exactly one BOM, and it is at the start of the file. That BOM wants to be read, to stick, and to guide recreation of this data in any associated unparse. That special case is why we had this unicode byte order mark feature on the document level. What we failed to appreciate is that the byte order of this data will not vary day to day, but will nearly always be constant. Data coming from big-endian systems will be big-endian, and from little-endian systems will be little endian. So the use case for a schema that needs to adapt to either is a more rare case. That's why the features we used to have were overkill, because in the vast number of cases above, the users data will always have the same byte order, because it is coming from one system. Where we are today: we have already modified the DFDL spec draft to remove everything about byte order marks EXCEPT, we didn't remove support for UTF-16 or UTF-32 where a BOM might come in handy. I think to fix this it's either plan A or plan B. Plan (A) - Keep it simple - Just disallow utf-16 and utf-32 without byte-order specifiers - make people use the more specific encodings that specify byte order. If they in fact have data which varies in byte order from instance to instance, they have to model that... just as they would for binary data with that behavior. (We can supply this as sample code.) Plan (B) go back to what we had before. All of it. Even though nobody implemented it nor wants to. My preference is plan (A). I think this is entirely sufficient for DFDL v1.0. There's one other Plan (C) option, which would be to document that Utf-16 unadorned means this: accept the BOM, keep it as a character in the string, and use it on parse to interpret the rest of the characters. It would also, preserve a BOM character if present for unparse, but unparse would always be big endian - the BOM written (only written if the character is present at start of string) will be written as a Big-endian BOM. If not present, none is added. The other characters are always written big-endian. This is the "converts to BE" model. It's what the java utf16 encoders/decoders do if you do nothing special to force them to behave any particular way. Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Apr 28, 2020 at 4:13 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Section 11.1 was where BOMs were discussed, and it said: UTF-16. If a BOM is found then this is used to set the document information item [unicodeByteOrderMark] member, and all data with dfdl:encoding UTF-16 throughout the rest of the stream are assumed to have the implied byte order. If no BOM is found then all data with dfdl:encoding UTF-16 throughout the rest of the stream are assumed to have big-endian byte order. There is no need to model the BOM explicitly. UTF-32. If a BOM is found then this is used to set the document information item [unicodeByteOrderMark] member, and all data with dfdl:encoding UTF-32 throughout the rest of the stream are assumed to have the implied byte order . If no BOM is found then all data with dfdl:encoding UTF-32 throughout the rest of the stream are assumed to have big-endian byte order. There is no need to model the BOM explicitly. Same for unparsing. So it looks like we threw the baby out with the bath water when the section was removed! Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 23/04/2020 20:23 Subject: [EXTERNAL] [DFDL-WG] Clarification on UTF-16 and UTF-32 encoding byte order Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Since we dropped the Unicode byte order mark functionality from DFDL v1.0, the issue arises of what byte order is used when dfdl:encoding="utf-16" or dfdl:encoding="utf-32". We are clear that encodings define their own byte and bit order, the dfdl:byteOrder property is not used. There are these options: 1) explicitly disallow these encoding names because they do not specify a byte order. Require utf-16BE or utf-16LE, utf-32BE or utf-32LE. 2) specify that these are synonyms for the BE versions 3) specify that these are synonyms for the LE versions This comes up in the definition of the dfdl:byteOrder property where the text currently says: This property is never used to establish the byte order for text /strings with Unicode fixed-width encodings that do not specify the byte order (UTF-16 and UTF-32). Having removed the unicode byte order mark feature, this statement leaves us without a stipulation of how UTF-16 and UTF-32 byte order would be determined. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Clarification on UTF-16 and UTF-32 encoding byte order
by Mike Beckerle 29 Apr '20

29 Apr '20

Since we dropped the Unicode byte order mark functionality from DFDL v1.0, the issue arises of what byte order is used when dfdl:encoding="utf-16" or dfdl:encoding="utf-32". We are clear that encodings define their own byte and bit order, the dfdl:byteOrder property is not used. There are these options: 1) explicitly disallow these encoding names because they do not specify a byte order. Require utf-16BE or utf-16LE, utf-32BE or utf-32LE. 2) specify that these are synonyms for the BE versions 3) specify that these are synonyms for the LE versions This comes up in the definition of the dfdl:byteOrder property where the text currently says: This property is never used to establish the byte order for text /strings with Unicode fixed-width encodings that do not specify the byte order (UTF-16 and UTF-32). Having removed the unicode byte order mark feature, this statement leaves us without a stipulation of how UTF-16 and UTF-32 byte order would be determined. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Apache Daffodil (incubating) 2.6.0 Released
by Beckerle, Mike 24 Apr '20

24 Apr '20

The Apache Daffodil (incubating) community is pleased to announce the release of version 2.6.0. Notable changes in this release include performance and API updates, and the addition of Sonarqube analysis to our workflow. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/2.6.0/ Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team ===== *Disclaimer* Apache Daffodil is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

1 0