September 2020 - dfdl-wg

DFDL Spec Working Draft r29
by Mike Beckerle 01 Oct '20

01 Oct '20

Steve H., and DFDL Workgroup, I have uploaded version r29 of the DFDL spec as MS-Word ".docx" This has all the comments from the external review by Martin Westhead applied to it, including numerous forward references fixed. In general the material flows better (IMHO) and is better organized with a number of out-of-place discussions/paragraphs repositioned to be in a better place with other related material. I believe this is a substantially better document now. I have tried to mark the moves of sections of text with comment bubbles. I'm sure there are some missing however. Not every comment bubble is resolved as yet. - Comment bubbles marked TBD mean a change is still expected or an issue to be resolved. Some of these are calls for new material such as new clarifying examples, and we may or may not decide to put them in. - Comment bubbles marked "Note" are FYI for read/review and may not involve any further change. Basically, it says "Note" if I currently think the issue that motivated the change has been resolved, but you may have a different opinion. You will find two variants of the document: one has MS-Word change tracking, the other has all changes accepted. When sections were renumbered I had to turn off MS-Word change tracking. So you'll notice that even in the track-changes version, renumberings are (mostly) not marked as tracked changes. It is my hope that after another read-through the renumbering and reorganization are accepted and we can go forward from here using the all-changes-accepted version, though I realize in review it may be important to see what text has been deleted/inserted via the change-tracked variant. It is also my hope that we are getting close to a draft ready for public review. Saving reviewer/proofread time is the most important thing, so you may print it out, and redline it with red ink if you wish, and ship the result to me via courier/fedex. Or if possible you can PDF it, and can virtual red-ink it with a tablet computer/stylus, and e-send the PDF to me. This is the model Martin Westhead used when reviewing, and it worked well for me. But basically, I will accept feedback/review in any form: MS-Word comment bubbles are fine, as is red-ink on paper. One possibly useful technique: If you open the "tracked" variant in MS-Word, and from the review tab, choose "Simple Markup" as the change display mode, you will get just margin side-bars marking where changes are, and the comment bubbles, but otherwise not too much change noise. You can click on any given sidebar and it will switch to "All Markup" mode showing you the detailed tracked changes, but you can switch back to "Simple Markup" after reviewing the details. I have started keeping these files on the OGF Github under docs/working-drafts, they are also on redmine in the current documents. OGF Github DFDL repository has URLs: https://github.com/OpenGridForum/DFDL.git or if you prefer ssh: git@github.com:OpenGridForum/DFDL.git Redmine the two docs are: https://redmine.ogf.org/dmsf_files/13621?download= https://redmine.ogf.org/dmsf_files/13622?download= -mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Agenda for OGF DFDL WG Call
by Steve Hanson 30 Sep '20

30 Sep '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13623?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

RFC2119 keywords and capitalization e.g., must vs. MUST
by Mike Beckerle 17 Sep '20

17 Sep '20

I am looking for guidance here while editing the DFDL spec. An open topic has been the RFC2119 keywords. Many specs just refer to these and say "we use RFC2119 conventions." We can't do that. We use keywords MUST, SHOULD, SHALL, etc. in 3 ways. 1) Discussing requirements/desires for all implementations - e.g., all implementations MUST detect schema definition errors. 2) Discussing "requirements" a DFDL schema author must ensure are followed in order to avoid getting a SDE. E.g., "The DFDL fillByte property must be a single byte or single character." Many times we follow this sort of requirement by "and it is an SDE otherwise." It is implied that implementations must detect this SDE and provide a diagnostic to the user. 3) Discussing "requirements" that data must conform to, in order for the data to conform to a DFDL schema. E.g., "The representation must be followed by the terminating delimiter." The assumption in these cases is that it is a processing error if the terminator is not found. It is the way these runtime-behaviors are expressed that is of note. They are not phrased in terms of how implementations must behave, but rather in terms of what users should expect. My use of the word "should" here is intentional - we do in fact use "should" in many cases to imply required behavior, not suggested behavior. These different categories of requirement areas are normal to a description of a language standard, and category (3) here is to some degree unique to a description language that describes other formal objects that may or may not in fact correspond to the description. E.g., there are unit test frameworks for code where one writes things like "add(1, 1) should equal 2", where "should" means the same thing as "shall" or "must". Next: the term "may" is used to mean (1) optional per RFC2119 (2) "allows" or "admits" when describing DFDL schemas E.g., "If the nil representation may not be zero length, but the empty representation is zero length, then the absent representation cannot occur because zero-length will be interpreted as the empty representation." (3) discussing possible alternatives for data E.g, "Encodings which are 5, 6, 7, and 9 bits wide are rare, but do exist, so the overall length of the content region may not be a multiple of 8 bits wide." My current working draft (r27, not yet on redmine, but soon) has put all the category 1 usages in all upper case (for MUST, SHOULD, SHALL, MAY, etc.) I have left all the category (2) and (3) as lower case, as there are many of them, and they really do need to be distinguished from category (1). We have to explain these 3 different kinds of usage if we're going to reference RFC2119 at all. But I am seriously considering whether we should just say we DON"T use RFC2119 conventions but rather explain these 3 categories of usage. Next the terms required and optional in RFC2119 are used (1) about requirements that MUST be implemented vs are OPTIONAL but RECOMMENDED, and if implemented are REQUIRED to be implemented in a conforming manner. (2) about language features that are optional for use. E.g., the testKind property of a dfdl:assert annotation is optional, and assumed to be "expression", not "pattern" if not provided. (3) as formal terms to discuss when default values are used or not for array/optional elements. I propose that category (1) above is again all caps, but categories (2) and (3) are not. I have not made any such changes as yet. So, what I'm looking for is feedback on these ideas. What I am planning to do is in the section on Terminology and Conventions to explain these 3 cateogories of "requirements" and mention that RFC2119 convention applies only to the first category. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

MS Word Limitations for DFDL Spec Review
by Mike Beckerle 17 Sep '20

17 Sep '20

Two topics: * change tracking * hyperlinks So... Change Tracking: I have reached the limits of MS-Word with change tracking in this roughly 250 page DFDL spec document. I believe at this point I am going to have to give up on change tracking, I.e., providing drafts with accumulated changes since a prior major version marked with change bars. I have found that a PDF export from MS Word with tracked changes even with only "simple markup" is quite illegible, with heading numbers appearing not on the same line even as their heading, etc. I believe this is an interaction of renumbering of sections and change tracking. Either way, MS Word crashes often and I am worried about losing work or corrupting the document. I found already that in revision r22, a cross-reference to the section about recoverable errors was not putting in a cross reference to that section, but rather was repeating the entire contents of the section at each point of cross reference. I had to hand delete all of these as I encountered them when going through the reviewer comments page by page. I think at this point we're forced to greatly reduce use of MS-Word change tracking, and if a reader wants to study the changes between two revisions, they have to fire up MS-Word, and use it to compare two working draft versions of the document. So the version I am going to push up for consideration soon (which is probably r27 or r28) will have change tracking, and also I will create a version with all changes accepted. Further changes will happen in the one with all (current) changes accepted, creating a new, smaller set of changes, not an accumulated set of all changes since the prior official draft. In addition, for various large changes like section moves, I plan to accept them, and just add a comment bubble to remind reviewers to read the section, as having the whole change visible with strikethrough of the deleted and colored/underlined text for the insertions ruins the flow of the document. The only reliable viewer for the document, which can show the tracked changes in "simple markup" so that you see change bars on sides of pages only, is MS Word itself. Creating a PDF with "simple markup" doesn't work right. Hyperlinks: I have determined that MS-Word cross references are simply NOT converted into navigable hyperlinks when the document is output as an HTML document. This appears to be simply a MS-Word limitation. The same limitation exists in OpenOffice. A PDF gets navigable hyperlinks, but not an HTML output. Furthermore, I have determined that an MS-Word Index results in a printable index, but again there are no navigable links from the index to the referenced pages/locations. Based on this I am going to abandon, for now, creating a easily/readily used HTML version of the spec., and stick with just PDF. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 5

Latest OGF DFDL WG Call Minutes
by Steve Hanson 17 Sep '20

17 Sep '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 17 Sep '20

17 Sep '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13619?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Re: [DFDL-WG] DFDL pointer & offset proposal
by Bradd Kadlecik 16 Sep '20

16 Sep '20

Have updated for comments. (See attached file: DFDL_Indirection_v5.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 07/24/2020 01:08 PM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal "Actually, the target of the indirection is just like an element or model-group, except without alignment regions. Ugly to have to repeat the whole grammar for that. ". Correct, and I haven't. My changes are for the indirection value, not the target. We have to handle 'indirectionUnusedValue' and reflect that initiator & terminator do not apply. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 24/07/2020 13:59 Subject: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal Actually, the target of the indirection is just like an element or model-group, except without alignment regions. Ugly to have to repeat the whole grammar for that. Perhaps we can get away with just documenting that the alignment regions must be zero length when target of an indirection? I did not try to put this modified grammar into my commented copy of the proposal (attached). The "connection" to the indirect pointer or offset is not something that the grammar can express. Attached * my review (with new comments that start with "V4:"), * my first cut of a DFDL schema for TIFF v6.0, * a copy of the TIFF v6.0 spec. I wasn't actually able to pull off TIFF. Comments in the schema discuss why, which is due to TIFF being a linked list, and I'm not able to terminate the chain. There's maybe a trick using negative offsets, but I don't want to consider that yet, as those are entirely problematic. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Jul 24, 2020 at 7:03 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Bradd, Mike - as discussed on WG call please review my version of the grammar for next call. The key issue is how to link the target of the indirection into the main grammar, as it is dis-contiguous. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 23/07/2020 14:52 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I incorporated some of your grammar changes but had a few modifications of my own we can discuss further in case I'm not understanding some of the subtleties. I think there may be a case for a nil value with indirection however as well. I've updated the document to include the changes as version 4. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 07/10/2020 08:11 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions --------------------------------------------------------------------------------------- Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement --------------------------------------------------------------------------------------- AbsentElementRep = Absent --------------------------------------------------------------------------------------- IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue --------------------------------------------------------------------------------------- SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment --------------------------------------------------------------------------------------- SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill --------------------------------------------------------------------------------------- SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice --------------------------------------------------------------------------------------- Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement --------------------------------------------------------------------------------------- LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: [attachment "DFDL_Indirection_v3.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "TIFF-v6.pdf" deleted by Bradd Kadlecik/Poughkeepsie/IBM] [attachment "DFDL_Indirection_v4_mjb_comments.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] [attachment "tiff.dfdl.xsd" deleted by Bradd Kadlecik/Poughkeepsie/IBM]

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 15 Sep '20

15 Sep '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group Call
by Steve Hanson 02 Sep '20

02 Sep '20

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 02 Sep '20

02 Sep '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13618?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0