- dfdl-wg - lists.ogf.org

Latest OGF DFDL WG Call Minutes
by Steve Hanson 13 Nov '20

13 Nov '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 12 Nov '20

12 Nov '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13628?download= Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Re: [DFDL-WG] DFDL pointer & offset proposal
by Bradd Kadlecik 12 Nov '20

12 Nov '20

I've updated the proposal for the following issues: 1) Undefined term "underlying elements" used in indirectionUnusedValue description 2) Undefined term "prior element" used in indirectionBase description 3) Grammar needs to have indirection elements follow document element (See attached file: DFDL_Indirection_v6.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 07/24/2020 01:08 PM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal "Actually, the target of the indirection is just like an element or model-group, except without alignment regions. Ugly to have to repeat the whole grammar for that. ". Correct, and I haven't. My changes are for the indirection value, not the target. We have to handle 'indirectionUnusedValue' and reflect that initiator & terminator do not apply. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 24/07/2020 13:59 Subject: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal Actually, the target of the indirection is just like an element or model-group, except without alignment regions. Ugly to have to repeat the whole grammar for that. Perhaps we can get away with just documenting that the alignment regions must be zero length when target of an indirection? I did not try to put this modified grammar into my commented copy of the proposal (attached). The "connection" to the indirect pointer or offset is not something that the grammar can express. Attached * my review (with new comments that start with "V4:"), * my first cut of a DFDL schema for TIFF v6.0, * a copy of the TIFF v6.0 spec. I wasn't actually able to pull off TIFF. Comments in the schema discuss why, which is due to TIFF being a linked list, and I'm not able to terminate the chain. There's maybe a trick using negative offsets, but I don't want to consider that yet, as those are entirely problematic. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Jul 24, 2020 at 7:03 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Bradd, Mike - as discussed on WG call please review my version of the grammar for next call. The key issue is how to link the target of the indirection into the main grammar, as it is dis-contiguous. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 23/07/2020 14:52 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I incorporated some of your grammar changes but had a few modifications of my own we can discuss further in case I'm not understanding some of the subtleties. I think there may be a case for a nil value with indirection however as well. I've updated the document to include the changes as version 4. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 07/10/2020 08:11 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions --------------------------------------------------------------------------------------- Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement --------------------------------------------------------------------------------------- AbsentElementRep = Absent --------------------------------------------------------------------------------------- IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue --------------------------------------------------------------------------------------- SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment --------------------------------------------------------------------------------------- SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill --------------------------------------------------------------------------------------- SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice --------------------------------------------------------------------------------------- Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement --------------------------------------------------------------------------------------- LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: [attachment "DFDL_Indirection_v3.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "TIFF-v6.pdf" deleted by Bradd Kadlecik/Poughkeepsie/IBM] [attachment "DFDL_Indirection_v4_mjb_comments.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] [attachment "tiff.dfdl.xsd" deleted by Bradd Kadlecik/Poughkeepsie/IBM]

1 0

OGF DFDL Working Group Call
by Steve Hanson 12 Nov '20

12 Nov '20

1 0

Action 320: Migration to Github from redmine trackers
by Mike Beckerle 06 Nov '20

06 Nov '20

We have 3 trackers left on redmine for DFDL v1.0 version. I propose these be re-created as github issues as public comments, and they be closed on redmine. The 16 redmine issues targeted at DFDL v2.0 version should become issue trackers on the DFDL project repository. Note that this is a separate pool of trackers from the public comments, which are trackers on the editor github repo, not the DFDL repo. That would then end use of redmine trackers. Going forward, if a public comment is resolved by being tracked for consideration in DFDL v2.0, then the public comment would be closed and a DFDL project issue opened reflecting the content of the public comment. As for the documents, I am not yet proposing a migration strategy for that, nor am I clear on what the replacement on github is for redmine "news". The trackers, documents, and news appear to be the only redmine features we were using. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Action 289: outputValueCalc refers back to expression that refers forward
by Mike Beckerle 03 Nov '20

03 Nov '20

One of the open questions w.r.t. this action item is whether DFDL is missing a property on dfdl:newVariableInstance and dfdl:setVariable to indicate if they are to be evaluated at parse, unparse, or both. I have updated the PCAP schema on github to illustrate this issue. This is a pull request (code review) of the changes https://github.com/DFDLSchemas/PCAP/pull/10 In the PCAP schema the IPSrcGrp group ref appears immediately before the IPSrc element. Similarly the IPDestGrp reference appears immediately before the IPDest element. These groups contain elements which are the individual binary bytes of the IP Source and IP Dest addresses. They are 4 unsigned bytes. When parsing these are parsed and then their values concatenated as strings into a string like "1.2.3.4". When unparsing, these groups use dfdl:outputValueCalc to take apart a string like "1.2.3.4" (which appears in the infoset after parsing), back into its individual digits 1, 2, 3, 4, and creates an unsignedByte value of each number. The problem is, these groups are absolute nonsense if you view the dfdl:newVariableInstance defaultValue expression as being evaluated at parse time. The expression is for use *only* at unparse time and immediately does a forward reference to the following IPSrc member containing the "1.2.3.4" type string. Without some sort of additional property on dfdl:newVariableInstance and dfdl:setVariable to indicate that this only should be evaluated at unparse time, there is no way to suppress evaluation and perform it only at unparse time. I suggest we need an additional property on newVariableInstance and setVariable which I suggest is called 'direction', with values 'parse', 'unparse', 'both'. This property defaults to 'both', but if set to 'unparse' only evaluates the expression (and binds/sets variable) during that direction of processing. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 30 Oct '20

30 Oct '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 29 Oct '20

29 Oct '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13627?download= Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 15 Oct '20

15 Oct '20

Today's WG call is replaced by a DFDL 1.0 draft spec review session, involving Mike & Steve. (Saves me having to create agenda & issue minutes :) Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Re: [DFDL-WG] DFDL Spec Working Draft r29
by Steve Hanson 07 Oct '20

07 Oct '20

Mike I've completed the review. To speed things up, are you around this Thursday at the usual DFDL WG call time (4pm UK) and we can go through the new comments (section 9.4 onwards)? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 01/10/2020 18:28 Subject: Re: [EXTERNAL] [DFDL-WG] DFDL Spec Working Draft r29 Mike Good review session on the WG call today. Updates to comments made during the call added. [attachment "gwdrp-dfdl-v1.0.5-r29-changes-tracked-SMH-comments.odt" deleted by Steve Hanson/UK/IBM] Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 01/10/2020 15:53 Subject: Re: [EXTERNAL] [DFDL-WG] DFDL Spec Working Draft r29 Mike Have reviewed up to start of 9.3. Comments added. Note I used LibreOffice. We can start going through them on call today if you like. [attachment "gwdrp-dfdl-v1.0.5-r29-changes-tracked-SMH-comments.odt" deleted by Steve Hanson/UK/IBM] Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 29/09/2020 23:29 Subject: [EXTERNAL] [DFDL-WG] DFDL Spec Working Draft r29 Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Steve H., and DFDL Workgroup, I have uploaded version r29 of the DFDL spec as MS-Word ".docx" This has all the comments from the external review by Martin Westhead applied to it, including numerous forward references fixed. In general the material flows better (IMHO) and is better organized with a number of out-of-place discussions/paragraphs repositioned to be in a better place with other related material. I believe this is a substantially better document now. I have tried to mark the moves of sections of text with comment bubbles. I'm sure there are some missing however. Not every comment bubble is resolved as yet. Comment bubbles marked TBD mean a change is still expected or an issue to be resolved. Some of these are calls for new material such as new clarifying examples, and we may or may not decide to put them in. Comment bubbles marked "Note" are FYI for read/review and may not involve any further change. Basically, it says "Note" if I currently think the issue that motivated the change has been resolved, but you may have a different opinion. You will find two variants of the document: one has MS-Word change tracking, the other has all changes accepted. When sections were renumbered I had to turn off MS-Word change tracking. So you'll notice that even in the track-changes version, renumberings are (mostly) not marked as tracked changes. It is my hope that after another read-through the renumbering and reorganization are accepted and we can go forward from here using the all-changes-accepted version, though I realize in review it may be important to see what text has been deleted/inserted via the change-tracked variant. It is also my hope that we are getting close to a draft ready for public review. Saving reviewer/proofread time is the most important thing, so you may print it out, and redline it with red ink if you wish, and ship the result to me via courier/fedex. Or if possible you can PDF it, and can virtual red-ink it with a tablet computer/stylus, and e-send the PDF to me. This is the model Martin Westhead used when reviewing, and it worked well for me. But basically, I will accept feedback/review in any form: MS-Word comment bubbles are fine, as is red-ink on paper. One possibly useful technique: If you open the "tracked" variant in MS-Word, and from the review tab, choose "Simple Markup" as the change display mode, you will get just margin side-bars marking where changes are, and the comment bubbles, but otherwise not too much change noise. You can click on any given sidebar and it will switch to "All Markup" mode showing you the detailed tracked changes, but you can switch back to "Simple Markup" after reviewing the details. I have started keeping these files on the OGF Github under docs/working-drafts, they are also on redmine in the current documents. OGF Github DFDL repository has URLs: https://github.com/OpenGridForum/DFDL.git or if you prefer ssh: git@github.com:OpenGridForum/DFDL.git Redmine the two docs are: https://redmine.ogf.org/dmsf_files/13621?download= https://redmine.ogf.org/dmsf_files/13622?download= -mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1