- dfdl-wg - lists.ogf.org

[ANNOUNCE] Apache Daffodil (incubating) 3.0.0 Released
by Mike Beckerle 23 Nov '20

23 Nov '20

The Apache Daffodil (incubating) community is pleased to announce the release of version 3.0.0. Notable changes in this release include SAX support, streaming, reduced memory usage, improved diagnostics, and bug fixes. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/3.0.0/ Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team ===== *Disclaimer* Apache Daffodil is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 12 Nov '20

12 Nov '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 12 Nov '20

12 Nov '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13628?download= Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Re: [DFDL-WG] DFDL pointer & offset proposal
by Bradd Kadlecik 11 Nov '20

11 Nov '20

I've updated the proposal for the following issues: 1) Undefined term "underlying elements" used in indirectionUnusedValue description 2) Undefined term "prior element" used in indirectionBase description 3) Grammar needs to have indirection elements follow document element (See attached file: DFDL_Indirection_v6.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 07/24/2020 01:08 PM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal "Actually, the target of the indirection is just like an element or model-group, except without alignment regions. Ugly to have to repeat the whole grammar for that. ". Correct, and I haven't. My changes are for the indirection value, not the target. We have to handle 'indirectionUnusedValue' and reflect that initiator & terminator do not apply. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 24/07/2020 13:59 Subject: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal Actually, the target of the indirection is just like an element or model-group, except without alignment regions. Ugly to have to repeat the whole grammar for that. Perhaps we can get away with just documenting that the alignment regions must be zero length when target of an indirection? I did not try to put this modified grammar into my commented copy of the proposal (attached). The "connection" to the indirect pointer or offset is not something that the grammar can express. Attached * my review (with new comments that start with "V4:"), * my first cut of a DFDL schema for TIFF v6.0, * a copy of the TIFF v6.0 spec. I wasn't actually able to pull off TIFF. Comments in the schema discuss why, which is due to TIFF being a linked list, and I'm not able to terminate the chain. There's maybe a trick using negative offsets, but I don't want to consider that yet, as those are entirely problematic. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Jul 24, 2020 at 7:03 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Bradd, Mike - as discussed on WG call please review my version of the grammar for next call. The key issue is how to link the target of the indirection into the main grammar, as it is dis-contiguous. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 23/07/2020 14:52 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I incorporated some of your grammar changes but had a few modifications of my own we can discuss further in case I'm not understanding some of the subtleties. I think there may be a case for a nil value with indirection however as well. I've updated the document to include the changes as version 4. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 07/10/2020 08:11 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions --------------------------------------------------------------------------------------- Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement --------------------------------------------------------------------------------------- AbsentElementRep = Absent --------------------------------------------------------------------------------------- IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue --------------------------------------------------------------------------------------- SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment --------------------------------------------------------------------------------------- SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill --------------------------------------------------------------------------------------- SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice --------------------------------------------------------------------------------------- Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement --------------------------------------------------------------------------------------- LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: [attachment "DFDL_Indirection_v3.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "TIFF-v6.pdf" deleted by Bradd Kadlecik/Poughkeepsie/IBM] [attachment "DFDL_Indirection_v4_mjb_comments.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] [attachment "tiff.dfdl.xsd" deleted by Bradd Kadlecik/Poughkeepsie/IBM]

1 0

OGF DFDL Working Group Call
by Steve Hanson 11 Nov '20

11 Nov '20

1 0

Action 320: Migration to Github from redmine trackers
by Mike Beckerle 05 Nov '20

05 Nov '20

We have 3 trackers left on redmine for DFDL v1.0 version. I propose these be re-created as github issues as public comments, and they be closed on redmine. The 16 redmine issues targeted at DFDL v2.0 version should become issue trackers on the DFDL project repository. Note that this is a separate pool of trackers from the public comments, which are trackers on the editor github repo, not the DFDL repo. That would then end use of redmine trackers. Going forward, if a public comment is resolved by being tracked for consideration in DFDL v2.0, then the public comment would be closed and a DFDL project issue opened reflecting the content of the public comment. As for the documents, I am not yet proposing a migration strategy for that, nor am I clear on what the replacement on github is for redmine "news". The trackers, documents, and news appear to be the only redmine features we were using. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Action 289: outputValueCalc refers back to expression that refers forward
by Mike Beckerle 03 Nov '20

03 Nov '20

One of the open questions w.r.t. this action item is whether DFDL is missing a property on dfdl:newVariableInstance and dfdl:setVariable to indicate if they are to be evaluated at parse, unparse, or both. I have updated the PCAP schema on github to illustrate this issue. This is a pull request (code review) of the changes https://github.com/DFDLSchemas/PCAP/pull/10 In the PCAP schema the IPSrcGrp group ref appears immediately before the IPSrc element. Similarly the IPDestGrp reference appears immediately before the IPDest element. These groups contain elements which are the individual binary bytes of the IP Source and IP Dest addresses. They are 4 unsigned bytes. When parsing these are parsed and then their values concatenated as strings into a string like "1.2.3.4". When unparsing, these groups use dfdl:outputValueCalc to take apart a string like "1.2.3.4" (which appears in the infoset after parsing), back into its individual digits 1, 2, 3, 4, and creates an unsignedByte value of each number. The problem is, these groups are absolute nonsense if you view the dfdl:newVariableInstance defaultValue expression as being evaluated at parse time. The expression is for use *only* at unparse time and immediately does a forward reference to the following IPSrc member containing the "1.2.3.4" type string. Without some sort of additional property on dfdl:newVariableInstance and dfdl:setVariable to indicate that this only should be evaluated at unparse time, there is no way to suppress evaluation and perform it only at unparse time. I suggest we need an additional property on newVariableInstance and setVariable which I suggest is called 'direction', with values 'parse', 'unparse', 'both'. This property defaults to 'both', but if set to 'unparse' only evaluates the expression (and binds/sets variable) during that direction of processing. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 29 Oct '20

29 Oct '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 28 Oct '20

28 Oct '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13627?download= Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 15 Oct '20

15 Oct '20

Today's WG call is replaced by a DFDL 1.0 draft spec review session, involving Mike & Steve. (Saves me having to create agenda & issue minutes :) Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0