July 2020 - dfdl-wg - lists.ogf.org

Re: [DFDL-WG] DFDL pointer & offset proposal
by Steve Hanson 24 Jul '20

24 Jul '20

Bradd, Mike - as discussed on WG call please review my version of the grammar for next call. The key issue is how to link the target of the indirection into the main grammar, as it is dis-contiguous. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 23/07/2020 14:52 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I incorporated some of your grammar changes but had a few modifications of my own we can discuss further in case I'm not understanding some of the subtleties. I think there may be a case for a nil value with indirection however as well. I've updated the document to include the changes as version 4. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 07/10/2020 08:11 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red ) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement AbsentElementRep = Absent IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: [attachment "DFDL_Indirection_v3.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Latest OGF DFDL WG Call Minutes
by Steve Hanson 24 Jul '20

24 Jul '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

clarification needed: choice with direct dispatch is/is-not a PoU
by Mike Beckerle 24 Jul '20

24 Jul '20

Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU. But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch." Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be. I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 7

Re: [DFDL-WG] DFDL pointer & offset proposal
by Bradd Kadlecik 23 Jul '20

23 Jul '20

I incorporated some of your grammar changes but had a few modifications of my own we can discuss further in case I'm not understanding some of the subtleties. I think there may be a case for a nil value with indirection however as well. I've updated the document to include the changes as version 4. (See attached file: DFDL_Indirection_v4.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 07/10/2020 08:11 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions --------------------------------------------------------------------------------------- Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement --------------------------------------------------------------------------------------- AbsentElementRep = Absent --------------------------------------------------------------------------------------- IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue --------------------------------------------------------------------------------------- SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment --------------------------------------------------------------------------------------- SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill --------------------------------------------------------------------------------------- SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice --------------------------------------------------------------------------------------- Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement --------------------------------------------------------------------------------------- LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: [attachment "DFDL_Indirection_v3.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 22 Jul '20

22 Jul '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13616?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group Call
by Steve Hanson 22 Jul '20

22 Jul '20

1 0

ODPi standardization effort
by Sill, Alan 10 Jul '20

10 Jul '20

Thought you would be interested in the following link. Topic: ODPi creates open source standards to help you use and understand data across all platforms. ODPi is a non-profit organization supported by The Linux Foundation and dozens of individuals and member organizations. Members are committed to simplifying and standardizing the big data ecosystem so that data can be easily and securely shared across products, platforms, and systems. Learn more about the people and organizations involved in ODPi or to access publicly available resources. Link: https://www.odpi.org/ It might be nice for this organization to learn about DFDL and other OGF standards to promote greater innovation and adoption. Alan

1 0

Re: [DFDL-WG] what do you think of this for Section 22 Property Precedence
by Steve Hanson 10 Jul '20

10 Jul '20

Hi Mike - did you get any time to think about the grammar driven approach to section 22 ? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 30/04/2020 18:35 Subject: Re: [EXTERNAL] [DFDL-WG] what do you think of this for Section 22 Property Precedence At the end of the call, Steve wondered whether an alternative approach might be better - relate properties to grammar constructs. Then everything is driven from the grammar. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 30/04/2020 00:02 Subject: [EXTERNAL] [DFDL-WG] what do you think of this for Section 22 Property Precedence Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> I just revamped the first section in the attached MS-Word nested table structure. This is tedious to assemble, but I think does a better job than just the indented trees particularly given the fact that we ran out of indent levels. I'm curious whether this is better/clearer. There's also a few comment bubbles where I discovered things I didn't understand or were surprising to me. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 10 Jul '20

10 Jul '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Re: [DFDL-WG] DFDL pointer & offset proposal
by Steve Hanson 10 Jul '20

10 Jul '20

I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red ) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement AbsentElementRep = Absent IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0