- dfdl-wg - lists.ogf.org

clarification needed? dfdl:textNumberCheckPolicy 'strict' - language suggests more strict than ICU libraries
by Mike Beckerle 02 Sep '20

02 Sep '20

Say we have this schema snippet: <xs:element name="SimpleDataFormat"> <xs:complexType> <xs:sequence> <xs:element name="NumStudents" type="xs:nonNegativeInteger" dfdl:textNumberCheckPolicy="strict" dfdl:textNumberPattern="#,###" dfdl:textStandardGroupingSeparator="," dfdl:textStandardDecimalSeparator="." /> </xs:sequence> </xs:complexType></xs:element> This successfully parses the data 1234 Even though textNumberCheckPolicy="strict" and the pattern contains a grouping separator, it still allows data that does not contain grouping separators. That said, we have generally tried to make DFDL's spec match the behavior of the ICU library for parsing numbers based on the textNumberPattern. This library has this to say about strict parsing of numbers: The following conditions cause a parse failure relative to [lax] mode (examples use the pattern "#,##0.#"): - The presence and position of special symbols, including currency, must match the pattern. '+123' fails (there is no plus sign in the pattern) - Leading or doubled grouping separators ',123' and '1,,234" fail - Groups of incorrect length when grouping is used '1,23' and '1234,567' fail, but '1234' passes - Grouping separators used in numbers followed by exponents '1,234E5' fails, but '1234E5' and '1,234E' pass ('E' is not an exponent when not followed by a number) So based on ICU's description of strict, this is the expected behavior. It doesn't say anything about missing grouping separators causing an error. Only that if they do exist then they must be in the right spot. The only thing the DFDL specification mentions regarding strict numbers is this: If 'strict' and dfdl:textNumberRep is 'standard' then the data must follow the pattern with the exceptions that digits 0-9, decimal separator and exponent separator are always recognised and parsed To me, that reads like the decimal separator should always be required in strict mode, so this feels like the ICU behavior and the behavior described in the DFDL specification do not match. And I believe the DFDL behavior was intended to match ICU behavior, so it's possible the DFDL specification needs to be updated. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 18 Aug '20

18 Aug '20

1 0

OGF DFDL WG Call 2020-08-06 cancelled
by Steve Hanson 06 Aug '20

06 Aug '20

Unable to make DFDL WG call today. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 06 Aug '20

06 Aug '20

1 0

clarification: DFDL behavior of intersect, except operators
by Mike Beckerle 04 Aug '20

04 Aug '20

Some time back there was an erratum to add intersect and except operators from XPath 2.0 to DFDL. I believe these were in use by the DFDL4S flavor of Daffodil, and so were added to DFDL v1.0. Are these operators restricted to accept or to produce 0 or 1 node only, or are they able to accept and produce lists of nodes that can then be counted with fn:count, for example? I guess I have lost track of the use-cases for these. The XPath 2.0 definition of these operators is of course fully general, accepting multiple node sequences, and producing multiple-node sequences. That would mean the ultimate result of these operators could be a sequence of more than one node, and that means it would have to be an argument to fn:count, fn:exists, .... not sure what other functions. They could never be the direct result-producing part of an expression. Can someone point me at the use case for these again? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Re: [DFDL-WG] DFDL pointer & offset proposal
by Steve Hanson 24 Jul '20

24 Jul '20

Bradd, Mike - as discussed on WG call please review my version of the grammar for next call. The key issue is how to link the target of the indirection into the main grammar, as it is dis-contiguous. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 23/07/2020 14:52 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I incorporated some of your grammar changes but had a few modifications of my own we can discuss further in case I'm not understanding some of the subtleties. I think there may be a case for a nil value with indirection however as well. I've updated the document to include the changes as version 4. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 07/10/2020 08:11 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red ) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement AbsentElementRep = Absent IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: [attachment "DFDL_Indirection_v3.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Latest OGF DFDL WG Call Minutes
by Steve Hanson 24 Jul '20

24 Jul '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

clarification needed: choice with direct dispatch is/is-not a PoU
by Mike Beckerle 24 Jul '20

24 Jul '20

Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU. But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch." Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be. I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 7

Re: [DFDL-WG] DFDL pointer & offset proposal
by Bradd Kadlecik 23 Jul '20

23 Jul '20

I incorporated some of your grammar changes but had a few modifications of my own we can discuss further in case I'm not understanding some of the subtleties. I think there may be a case for a nil value with indirection however as well. I've updated the document to include the changes as version 4. (See attached file: DFDL_Indirection_v4.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 07/10/2020 08:11 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions --------------------------------------------------------------------------------------- Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement --------------------------------------------------------------------------------------- AbsentElementRep = Absent --------------------------------------------------------------------------------------- IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue --------------------------------------------------------------------------------------- SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment --------------------------------------------------------------------------------------- SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill --------------------------------------------------------------------------------------- SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice --------------------------------------------------------------------------------------- Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement --------------------------------------------------------------------------------------- LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: [attachment "DFDL_Indirection_v3.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 22 Jul '20

22 Jul '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13616?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0