- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] complex nillable ES restriction - more
by Steve Hanson 28 Apr '15

28 Apr '15

Some more on this ... Here's the words from section 13.15 of the spec: DFDL allows elements of complex type to be nillable. However, to avoid the concept of a complex element having a value, which does not exist in DFDL, the only permissible nil value is the empty string, represented by the DFDL %ES; entity and using dfdl:nilKind 'literalValue'. Here's the original words from experience document #2 on this. Note the lengthKind 'implicit' restriction. For a complex element, length plus initiator and terminator enables the nil representation to be established (It is a schema definition error if a complex element is nillable ‘true’ and lengthKind ‘implicit’), but all other representations can only be determined by descending into the complex type for the element. If the descent returns successfully (that is, no unsuppressed processing error occurs) then the other representations may be established. So ... if you can establish the length of the complex element up front then potentially you could check against any nil literal value. Such a check would be different from that for a simple element though, as pad characters are trimmed first, and those properties do not apply to complex elements. I am not sure about lengthKind 'delimited'. With the current ES only rule, the parser can look at the next byte and if it immediately finds an in-scope delimiter, it can match ES. If any nil literal value is allowed, the parser is going to have to scan. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 28/04/2015 09:23 Subject: Re: [DFDL-WG] complex nillable ES restriction It's not an ad-hoc restriction. It's because a complex element has no value, so there are no DFDL properties to describe a value. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 27/04/2015 19:06 Subject: [DFDL-WG] complex nillable ES restriction Sent by: dfdl-wg-bounces(a)ogf.org Right now complex elements can be nillable, but we have this ad-hoc restriction that says the nilValue can only be %ES;. The rationale for this is unclear to me. Can we review what the rationale for this restriction was? Does anyone recall? We have a format where the literal nil value for the complex type wants to be "%WSP*;-%WSP*;" that is, a hyphen, but with surrounding whitespace absorbed. We can model this a different way, but the natural thing to do is to model it as suggested. (This also just happens to run on Daffodil - because we're not detecting this ad-hoc restriction - a bug) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

spec bug - nillable properties say they're only allowed on elements of simple type.
by Mike Beckerle 28 Apr '15

28 Apr '15

The various nil-related properties all have this at the bottom of their descriptions: Annotation: dfdl:element(simpleType) The "(simpleType)" part should be dropped. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

complex nillable ES restriction
by Mike Beckerle 28 Apr '15

28 Apr '15

Right now complex elements can be nillable, but we have this ad-hoc restriction that says the nilValue can only be %ES;. The rationale for this is unclear to me. Can we review what the rationale for this restriction was? Does anyone recall? We have a format where the literal nil value for the complex type wants to be "%WSP*;-%WSP*;" that is, a hyphen, but with surrounding whitespace absorbed. We can model this a different way, but the natural thing to do is to model it as suggested. (This also just happens to run on Daffodil - because we're not detecting this ad-hoc restriction - a bug) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Action 228: TDML file with tutorial content - mock-up
by Mike Beckerle 27 Apr '15

27 Apr '15

This is a mock up of how a TDML file might be turned into a self-contained executable from which tutorial HTML can be directly generated. I started from tutorial 6. The HTML file should be viewable in any browser. Keep in mind it lacks a corresponding stylesheet, so it's rather ugly as HTML pages go, but it does use this very nice javascript formatter for the data and XML sections which does colorizing (which can be customized as to colors/shading, etc.). I am not sure how to reproduce the bolding of the interesting lines that is done in the original schemas of the tutorial. Nor is there a robust way to refer to line numbers without counting them. But I still think this is pretty interesting. I think the next exercise would be trying to use HTML Table nesting to provide richer data descriptions for binary data where nested boxes are really necessary. I'm thinking along the lines of: <document tutorialBoxes="[ ]">[...abc][def[ghi]jkl][mnop][qrs...]</document> translating the brackets there into boxes that nest in the expected manner. If data contains brackets, then the tutoriaBoxes attribute would be set to use different characters not found in the data. I want to have there be some way to provide data pictures which show nested boxes without a lot of effort.

1 0

OGF DFDL WG Call Agenda 2015-04-28
by Steve Hanson 27 Apr '15

27 Apr '15

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13421?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] OCK expression and count of 0 for a choice member....
by Alex Wood1 27 Apr '15

27 Apr '15

Hi Mike, Can you clarify if you are saying that OCK expression should be prohibited completely on a choice member (as occurrences for OCK expression are potentially optional regardless of minOccurs value) Or is your statement that it should cause an SDE specific to the count==0 case? Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Alex Wood1/UK/IBM@IBMGB Date: 24/04/2015 15:10 Subject: Re: [DFDL-WG] OCK expression and count of 0 for a choice member.... I think this is an SDE. Choice branches cannot be optional. Optional here, does not mean minOccurs == 0, because for many occursCountKinds, that's never checked unless validation is on, and validation doesn't guide parsing anyway. Optional here means "not required by the DFDL format", as in occursCountKind cannot be 'parsed' at all, because all occurrences are then not required, and the min/maxOccurs are only examined for validation purposes, also occursCountKind cannot be 'implicit' for the same reasons, and occursCountKind 'expression' also. Wrapping the array element in a sequence doesn't solve the problem unless the sequence has a required piece of syntax such as an initiator or terminator, or a hiddenGroupRef to a not-optional (recursively) thing. Even initiator and terminator are tricky, because in a non-delimited format, those can be %WSP*; which can match nothing at all; hence, they do not "require" any syntax. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Apr 24, 2015 at 9:07 AM, Alex Wood1 <WOODA(a)uk.ibm.com> wrote: Hi All, Please see below for a history of the issue. This arose from fuzz testing of the IBM DFDL parser which produced a test with a coutn of 0 for an OCK expression array which was a choice member. And subsequent reference to the specification. It was not clear what the correct outcome should be in a choice where the first member is an array using OCK expression where the count resolves to 0. a.) resolve the choice to the zero length array b.) move to the next choice branch c.) throw an error Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com From: Steve Hanson/UK/IBM To: Alex Wood1/UK/IBM@IBMGB Cc: Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM Date: 24/04/2015 09:19 Subject: Re: OCK expression and count of 0 for a choice member.... When I wrote the paragraph below, the one thing that troubled me was that the spec defines known-to-exist and known-not-to-exist in terms of occurrences. In the choice branch example, it is the element as a whole we are looking at. That's fine for scalar as element == occurrence but for an array it's not the same. I think the spec is missing a definition of what 'missing' means for an array element. I would say that an array element is missing if all occurrences are missing. And an array element is not missing if any occurrence has a representation (empty, nil, normal). With that in place, my paragraph makes sense, I think. I believe we have the same issue with 'parsed' and 'stopValue'. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Alex Wood1/UK/IBM@IBMGB Cc: Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB Date: 23/04/2015 18:52 Subject: Re: OCK expression and count of 0 for a choice member.... Here is one interpretation... A choice is resolved by parsing the branches until one is known-to-exist as described in section 9.3.3. Section 9.3.1.2 defines known-to-exist (in the absence of a discriminator, initiator or direct dispatch) as an occurrence having empty, nil or normal representation. Section 9.3.1.3 defines known-not-to-exist (again in the absence of a discriminator, initiator or direct dispatchm or an assert) as an occurrence being missing or causing a processing error. If occursCount is zero no occurrences are looked for in the data (erratum 5.9) so the element has no representation and must be missing. Therefore a choice branch containing such an element is known-not-to-exist. So in your example, the first choice branch containing myInt is known-not-to-exist and the parser tries the next branch. This appears to contradict section 15.1.1 though. I suspect that 15.1.1 was not updated to match section 9.3 when the latter was added. If you want to make the first choice branch known-to-exist when the count is zero then I think wrapping myInt in a sequence would work. Or wrapping myInt in a complex element. Definitely one to take to the WG though, if only to correct section 15.1.1 to match section 9. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB Date: 23/04/2015 16:33 Subject: OCK expression and count of 0 for a choice member.... Hi Steve Just been discussing this with Andy and Mark. I think the spec <xs:element name="Choice_Expression" dfdl:ref="config" dfdl:lengthKind="implicit"> <xs:complexType> <xs:sequence dfdl:ref="config"> <xs:element ref="myCount"></xs:element> <xs:choice dfdl:choiceLengthKind="implicit" dfdl:ref="config"> <xs:element ref="myInt" minOccurs="1" maxOccurs="3"></xs:element> <xs:element ref="myTxt"></xs:element> </xs:choice> </xs:sequence> </xs:complexType> </xs:element> Where myInt has occursCountKind="expression" occursCount="{../myCount}" A given instance of this message could have myCount==0 Is this valid? Should it resolve to 0 occurrences of myInt or move on to myTxt ? Section15 of the spec says: The Root of the Branch MUST NOT be optional. That is XSDL minOccurs MUST BE greater than 0. But in this case minOccurs is >0. Assuming this is not an error then in terms of resolving the choice section 15.1.1 says.. 15.1.1 Resolving Choices via Speculation Speculative resolution works as follows: 1) Attempt to parse the first branch of the choice. 2) If this fails with a processing error a) If a dfdl:discriminator evaluated to true earlier on this branch then the parser is 'bound' to this branch and parsing of the entire choice construct fails with a processing error. b) If the branch has a dfdl:initiator and the choice has dfdl:initiatedContent ‘yes’ then the parser is 'bound' to this branch and parsing of the entire choice construct fails with a processing error. c) Otherwise we repeat from step 1 for the next branch of the choice. 3) It is a processing error if the branches of the choice are exhausted. 4) If a branch is successfully parsed without error, then that branch's infoset becomes the infoset for the parse of the choice construct. So seems like this is 4.) we did not fail to parse myInt... However talking with mark about real scenarios that this might apply to, a choice two repeating fields with counts earlier in the data only one of which must appear. you'd expect 0 of the first means >0 of the second and visa versa... So you'd probably want 0 myInt allowed the choice to resolve to myTxt. Thoughts ? If you agree we need more clarity in he spec will forward to WG. Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

Re: [DFDL-WG] Fw: typographical errors - broken cross refs
by Steve Hanson 27 Apr '15

27 Apr '15

Mike The broken references come from the GFD.207 spec original, so need fixing there. I think both references should be to Table 2 in section 6.3.1.2. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 24/04/2015 09:11 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 23/04/2015 23:44 Subject: [DFDL-WG] typographical errors - broken cross refs Sent by: dfdl-wg-bounces(a)ogf.org Broken cross references - Found in the HTML spec: 6.3.1.4 DFDL Byte Value Entities in DFDL String Literals DFDL byte value entities provide a way to specify a single byte as it appears in the data stream without any character set encoding translation. To specify a string of byte values, a sequence of two or more byte value entities must be used. The syntax is in Error! Reference source not found. above. Example: 33.1 Interpreting a DFDL String Literal A DFDL String literal is a sequence of one or more string literal parts. Each string literal part describes zero or more characters or exactly one byte. To process a DFDL string literal correctly it is necessary to parse it into its string literal parts using the grammar in Error! Reference source not found. of Section 6.3.1 DFDL String Literals. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] OCK expression and count of 0 for a choice member....
by Alex Wood1 24 Apr '15

24 Apr '15

Hi All, Please see below for a history of the issue. This arose from fuzz testing of the IBM DFDL parser which produced a test with a coutn of 0 for an OCK expression array which was a choice member. And subsequent reference to the specification. It was not clear what the correct outcome should be in a choice where the first member is an array using OCK expression where the count resolves to 0. a.) resolve the choice to the zero length array b.) move to the next choice branch c.) throw an error Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com From: Steve Hanson/UK/IBM To: Alex Wood1/UK/IBM@IBMGB Cc: Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM Date: 24/04/2015 09:19 Subject: Re: OCK expression and count of 0 for a choice member.... When I wrote the paragraph below, the one thing that troubled me was that the spec defines known-to-exist and known-not-to-exist in terms of occurrences. In the choice branch example, it is the element as a whole we are looking at. That's fine for scalar as element == occurrence but for an array it's not the same. I think the spec is missing a definition of what 'missing' means for an array element. I would say that an array element is missing if all occurrences are missing. And an array element is not missing if any occurrence has a representation (empty, nil, normal). With that in place, my paragraph makes sense, I think. I believe we have the same issue with 'parsed' and 'stopValue'. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Alex Wood1/UK/IBM@IBMGB Cc: Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB Date: 23/04/2015 18:52 Subject: Re: OCK expression and count of 0 for a choice member.... Here is one interpretation... A choice is resolved by parsing the branches until one is known-to-exist as described in section 9.3.3. Section 9.3.1.2 defines known-to-exist (in the absence of a discriminator, initiator or direct dispatch) as an occurrence having empty, nil or normal representation. Section 9.3.1.3 defines known-not-to-exist (again in the absence of a discriminator, initiator or direct dispatchm or an assert) as an occurrence being missing or causing a processing error. If occursCount is zero no occurrences are looked for in the data (erratum 5.9) so the element has no representation and must be missing. Therefore a choice branch containing such an element is known-not-to-exist. So in your example, the first choice branch containing myInt is known-not-to-exist and the parser tries the next branch. This appears to contradict section 15.1.1 though. I suspect that 15.1.1 was not updated to match section 9.3 when the latter was added. If you want to make the first choice branch known-to-exist when the count is zero then I think wrapping myInt in a sequence would work. Or wrapping myInt in a complex element. Definitely one to take to the WG though, if only to correct section 15.1.1 to match section 9. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB Date: 23/04/2015 16:33 Subject: OCK expression and count of 0 for a choice member.... Hi Steve Just been discussing this with Andy and Mark. I think the spec <xs:element name="Choice_Expression" dfdl:ref="config" dfdl:lengthKind="implicit"> <xs:complexType> <xs:sequence dfdl:ref="config"> <xs:element ref="myCount"></xs:element> <xs:choice dfdl:choiceLengthKind="implicit" dfdl:ref="config"> <xs:element ref="myInt" minOccurs="1" maxOccurs="3"></xs:element> <xs:element ref="myTxt"></xs:element> </xs:choice> </xs:sequence> </xs:complexType> </xs:element> Where myInt has occursCountKind="expression" occursCount="{../myCount}" A given instance of this message could have myCount==0 Is this valid? Should it resolve to 0 occurrences of myInt or move on to myTxt ? Section15 of the spec says: The Root of the Branch MUST NOT be optional. That is XSDL minOccurs MUST BE greater than 0. But in this case minOccurs is >0. Assuming this is not an error then in terms of resolving the choice section 15.1.1 says.. 15.1.1 Resolving Choices via Speculation Speculative resolution works as follows: 1) Attempt to parse the first branch of the choice. 2) If this fails with a processing error a) If a dfdl:discriminator evaluated to true earlier on this branch then the parser is 'bound' to this branch and parsing of the entire choice construct fails with a processing error. b) If the branch has a dfdl:initiator and the choice has dfdl:initiatedContent ‘yes’ then the parser is 'bound' to this branch and parsing of the entire choice construct fails with a processing error. c) Otherwise we repeat from step 1 for the next branch of the choice. 3) It is a processing error if the branches of the choice are exhausted. 4) If a branch is successfully parsed without error, then that branch's infoset becomes the infoset for the parse of the choice construct. So seems like this is 4.) we did not fail to parse myInt... However talking with mark about real scenarios that this might apply to, a choice two repeating fields with counts earlier in the data only one of which must appear. you'd expect 0 of the first means >0 of the second and visa versa... So you'd probably want 0 myInt allowed the choice to resolve to myTxt. Thoughts ? If you agree we need more clarity in he spec will forward to WG. Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

typographical errors - broken cross refs
by Mike Beckerle 23 Apr '15

23 Apr '15

Broken cross references - Found in the HTML spec: 6.3.1.4 DFDL Byte Value Entities in DFDL String Literals DFDL byte value entities provide a way to specify a single byte as it appears in the data stream without any character set encoding translation. To specify a string of byte values, a sequence of two or more byte value entities must be used. The syntax is in *Error! Reference source not found.* above. Example: 33.1 Interpreting a DFDL String Literal A DFDL String literal is a sequence of one or more *string literal parts*. Each string literal part describes zero or more characters or exactly one byte. To process a DFDL string literal correctly it is necessary to parse it into its string literal parts using the grammar in *Error! Reference source not found.* of Section 6.3.1 DFDL String Literals. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

typographical
by Mike Beckerle 23 Apr '15

23 Apr '15

A simple element must have one of: type xs:string dfdl:representation 'text' type xs:hexBinary dfdl:representation 'binary' and a packed decimal representation Is missing the bullets. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0