- dfdl-wg - lists.ogf.org

Consolidated Notes including both DFDL WG Call 2018-08-07 along with other recent clarification email threads
by Mike Beckerle 03 Jul '19

03 Jul '19

This message includes two parts. Part 1 is the things we discussed on the DFDL WG Call of 2018-08-07. Part 2 is the other recent emails where the conclusion from the email thread is repeated here for finalization/refinement, and for consolidation so all these changes can be considered together. ================================ Part 1 - Discussed on the call. Re: [DFDL-WG] Clarification discussion points for next call(s) - DFDL Spec issues around nil, empty, normal, absent, defaulting, and separator suppression. Dated August 2nd Conclusions: Section 9.3.1.1 Delete phrase "...this of course implies that....". Note that there is already a correction to create numbered bullets of 3 sentences. The sentence containing this phrase will be bullet #3 of that list. Section 9.4 Item 2 under "For elements and element refs:" Change to: "dfdl:element following property scoping rules, which includes establishing representation as described in Section 9.3.2 and conversion to element type for simple types." Section 9.3.2 The phrase "The first step is to see if the content is trivaill of length zero." Change to: "The first step is to see if the SimpleContent or ComplexContnet region is of length zero as a first approximation." The bullet "delimited => length is zero (delimiter is immediately encountered)" Insert "in scope" after the open parenthesis. Section 9.4.2.3. We agreed that the paragraphs beginning with "For both required and optional..." need to be better tied to the material above. Wording TBD - pending Steve Hanson doing some tests on IBM DFDL. ============================ Part 2 - Below are conclusions from the prior email threads. This is for email review in lieu of discussion on this week's call. Re: [DFDL-WG] clarification: on suppressed ZL string/hexBinary - do we keep variable assignments?Of Aug 6 (last date in the thread) These corrections apply: Sections 9.4.2.2 and 9.4.2.3 The phrase "Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none'[12] <http://daffodil.apache.org/docs/dfdl/#_ftn12>," Change to "Optional occurrence: if dfdl:emptyValueDelimiterPolicy is applicable and is not 'none',...." (retaining the footnote) Section 9.4.2 Before the final phrase "There are three main cases to consider:" Insert this sentence: "The sections below indicate when an item is added to the infoset, and whether it has a default or other value. If there is no processing error then regardless of whether an item is added to the infoset or not, any side-effects due to dfdl:discriminator statements evaluating to true, or dfdl:setVariable statements, are retained." Section 12.2 For property emptyValueDelimiterPolicy, before the phrase "It is a schema definition error if...", insert this sentence: "The value of dfdl:emptyValueDelimiterPolicy should only be checked if there is a dfdl:initiator or dfdl:terminator in scope. If so, and dfdl:emptyValueDelimiterPolicy is not set, it is a schema definition error. If dfdl:initiator is not "" and dfdl:terminator is "" and dfdl:emptyValueDelimiterPolicy is 'terminator' it is a schema definition error. If dfdl:terminator is not "" and dfdl:initiator is " and dfdl:emptyValueDelimiterPolicy is 'initiator' it is a schema definition error." Section 13.16 For property nilValueDelimiterPolicy, before the phrase "It is a schema definition error if...", insert this sentence: "The value of dfdl:nilValueDelimiterPolicy should only be checked if there is a dfdl:initiator or dfdl:terminator in scope. If so, and dfdl:nilValueDelimiterPolicy is not set, it is a schema definition error. If dfdl:initiator is not "" and dfdl:terminator is "" and dfdl:nilValueDelimiterPolicy is 'terminator' it is a schema definition error. If dfdl:terminator is not "" and dfdl:initiator is " and dfdl:nilValueDelimiterPolicy is 'initiator' it is a schema definition error ." Section 9.2.2 The phrase "the occurrence's content in the data..." replace with "the occurrence's SimpleContent or ComplexContent region in the data..." The sentence: "The *empty representation* is special in DFDL, because when parsing it is this condition that can trigger the creation of a default value for an element occurrence." replace with: "The empty representation is special in DFDL because when parsing it it is used to determine when default values are created in the Infoset, and when optional recurring elements are omitted from the Infoset. The empty representation can require initiators or terminators be present so as to enable data formats to explicitly distinguish empty-string/hexBinary values (which might cause default values to be used) from emptiness meaning the absence of any representation." (This is to clarify an error of omission - prior language suggested that EVDP is only relevant when the element has a default value, because only that need was mentioned.) Re: [DFDL-WG] Clarification needed: separator for empty sequence Of Aug 2 Section 14.2 For property dfdl:separator. The sentence: "Separators occur in the data either before, between or after all occurrences of the elements or groups that are the children of the sequence." replaced with "Separators occur in the data either before, between or after all occurrences of *represented* elements (that is, elements without the dfdl:inputValueCalc property) or model groups that are the children of the sequence. Elements with dfdl:inputValueCalc have no representation in the data stream, and so never have separators. Children of a sequence that are model groups are always separated, even if they are empty (meaning have no children of their own - which is allowed for sequence groups), or both the model group child and its contained children occupy zero-length in the data stream." (note: Some of the above is redundant with stipulations in the dfdl:inputValueCalc property description, but I believe it is wise to have this little redundancy.) ====================== These email threads are mentioned here to indicate that they are resolved by one or another of the above corrections: Re: [DFDL-WG] clarification needed - ambiguity about empty string and optional element Of Aug 2 Re: [DFDL-WG] Spec correction ? - Section 9.3.2.1 - second list missing "empty" representationOf Aug 2 ----------- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 7

Latest OGF DFDL WG Call Minutes
by Steve Hanson 27 Jun '19

27 Jun '19

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 27 Jun '19

27 Jun '19

Please find agenda for today's call on Redmine at https://redmine.ogf.org/dmsf_files/13590?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 27 Jun '19

27 Jun '19

1 0

Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings
by Steve Hanson 25 Jun '19

25 Jun '19

Testing with IBM DFDL proves the behaviour for the zero-length normal rep case. It is a real edge case though, and on consideration I don't think we have to reflect it in the property name, as long as it is documented. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 25/06/2019 18:33 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Thinking this through, I don't think that IBM DFDL supporting dfdl:nilValue="%ES;" is a problem. It's the first check that is made when we get a rep with length 0. However I think that the enum "treatAsMissing" would be better as "treatAsAbsent" which is one of the specific cases of "missing" and what we technically fall through to. One subtlety not discussed is this .... 9.3.2.1 Simple element If the result is length zero as described above, the representation is then established by checking, in order for: 1. nil representation (if %ES; is a literal nil value). 2. empty representation. 3. normal representation (xs:string or xs:hexBinary only) 4. absent representation (if none of the prior representations apply). This is intended to handle the case when the length of the rep is 0 but we are not conforming with EVDP. For example, element has init <a> & term </a>, the EVDP is term only or init only, and the data contains "<a></a>". We are not conforming to EVDP, so can't be empty, but we are conforming to normal rep, not absent rep. The implication is that empty string would be added to the infoset. I'm pretty sure that IBM DFDL will not add anything to the infoset for this case, but I will test it to be sure. The proposed name of the property does not capture this subtlety though ... dfdl:emptyElementParsePolicy = ( "treatAsAbsent" | "treatAsEmpty" ) using 'zeroLength' instead of 'empty' goes too far the other away, as it encompasses the nilValue=%ES case. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 31/05/2019 14:42 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Re-reading this thread from the bottom, I'm not sure the proposal is correct. I may have over-simplified it. I'm going to have to do some more tests. Specifically around dfdl:nilValue="%ES;". IBM DFDL supports this, which means we can't just be treating empty elements as missing all the time. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/05/2019 08:40 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings It will hopefully be possible for you to apply the new 'treatAsMissing' enum to just 2 places in the Daffodil code: 1) Empty rep found for required occurrence -> processing error 2) Empty rep found for optional occurrence -> don't add anything to infoset Maybe 4 places if you have separate paths for simple v complex. That's an indicator of how conceptually simple this property is, once you know the difference between empty & missing. I am pretty sure the IBM DFDL behaviour deviation around empty/missing can be encapsulated by just this. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/05/2019 08:12 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings I prefer dfdl:emptyElementParsePolicy = ( "treatAsMissing" | "treatAsEmpty" ) You have to understand the difference between empty and missing in DFDL. It has an effect on all types - for example, if you set "treatAsMissing" for a required number, it means empty always causes a processing error instead of potentially applying a default. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 08/05/2019 19:26 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings I suggest we stick with the "...Policy" naming convention for new things that control modes of behavior. I'd prefer to avoid the terms empty and missing in the property values and go with something that is more explanatory of what difference it makes. E.g, emptyElementParsePolicy with values "excludeEmptyStringAndHexBinaryValues" and "allowEmptyStringAndHexBinaryValues" The doc for these values will of course have to be in terms of Absent/Missing/Empty, etc. but at least the names give some intuition as to what they control without having to understand all of DFDL's nuances about the difference between what Absent and Missing is. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, May 8, 2019 at 12:52 PM Steve Hanson <smh(a)uk.ibm.com> wrote: Maybe this is better; dfdl:parseEmptyAsMissing = yes | no Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 08/05/2019 16:48 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Interesting. Many DFDL schemas I've created have a simpleType defintion named "nzString" which is string, plus an assertion that it is non-empty. That's to achieve exactly the behavior you have in IBM DFDL, because, as you say, many formats want this. We could rename the suggested property emptyElementParsePolicy to make it clear it is only about parsing. I like treatAsMissing. Easy to say what it means. treatAsEmpty begs the question of what empty elements do, but that's already complicated in the spec due to optionals and EVDP, so I'm happy with this also. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, May 7, 2019 at 3:48 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Hi Mike I think what you have highlighted is that there are formats which require that empty elements should not be treated as empty but as missing, which is effectively what IBM DFDL is doing (our code was written prior to action 140 when there was no distinction between empty & missing). That could be achieved with assertions. So maybe we should view the new property as a convenience property for such formats, as well as handling IBM DFDL's behaviour? If so, then can I suggest new names for the enums, which I think makes the intent clearer? dfdl:emptyElementPolicy = ( "treatAsMissing" | "treatAsEmpty" ) This only applies when parsing, maybe names should reflect that also? Further, "treatAsMissing" would imply that a default value was never used when parsing, as they are only used when the representation is empty. I think we can do away with the SDE clause for "treatAsMissing". The clause is only needed for "treatAsEmpty". IBM DFDL does implement nillable processing, including use of ES as nil literal value. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/05/2019 21:23 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Under testing with the EDIFACT schema (from DFDLSchemas on github) against new code in daffodil, I see that my proposal was not sufficient. Steve Hanson stated that IBM DFDL current behavior for required empty strings includes "An empty occurrence with no default gives a Processing Error." I misinterpreted this. I was thinking required occurrence of an array element (as in with index <= minOccurs). But this should not be interpreted that narrowly, but any required occurrence at all including scalar elements. The EDIFACT schema depends on this behavior and backtracking driven by it, in order to work. So my suggestion for new properties to control this is revised to: dfdl:emptyElementPolicy enum with values noEmptyElements - matches current IBM DFDL behavior where * required elements without default values that are empty (specifically which satisfy the empty syntax - defined below) always cause Processing Errors. ** If a default value is specified that is provided as the value instead. When a default value is specified, then implementations that don't support default values when parsing must issue a runtime SDE here, not a processing error. * optional elements which satisfy the empty syntax are not added to the infoset. Defaulting is never considered. emptyElements - matches current description in the DFDL spec where * required elements: if the string/hexBinary satisfies the empty syntax then required elements are created with an empty string or empty hexBinary as their value. If a default value is specified that is substituted as the value instead. When a default value is specified, then implementations that don't support default values when parsing must issue a SDE here, not a processing error. * optional elements: if the string/hexBinary satisfies the empty syntax, and emptyValueDelimiterPolicy is not 'none' then an empty string (or hexbinary) is added to the infoset. If emptyValueDelimiterPolicy is 'none', nothing is added to the infoset. The term "satisfy the empty syntax" means what is found in the data stream may require initiator and/or terminator depending on emptyValueDelimiterPolicy, but if that is 'none' then this is satisfied just by empty string (or no bytes for hexBinary). Having said the above, I believe we also have to consider nillable elements. There are two topics: 1) defaulting to nilled - For the case of a nillable element, where the data syntax does NOT match the nil representation, then in the above anywhere a default value is specified, and there is behavior associated with that, well if the element is nillable, and dfdl:useNilAsDefault='true' is specified, then the element is default valued to being nilled. When nillable and dfdl:useNilAsDefault='true' is specified, then implementations that don't support defaulting to nilled when parsing must issue an SDE here, not a processing error. That takes care of the defaulting aspect of nillables. The second topic is: 2) nillable, and dfdl:nilValue contains %ES; as one of the possible nil representations. Hence, there is the possibility of empty string (or empty hexBinary) matching the nil representation. I think the DFDL spec is clear here that if the data stream satisfies the nil syntax, then required or optional, you get a nilled element, period. Does IBM DFDL implement that behavior? If so great. If not I think we may have to amend the above description of noEmptyElements case for dfdl:emptyElementPolicy to specify the special cases. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Sun, Apr 28, 2019 at 9:36 AM Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: One clarification: is the IBM DFDL behavior the same for empty hexBinary elements as it is for text strings? I'm going to suggest we need a policy property e.g., dfdl:emptyElementPolicy which is an enum with at least these options: noOptionalEmptyElements - matches current IBM DFDL behavior optionalEmptyElementsWithSyntax - matches current description in the DFDL spec where initiator and/or terminator found triggers creation of an empty string value. (Daffodil implements this.) This would apply (I think) to both types xs:string ad xs:hexBinary I'm open to suggestions for better naming for the property and the property values, but these are the two settings we need I think. I do believe that the latter optionalEmptyElementsWithSyntax behavior is what the DFDL spec describes, and is most consistent given the available properties such as emptyValueDelimiterPolicy. We can make implementation of optionalEmptyElementsWithSyntax a DFDL optional language feature, thereby avoiding issues of conformance with the DFDL standard. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Apr 5, 2019 at 12:43 PM Steve Hanson <smh(a)uk.ibm.com> wrote: Daffodil to perform identical tests but the belief is that they implement the spec as published (except maybe for one bug with default values for strings). So there is a mis-match between Daffodil and IBM DFDL. It sounds like a new property is going to be needed which toggles the way that empty strings are handled. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 03/04/2019 12:04 Subject: Action 306 - IBM DFDL behaviour when parsing empty strings 306 Confirm IBM DFDL behaviour when parsing empty strings (Steve) 7/8: IBM DFDL has not fully implemented the behaviour changes arising from action 140 with respect to empty string elements. Daffodil is about to do so. IBM DFDL users have complained about lack of defaults when parsing but other than that appear happy. Are the rules in the spec for empty strings over complicated? Steve to document the behaviour for IBM DFDL to inform the discussion. ... 1/11: In progress - there are a lot of subtle scenarios 15/11: Not discussed ... 7/2/19: No further progress Some progress :) 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the Infoset. IBM DFDL behaviour: Required. IBM DFDL does not implement default values when parsing, so an empty occurrence with a default value gives an SDE (to prevent backtracking). An empty occurrence with no default gives a Processing Error. If you need to add an empty string to the infoset, you can add default=""(when default values implemented, of course). Optional. IBM DFDL adds nothing to the infoset regardless of presence of initiator and/or terminator. No way to get empty string into the infoset. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then an empty string (type xs:string), empty hexBinary (type xs:hexBinary), or default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) IBM DFDL behaviour: Required. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). Optional. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). So ... The spec today is consistent in one way, in that for both complex & string elements a) a required empty occurrence always adds to the infoset; & b) an optional empty occurrence adds to the infoset if initiator/terminator present; & c) an optional empty occurrence does not add to the infoset if no initiator/terminator present. If the simple string behaviour was to change to match IBM DFDL then that consistency is lost, but the string behaviour then matches that for other simple types. Section 9.4.2.2 disappears as the behaviour is same as 9.4.2.1. Section 9.4.2.3 becomes as below. We lose the ability to get an empty string into the infoset for an optional string with initiator/terminator. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then a default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) We also need to be sure that any other implementations have not yet implemented the current spec behaviour. Need to check with DFDL4S and IBM TPF. To be discussed on next WG call ... Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings
by Steve Hanson 25 Jun '19

25 Jun '19

Thinking this through, I don't think that IBM DFDL supporting dfdl:nilValue="%ES;" is a problem. It's the first check that is made when we get a rep with length 0. However I think that the enum "treatAsMissing" would be better as "treatAsAbsent" which is one of the specific cases of "missing" and what we technically fall through to. One subtlety not discussed is this .... 9.3.2.1 Simple element If the result is length zero as described above, the representation is then established by checking, in order for: 1. nil representation (if %ES; is a literal nil value). 2. empty representation. 3. normal representation (xs:string or xs:hexBinary only) 4. absent representation (if none of the prior representations apply). This is intended to handle the case when the length of the rep is 0 but we are not conforming with EVDP. For example, element has init <a> & term </a>, the EVDP is term only or init only, and the data contains "<a></a>". We are not conforming to EVDP, so can't be empty, but we are conforming to normal rep, not absent rep. The implication is that empty string would be added to the infoset. I'm pretty sure that IBM DFDL will not add anything to the infoset for this case, but I will test it to be sure. The proposed name of the property does not capture this subtlety though ... dfdl:emptyElementParsePolicy = ( "treatAsAbsent" | "treatAsEmpty" ) using 'zeroLength' instead of 'empty' goes too far the other away, as it encompasses the nilValue=%ES case. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 31/05/2019 14:42 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Re-reading this thread from the bottom, I'm not sure the proposal is correct. I may have over-simplified it. I'm going to have to do some more tests. Specifically around dfdl:nilValue="%ES;". IBM DFDL supports this, which means we can't just be treating empty elements as missing all the time. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/05/2019 08:40 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings It will hopefully be possible for you to apply the new 'treatAsMissing' enum to just 2 places in the Daffodil code: 1) Empty rep found for required occurrence -> processing error 2) Empty rep found for optional occurrence -> don't add anything to infoset Maybe 4 places if you have separate paths for simple v complex. That's an indicator of how conceptually simple this property is, once you know the difference between empty & missing. I am pretty sure the IBM DFDL behaviour deviation around empty/missing can be encapsulated by just this. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/05/2019 08:12 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings I prefer dfdl:emptyElementParsePolicy = ( "treatAsMissing" | "treatAsEmpty" ) You have to understand the difference between empty and missing in DFDL. It has an effect on all types - for example, if you set "treatAsMissing" for a required number, it means empty always causes a processing error instead of potentially applying a default. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 08/05/2019 19:26 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings I suggest we stick with the "...Policy" naming convention for new things that control modes of behavior. I'd prefer to avoid the terms empty and missing in the property values and go with something that is more explanatory of what difference it makes. E.g, emptyElementParsePolicy with values "excludeEmptyStringAndHexBinaryValues" and "allowEmptyStringAndHexBinaryValues" The doc for these values will of course have to be in terms of Absent/Missing/Empty, etc. but at least the names give some intuition as to what they control without having to understand all of DFDL's nuances about the difference between what Absent and Missing is. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, May 8, 2019 at 12:52 PM Steve Hanson <smh(a)uk.ibm.com> wrote: Maybe this is better; dfdl:parseEmptyAsMissing = yes | no Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 08/05/2019 16:48 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Interesting. Many DFDL schemas I've created have a simpleType defintion named "nzString" which is string, plus an assertion that it is non-empty. That's to achieve exactly the behavior you have in IBM DFDL, because, as you say, many formats want this. We could rename the suggested property emptyElementParsePolicy to make it clear it is only about parsing. I like treatAsMissing. Easy to say what it means. treatAsEmpty begs the question of what empty elements do, but that's already complicated in the spec due to optionals and EVDP, so I'm happy with this also. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, May 7, 2019 at 3:48 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Hi Mike I think what you have highlighted is that there are formats which require that empty elements should not be treated as empty but as missing, which is effectively what IBM DFDL is doing (our code was written prior to action 140 when there was no distinction between empty & missing). That could be achieved with assertions. So maybe we should view the new property as a convenience property for such formats, as well as handling IBM DFDL's behaviour? If so, then can I suggest new names for the enums, which I think makes the intent clearer? dfdl:emptyElementPolicy = ( "treatAsMissing" | "treatAsEmpty" ) This only applies when parsing, maybe names should reflect that also? Further, "treatAsMissing" would imply that a default value was never used when parsing, as they are only used when the representation is empty. I think we can do away with the SDE clause for "treatAsMissing". The clause is only needed for "treatAsEmpty". IBM DFDL does implement nillable processing, including use of ES as nil literal value. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/05/2019 21:23 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Under testing with the EDIFACT schema (from DFDLSchemas on github) against new code in daffodil, I see that my proposal was not sufficient. Steve Hanson stated that IBM DFDL current behavior for required empty strings includes "An empty occurrence with no default gives a Processing Error." I misinterpreted this. I was thinking required occurrence of an array element (as in with index <= minOccurs). But this should not be interpreted that narrowly, but any required occurrence at all including scalar elements. The EDIFACT schema depends on this behavior and backtracking driven by it, in order to work. So my suggestion for new properties to control this is revised to: dfdl:emptyElementPolicy enum with values noEmptyElements - matches current IBM DFDL behavior where * required elements without default values that are empty (specifically which satisfy the empty syntax - defined below) always cause Processing Errors. ** If a default value is specified that is provided as the value instead. When a default value is specified, then implementations that don't support default values when parsing must issue a runtime SDE here, not a processing error. * optional elements which satisfy the empty syntax are not added to the infoset. Defaulting is never considered. emptyElements - matches current description in the DFDL spec where * required elements: if the string/hexBinary satisfies the empty syntax then required elements are created with an empty string or empty hexBinary as their value. If a default value is specified that is substituted as the value instead. When a default value is specified, then implementations that don't support default values when parsing must issue a SDE here, not a processing error. * optional elements: if the string/hexBinary satisfies the empty syntax, and emptyValueDelimiterPolicy is not 'none' then an empty string (or hexbinary) is added to the infoset. If emptyValueDelimiterPolicy is 'none', nothing is added to the infoset. The term "satisfy the empty syntax" means what is found in the data stream may require initiator and/or terminator depending on emptyValueDelimiterPolicy, but if that is 'none' then this is satisfied just by empty string (or no bytes for hexBinary). Having said the above, I believe we also have to consider nillable elements. There are two topics: 1) defaulting to nilled - For the case of a nillable element, where the data syntax does NOT match the nil representation, then in the above anywhere a default value is specified, and there is behavior associated with that, well if the element is nillable, and dfdl:useNilAsDefault='true' is specified, then the element is default valued to being nilled. When nillable and dfdl:useNilAsDefault='true' is specified, then implementations that don't support defaulting to nilled when parsing must issue an SDE here, not a processing error. That takes care of the defaulting aspect of nillables. The second topic is: 2) nillable, and dfdl:nilValue contains %ES; as one of the possible nil representations. Hence, there is the possibility of empty string (or empty hexBinary) matching the nil representation. I think the DFDL spec is clear here that if the data stream satisfies the nil syntax, then required or optional, you get a nilled element, period. Does IBM DFDL implement that behavior? If so great. If not I think we may have to amend the above description of noEmptyElements case for dfdl:emptyElementPolicy to specify the special cases. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Sun, Apr 28, 2019 at 9:36 AM Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: One clarification: is the IBM DFDL behavior the same for empty hexBinary elements as it is for text strings? I'm going to suggest we need a policy property e.g., dfdl:emptyElementPolicy which is an enum with at least these options: noOptionalEmptyElements - matches current IBM DFDL behavior optionalEmptyElementsWithSyntax - matches current description in the DFDL spec where initiator and/or terminator found triggers creation of an empty string value. (Daffodil implements this.) This would apply (I think) to both types xs:string ad xs:hexBinary I'm open to suggestions for better naming for the property and the property values, but these are the two settings we need I think. I do believe that the latter optionalEmptyElementsWithSyntax behavior is what the DFDL spec describes, and is most consistent given the available properties such as emptyValueDelimiterPolicy. We can make implementation of optionalEmptyElementsWithSyntax a DFDL optional language feature, thereby avoiding issues of conformance with the DFDL standard. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Apr 5, 2019 at 12:43 PM Steve Hanson <smh(a)uk.ibm.com> wrote: Daffodil to perform identical tests but the belief is that they implement the spec as published (except maybe for one bug with default values for strings). So there is a mis-match between Daffodil and IBM DFDL. It sounds like a new property is going to be needed which toggles the way that empty strings are handled. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 03/04/2019 12:04 Subject: Action 306 - IBM DFDL behaviour when parsing empty strings 306 Confirm IBM DFDL behaviour when parsing empty strings (Steve) 7/8: IBM DFDL has not fully implemented the behaviour changes arising from action 140 with respect to empty string elements. Daffodil is about to do so. IBM DFDL users have complained about lack of defaults when parsing but other than that appear happy. Are the rules in the spec for empty strings over complicated? Steve to document the behaviour for IBM DFDL to inform the discussion. ... 1/11: In progress - there are a lot of subtle scenarios 15/11: Not discussed ... 7/2/19: No further progress Some progress :) 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the Infoset. IBM DFDL behaviour: Required. IBM DFDL does not implement default values when parsing, so an empty occurrence with a default value gives an SDE (to prevent backtracking). An empty occurrence with no default gives a Processing Error. If you need to add an empty string to the infoset, you can add default=""(when default values implemented, of course). Optional. IBM DFDL adds nothing to the infoset regardless of presence of initiator and/or terminator. No way to get empty string into the infoset. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then an empty string (type xs:string), empty hexBinary (type xs:hexBinary), or default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) IBM DFDL behaviour: Required. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). Optional. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). So ... The spec today is consistent in one way, in that for both complex & string elements a) a required empty occurrence always adds to the infoset; & b) an optional empty occurrence adds to the infoset if initiator/terminator present; & c) an optional empty occurrence does not add to the infoset if no initiator/terminator present. If the simple string behaviour was to change to match IBM DFDL then that consistency is lost, but the string behaviour then matches that for other simple types. Section 9.4.2.2 disappears as the behaviour is same as 9.4.2.1. Section 9.4.2.3 becomes as below. We lose the ability to get an empty string into the infoset for an optional string with initiator/terminator. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then a default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) We also need to be sure that any other implementations have not yet implemented the current spec behaviour. Need to check with DFDL4S and IBM TPF. To be discussed on next WG call ... Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Tracker 299 and clarifications needed for dfdl:initiatedContent="yes" behaviors
by Mike Beckerle 20 Jun '19

20 Jun '19

Clarifications around initiatedContent="yes", choices, and array elements DFDL spec says in Section 15 that the root of a choice branch must not be optional, meaning minOccurs cannot be 0. This was questioned and resulted in tracker 299. The last comment there suggests a need to revisit. Here's the link: https://redmine.ogf.org/issues/299 The additional clarification I seek is whether I have properly understood how dfdl:initiatedContent works in scenarios involving a combination of choices with array elements. Consider this schema fragment:  <xs:choice dfdl:initiatedContent="yes"> .... <xs:element name="foo" minOccurs="5" dfdl:initiator="foo:" dfdl:occursCountKind="parsed".../> ... </xs:choice> When parsing this choice, if you encounter "foo:" in the data stream, that discriminates two things: the choice itself, and the first occurrence of the array foo. Both that choice branch (aka the array), and the first occurrence of the array are "known to exist" per section 9.3.1.1. Subsequent occurrences of additional foo elements with their initiators "foo:" discriminate only the array occurrences. The minOccurs="5" is ignored (except for validation)  <xs:choice dfdl:initiatedContent="yes"> .... <xs:element name="foo" minOccurs="5" dfdl:initiator="foo:" dfdl:occursCountKind="fixed".../> ... </xs:choice> When parsing this choice, if you encounter "foo:" in the data stream, that discriminates the choice, but only for the first occurrence of element foo. There is no point of uncertainty for the array. Subsequent occurrences of element foo do not discriminate anything. There must be exactly 5 occurrences of element foo with initiators.  <xs:choice dfdl:initiatedContent="yes"> .... <xs:element name="foo" minOccurs="5" dfdl:initiator="foo:" dfdl:occursCountKind="expression".../> ... </xs:choice> When parsing this choice, if you encounter "foo:" that discriminates the choice, but there are no points of uncertainty for the array. MinOccurs is ignored (except for validation). Subsequent occurrences of element "foo" do not discriminate anything.  <xs:choice dfdl:initiatedContent="yes"> .... <xs:element name="foo" minOccurs="5" dfdl:initiator="foo:" dfdl:occursCountKind="implicit".../> ... </xs:choice> When parsing this choice, if you encounter "foo:" that discriminates the choice, but there is no point of uncertainty for the array until 5 occurrences have been parsed. Once the occurs index is 6 then if you encounter "foo:" that discriminates the occurrence of the array element foo, but the choice is already decided. I raise these scenarios because they show that there is quite a bit of complexity associated with dfdl:initiatedContent if viewed as a mechanism for setting discriminators. Sometimes more than one discriminator must be set, other times only one, and sometimes it depends on the array index. So the question is: Is my understanding of the above scenarios correct? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

draft interoperability document
by Mike Beckerle 19 Jun '19

19 Jun '19

Completion of the OGF standards process for DFDL is gated by the need to show interoperability across two implementations. Today we got EDIFACT to work on Apache Daffodil (incubating). Of the schemas published on DFDLSchemas site, all of them (save VCard which we're still debugging/implementing) run on Daffodil now. I have created an initial draft of a document that describes the list of DFDL schemas that are portable/interoperable across IBM DFDL (from App Connect) implementation, and Apache Daffodil (incubating). It is here: https://redmine.ogf.org/dmsf_files/13589?download= Comments are welcome. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Cancelled: OGF DFDL Working Group call
by Steve Hanson 07 Jun '19

07 Jun '19

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 31 May '19

31 May '19

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0