- dfdl-wg - lists.ogf.org

Agenda for OGF DFDL WG Call
by Steve Hanson 31 May '19

31 May '19

Please find agenda for today's call on Redmine at https://redmine.ogf.org/dmsf_files/13588?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

News article published: Use of DFDL by IBM z/Transaction Processing Facility (z/TPF)
by Steve Hanson 31 May '19

31 May '19

https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings
by Steve Hanson 31 May '19

31 May '19

Re-reading this thread from the bottom, I'm not sure the proposal is correct. I may have over-simplified it. I'm going to have to do some more tests. Specifically around dfdl:nilValue="%ES;". IBM DFDL supports this, which means we can't just be treating empty elements as missing all the time. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/05/2019 08:40 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings It will hopefully be possible for you to apply the new 'treatAsMissing' enum to just 2 places in the Daffodil code: 1) Empty rep found for required occurrence -> processing error 2) Empty rep found for optional occurrence -> don't add anything to infoset Maybe 4 places if you have separate paths for simple v complex. That's an indicator of how conceptually simple this property is, once you know the difference between empty & missing. I am pretty sure the IBM DFDL behaviour deviation around empty/missing can be encapsulated by just this. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/05/2019 08:12 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings I prefer dfdl:emptyElementParsePolicy = ( "treatAsMissing" | "treatAsEmpty" ) You have to understand the difference between empty and missing in DFDL. It has an effect on all types - for example, if you set "treatAsMissing" for a required number, it means empty always causes a processing error instead of potentially applying a default. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 08/05/2019 19:26 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings I suggest we stick with the "...Policy" naming convention for new things that control modes of behavior. I'd prefer to avoid the terms empty and missing in the property values and go with something that is more explanatory of what difference it makes. E.g, emptyElementParsePolicy with values "excludeEmptyStringAndHexBinaryValues" and "allowEmptyStringAndHexBinaryValues" The doc for these values will of course have to be in terms of Absent/Missing/Empty, etc. but at least the names give some intuition as to what they control without having to understand all of DFDL's nuances about the difference between what Absent and Missing is. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, May 8, 2019 at 12:52 PM Steve Hanson <smh(a)uk.ibm.com> wrote: Maybe this is better; dfdl:parseEmptyAsMissing = yes | no Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 08/05/2019 16:48 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Interesting. Many DFDL schemas I've created have a simpleType defintion named "nzString" which is string, plus an assertion that it is non-empty. That's to achieve exactly the behavior you have in IBM DFDL, because, as you say, many formats want this. We could rename the suggested property emptyElementParsePolicy to make it clear it is only about parsing. I like treatAsMissing. Easy to say what it means. treatAsEmpty begs the question of what empty elements do, but that's already complicated in the spec due to optionals and EVDP, so I'm happy with this also. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, May 7, 2019 at 3:48 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Hi Mike I think what you have highlighted is that there are formats which require that empty elements should not be treated as empty but as missing, which is effectively what IBM DFDL is doing (our code was written prior to action 140 when there was no distinction between empty & missing). That could be achieved with assertions. So maybe we should view the new property as a convenience property for such formats, as well as handling IBM DFDL's behaviour? If so, then can I suggest new names for the enums, which I think makes the intent clearer? dfdl:emptyElementPolicy = ( "treatAsMissing" | "treatAsEmpty" ) This only applies when parsing, maybe names should reflect that also? Further, "treatAsMissing" would imply that a default value was never used when parsing, as they are only used when the representation is empty. I think we can do away with the SDE clause for "treatAsMissing". The clause is only needed for "treatAsEmpty". IBM DFDL does implement nillable processing, including use of ES as nil literal value. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/05/2019 21:23 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Under testing with the EDIFACT schema (from DFDLSchemas on github) against new code in daffodil, I see that my proposal was not sufficient. Steve Hanson stated that IBM DFDL current behavior for required empty strings includes "An empty occurrence with no default gives a Processing Error." I misinterpreted this. I was thinking required occurrence of an array element (as in with index <= minOccurs). But this should not be interpreted that narrowly, but any required occurrence at all including scalar elements. The EDIFACT schema depends on this behavior and backtracking driven by it, in order to work. So my suggestion for new properties to control this is revised to: dfdl:emptyElementPolicy enum with values noEmptyElements - matches current IBM DFDL behavior where * required elements without default values that are empty (specifically which satisfy the empty syntax - defined below) always cause Processing Errors. ** If a default value is specified that is provided as the value instead. When a default value is specified, then implementations that don't support default values when parsing must issue a runtime SDE here, not a processing error. * optional elements which satisfy the empty syntax are not added to the infoset. Defaulting is never considered. emptyElements - matches current description in the DFDL spec where * required elements: if the string/hexBinary satisfies the empty syntax then required elements are created with an empty string or empty hexBinary as their value. If a default value is specified that is substituted as the value instead. When a default value is specified, then implementations that don't support default values when parsing must issue a SDE here, not a processing error. * optional elements: if the string/hexBinary satisfies the empty syntax, and emptyValueDelimiterPolicy is not 'none' then an empty string (or hexbinary) is added to the infoset. If emptyValueDelimiterPolicy is 'none', nothing is added to the infoset. The term "satisfy the empty syntax" means what is found in the data stream may require initiator and/or terminator depending on emptyValueDelimiterPolicy, but if that is 'none' then this is satisfied just by empty string (or no bytes for hexBinary). Having said the above, I believe we also have to consider nillable elements. There are two topics: 1) defaulting to nilled - For the case of a nillable element, where the data syntax does NOT match the nil representation, then in the above anywhere a default value is specified, and there is behavior associated with that, well if the element is nillable, and dfdl:useNilAsDefault='true' is specified, then the element is default valued to being nilled. When nillable and dfdl:useNilAsDefault='true' is specified, then implementations that don't support defaulting to nilled when parsing must issue an SDE here, not a processing error. That takes care of the defaulting aspect of nillables. The second topic is: 2) nillable, and dfdl:nilValue contains %ES; as one of the possible nil representations. Hence, there is the possibility of empty string (or empty hexBinary) matching the nil representation. I think the DFDL spec is clear here that if the data stream satisfies the nil syntax, then required or optional, you get a nilled element, period. Does IBM DFDL implement that behavior? If so great. If not I think we may have to amend the above description of noEmptyElements case for dfdl:emptyElementPolicy to specify the special cases. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Sun, Apr 28, 2019 at 9:36 AM Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: One clarification: is the IBM DFDL behavior the same for empty hexBinary elements as it is for text strings? I'm going to suggest we need a policy property e.g., dfdl:emptyElementPolicy which is an enum with at least these options: noOptionalEmptyElements - matches current IBM DFDL behavior optionalEmptyElementsWithSyntax - matches current description in the DFDL spec where initiator and/or terminator found triggers creation of an empty string value. (Daffodil implements this.) This would apply (I think) to both types xs:string ad xs:hexBinary I'm open to suggestions for better naming for the property and the property values, but these are the two settings we need I think. I do believe that the latter optionalEmptyElementsWithSyntax behavior is what the DFDL spec describes, and is most consistent given the available properties such as emptyValueDelimiterPolicy. We can make implementation of optionalEmptyElementsWithSyntax a DFDL optional language feature, thereby avoiding issues of conformance with the DFDL standard. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Apr 5, 2019 at 12:43 PM Steve Hanson <smh(a)uk.ibm.com> wrote: Daffodil to perform identical tests but the belief is that they implement the spec as published (except maybe for one bug with default values for strings). So there is a mis-match between Daffodil and IBM DFDL. It sounds like a new property is going to be needed which toggles the way that empty strings are handled. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 03/04/2019 12:04 Subject: Action 306 - IBM DFDL behaviour when parsing empty strings 306 Confirm IBM DFDL behaviour when parsing empty strings (Steve) 7/8: IBM DFDL has not fully implemented the behaviour changes arising from action 140 with respect to empty string elements. Daffodil is about to do so. IBM DFDL users have complained about lack of defaults when parsing but other than that appear happy. Are the rules in the spec for empty strings over complicated? Steve to document the behaviour for IBM DFDL to inform the discussion. ... 1/11: In progress - there are a lot of subtle scenarios 15/11: Not discussed ... 7/2/19: No further progress Some progress :) 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the Infoset. IBM DFDL behaviour: Required. IBM DFDL does not implement default values when parsing, so an empty occurrence with a default value gives an SDE (to prevent backtracking). An empty occurrence with no default gives a Processing Error. If you need to add an empty string to the infoset, you can add default=""(when default values implemented, of course). Optional. IBM DFDL adds nothing to the infoset regardless of presence of initiator and/or terminator. No way to get empty string into the infoset. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then an empty string (type xs:string), empty hexBinary (type xs:hexBinary), or default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) IBM DFDL behaviour: Required. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). Optional. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). So ... The spec today is consistent in one way, in that for both complex & string elements a) a required empty occurrence always adds to the infoset; & b) an optional empty occurrence adds to the infoset if initiator/terminator present; & c) an optional empty occurrence does not add to the infoset if no initiator/terminator present. If the simple string behaviour was to change to match IBM DFDL then that consistency is lost, but the string behaviour then matches that for other simple types. Section 9.4.2.2 disappears as the behaviour is same as 9.4.2.1. Section 9.4.2.3 becomes as below. We lose the ability to get an empty string into the infoset for an optional string with initiator/terminator. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then a default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) We also need to be sure that any other implementations have not yet implemented the current spec behaviour. Need to check with DFDL4S and IBM TPF. To be discussed on next WG call ... Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fwd: TPF extension for pointers and offsets - Fw: Latest OGF DFDL WG Call Minutes
by Mike Beckerle 30 May '19

30 May '19

Just making sure this thread is archived in the dfdl-wg mail archives. Some of the messages I see were addressed directly to participants. ---------- Forwarded message --------- From: Beckerle, Mike <mbeckerle(a)tresys.com> Date: Thu, May 2, 2019 at 12:03 PM Subject: TPF extension for pointers and offsets - Fw: Latest OGF DFDL WG Call Minutes To: Cc: mbeckerle.dfdl(a)gmail.com <mbeckerle.dfdl(a)gmail.com> ------------------------------ *From:* Bradd Kadlecik <braddk(a)us.ibm.com> *Sent:* Thursday, May 2, 2019 9:58 AM *To:* Steve Hanson *Cc:* Beckerle, Mike; Michele Zundo *Subject:* Re: Latest OGF DFDL WG Call Minutes I have the following DFDL that I have ready on a z/TPF system for the call today that can do both unparse and parse. *(See attached file: tpf_service_responses.gen.dfdl.xsd)* It represents the following C structure: typedef struct tpf_service_responses { ushort version; ushort respNum; /* Number of response formats */ uint resv; csvc_resp *resp[5]; /* Array of response formats */ } __attribute__ ((packed)) csvc_resps; typedef struct tpf_service_response { int status; /* response status code */ int resv; csvc_format_elms *format; /* response format */ } __attribute__ ((packed)) csvc_resp; typedef struct tpf_service_format_elements { char *schema; char root[8]; } __attribute__ ((packed)) csvc_format_elms; Regards, *Bradd Kadlecik* z/TPF Development ------------------------------ *Phone:* 1-845-433-1573 * E-mail:* *braddk(a)us.ibm.com* <braddk(a)us.ibm.com> 2455 South Rd Poughkeepsie, NY 12601-5400 United States [image: Inactive hide details for Steve Hanson---04/05/2019 12:42:34 PM---Regards]Steve Hanson---04/05/2019 12:42:34 PM---Regards From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int> Date: 04/05/2019 12:42 PM Subject: Re: Latest OGF DFDL WG Call Minutes ------------------------------ *No* *Action * *309* *Create example scenarios to illustrate offset & pointer requirements (Bradd)* 5/4/19: Daffodil have a draft proposal for offset support, TPF have experimental implementation for pointer support. Need examples to show the requirement, especially unparsing. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday [image: Inactive hide details for Steve Hanson---05/04/2019 10:29:24---I can see a difference between offsets and pointers. If I follo]Steve Hanson---05/04/2019 10:29:24---I can see a difference between offsets and pointers. If I follow an offset and parse an element x t From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int> Date: 05/04/2019 10:29 Subject: Re: Latest OGF DFDL WG Call Minutes ------------------------------ I can see a difference between offsets and pointers. If I follow an offset and parse an element x then I won't automatically jump back to where I was - the next element y I parse will continue from location offset + length x unless I use offset again to jump back to the original location. If I follow a pointer and parse an element x, then the next element y I parse will continue from original location + length(pointer). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday [image: Inactive hide details for Steve Hanson---04/04/2019 09:42:36---This is a proposal from the Daffodil team for offset support, ne]Steve Hanson---04/04/2019 09:42:36---This is a proposal from the Daffodil team for offset support, needed for formats like TIFF and if we From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int> Date: 04/04/2019 09:42 Subject: Re: Latest OGF DFDL WG Call Minutes ------------------------------ This is a proposal from the Daffodil team for offset support, needed for formats like TIFF and if we ever want to be able to handle zip files. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74687382 I think this proposal can implement your requirement - the dfdl:offset property can be an expression that refers to another element (your pointer element), which can be hidden so as not to appear in the infoset. I think what you are proposing is a more convenient way of handling offsets that are defined dynamically in the data, as opposed to defined statically with fixed values (though as I said below I need unparsing explained). But I may be mis-understanding your use cases. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday [image: Inactive hide details for Bradd Kadlecik---03/04/2019 16:50:13---Ok, I'll work on putting a scenario together for various point]Bradd Kadlecik---03/04/2019 16:50:13---Ok, I'll work on putting a scenario together for various pointer setups (array, complex element, str From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int> Date: 03/04/2019 16:50 Subject: Re: Latest OGF DFDL WG Call Minutes ------------------------------ Ok, I'll work on putting a scenario together for various pointer setups (array, complex element, string) and show how it looks for both the JSON and the binary. I'm currently at a conference this week but will be returning late Thursday so expect to be available for the call Friday. I presented the TPF specific pointer implementation at the conference this week and there are some that will be trying to use it soon. Regards, *Bradd Kadlecik* z/TPF Development ------------------------------ *Phone:* 1-845-433-1573 * E-mail:* *braddk(a)us.ibm.com* <braddk(a)us.ibm.com> 2455 South Rd Poughkeepsie, NY 12601-5400 United States [image: Inactive hide details for Steve Hanson---04/03/2019 10:29:25 AM---Hi Bradd I have a few questions please ... inline below.]Steve Hanson---04/03/2019 10:29:25 AM---Hi Bradd I have a few questions please ... inline below. From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 04/03/2019 10:29 AM Subject: Re: Latest OGF DFDL WG Call Minutes ------------------------------ Hi Bradd I have a few questions please ... inline below. I'm still going to need a real worked example, starting with some actual data and its schema, what it appears like after paring in the infoset, and how unparsing lays it back out again. Are you able to make the rescheduled call this Friday? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday [image: Inactive hide details for Bradd Kadlecik---27/02/2019 22:22:48---Regarding proposal for offsets and pointers: The following are]Bradd Kadlecik---27/02/2019 22:22:48---Regarding proposal for offsets and pointers: The following are the properties to be defined: From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 27/02/2019 22:22 Subject: Re: Latest OGF DFDL WG Call Minutes ------------------------------ Regarding proposal for offsets and pointers: The following are the properties to be defined: SMH: I was expecting to see these properties only on dfdl:element, especially as you say '...the element contents... ? indirectKind Enum Valid values 'pointer', 'offset' (there is also a thought of objectId or refId for handling BSON but not at this time) Specifies the type of indirection used to access the element contents in the data stream. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I am missing the distinction between offset and pointer. Is one relative to current position and the other relative to start of bitstream? SMH: In earlier DFDL proposals for offset support, we had used the term to refer to a property to be used to establish position of the *current* element instead of assuming the current element followed straight after the previous one. It would allow sparse modelling of fixed structures. The offset could be relative to start of bitstream or some other point. I don't think that's what you mean when you say 'offset' so I will refer to your new concept as 'pointer'. SMH: Assuming that indirectKind is a normal DFDL property, it can be in scope. It would therefore need to have an enum 'None' which would be the default used in most schemas. indirectLength Non-negative Integer or DFDL expression Specifies the length of the indirection in units according to the indirectUnits property. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group indirectUnits Enum Valid values 'bytes','bits' Specifies the units to be used for reading or writing the indirection according to indirectLength. The default value is 'bytes'. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I think a better approach is to provide a property dfdl:indirectType, instead of indirectLength/indirectUnits, which refers to a simple type (not element) that carries its own lengthKind, length & lengthUnits properties. Similar idea to dfdl:prefixLengthType. That allows a lot of flexibility on how the pointer can appear. offsetBase non-empty string containing an absolute or relative XPath expression for the base element. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group The proposal would be to have the contents of the indirection be after the LeadingAlignment and before the TrailingAlignment. This would mean the aligment and skip factors apply to the indirection values in the data stream instead of the the contents of the indirection. SMH: Agree. This also then means in an array element, each element has its own indirection value (pointer or offset) and the alignment,skip factors then apply to each of these indirection values. SMH: Do you mean '...each occurrence...' ? It would be thought that the indirection values apply only to the data stream and not the infoset. During parse when the infoset is populated from the data stream, the indirection values are replaced by the contents. During unparse, the indirection values don't exist in the infoset and are created during the writing to/creation of the data stream. SMH: I agree that the indirection should be a purely physical thing, but I am not clear how the value is filled in when unparsing. Where does the value come from? outputValueCalc? Or maybe it's not needed when unparsing, and the data is always contiguous? For pointers, a null pointer creates the scenario of either nil representation or empty representation depending on whether or not nillable is defined as true. Unless default values (or 0 occurrence) are defined for all underlying content, then this is a processing error. During unparse, the only scenario in which a null pointer would be created is for a nil representation. SMH: This needs more thought. The nil & default properties apply to the *contents* of the indirection, not to the pointer. If you want to give a nil semantic to the pointer value itself, then that would require a new enum for dfdl:nilKind. I don't see why a pointer value 0 can't be treated like any other indirection value. A missing pointer is an error - it must be present - there is no way to control optionality because minOccurs/maxOccurs apply to the contents. (Alternatively, if you want the concepts of nil, default, occurs to apply to the indirect value, then dfdl:indirectType could point at an element instead of a simple type - but that seems way too over engineered). Examples: The following is the definition for the address of a null-terminated string in which the string address may be NULL as indicated by a nillable value of true: <xs:element name="myString" type="xs:string" dfdl:lengthKind="delimited" dfdl:encoding="UTF-8" dfdl:terminator="%NUL;" dfdl:indirectKind="pointer" dfdl:indirectLength="8" dfdl:indirectUnits="bytes" nillable="true" /> The following is the definition for an array of three 4 byte addresses of a complex element defined by ns0:myStruct: <xs:element name="myArray" type="ns0:myStruct" dfdl:lengthKind="implicit" dfdl:indirectKind="pointer" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" minOccurs="3" maxOccurs="3" dfdl:occursCountKind="fixed" /> The following is the definition for a 4 byte offset to a 100 byte hexBinary value from the start of the parent element definition: <xs:element name="myData" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="100" dfdl:lengthUnits="bytes" dfdl:indirectKind="offset" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" dfdl:offsetBase=".." /> SMH: I don't see how unparsing works. What provides the value? The proposal would also allow for the following optional item but I don't currently see a need for this: dfdl:offsetKind with values "startToStart" or "endToStart" - indicates if the offset is from the start of the base element or the end of the base element. I tried getting this out before my vacation so it might take a little bit to respond for issues. Thank you for your time. Regards, *Bradd Kadlecik* z/TPF Development ------------------------------ *Phone:* 1-845-433-1573 * E-mail:* *braddk(a)us.ibm.com* <braddk(a)us.ibm.com> 2455 South Rd Poughkeepsie, NY 12601-5400 United States [image: Inactive hide details for Steve Hanson---02/07/2019 12:32:26 PM---Please find minutes from the latest call at https://redmine.o]Steve Hanson---02/07/2019 12:32:26 PM---Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/newsRegards Ste From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 02/07/2019 12:32 PM Subject: Latest OGF DFDL WG Call Minutes ------------------------------ Please find minutes from the latest call at *https://redmine.ogf.org/projects/dfdl-wg/news* <https://redmine.ogf.org/projects/dfdl-wg/news> Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL Working Group call
by Steve Hanson 28 May '19

28 May '19

1 0

OGF DFDL WG Call on 2018-05-16 - cancelled
by Steve Hanson 15 May '19

15 May '19

Apologies but I have a deadline for this Friday. Should be ok for next call in 2 weeks time. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 15 May '19

15 May '19

1 0

Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings
by Steve Hanson 08 May '19

08 May '19

It will hopefully be possible for you to apply the new 'treatAsMissing' enum to just 2 places in the Daffodil code: 1) Empty rep found for required occurrence -> processing error 2) Empty rep found for optional occurrence -> don't add anything to infoset Maybe 4 places if you have separate paths for simple v complex. That's an indicator of how conceptually simple this property is, once you know the difference between empty & missing. I am pretty sure the IBM DFDL behaviour deviation around empty/missing can be encapsulated by just this. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/05/2019 08:12 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings I prefer dfdl:emptyElementParsePolicy = ( "treatAsMissing" | "treatAsEmpty" ) You have to understand the difference between empty and missing in DFDL. It has an effect on all types - for example, if you set "treatAsMissing" for a required number, it means empty always causes a processing error instead of potentially applying a default. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 08/05/2019 19:26 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings I suggest we stick with the "...Policy" naming convention for new things that control modes of behavior. I'd prefer to avoid the terms empty and missing in the property values and go with something that is more explanatory of what difference it makes. E.g, emptyElementParsePolicy with values "excludeEmptyStringAndHexBinaryValues" and "allowEmptyStringAndHexBinaryValues" The doc for these values will of course have to be in terms of Absent/Missing/Empty, etc. but at least the names give some intuition as to what they control without having to understand all of DFDL's nuances about the difference between what Absent and Missing is. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, May 8, 2019 at 12:52 PM Steve Hanson <smh(a)uk.ibm.com> wrote: Maybe this is better; dfdl:parseEmptyAsMissing = yes | no Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 08/05/2019 16:48 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Interesting. Many DFDL schemas I've created have a simpleType defintion named "nzString" which is string, plus an assertion that it is non-empty. That's to achieve exactly the behavior you have in IBM DFDL, because, as you say, many formats want this. We could rename the suggested property emptyElementParsePolicy to make it clear it is only about parsing. I like treatAsMissing. Easy to say what it means. treatAsEmpty begs the question of what empty elements do, but that's already complicated in the spec due to optionals and EVDP, so I'm happy with this also. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, May 7, 2019 at 3:48 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Hi Mike I think what you have highlighted is that there are formats which require that empty elements should not be treated as empty but as missing, which is effectively what IBM DFDL is doing (our code was written prior to action 140 when there was no distinction between empty & missing). That could be achieved with assertions. So maybe we should view the new property as a convenience property for such formats, as well as handling IBM DFDL's behaviour? If so, then can I suggest new names for the enums, which I think makes the intent clearer? dfdl:emptyElementPolicy = ( "treatAsMissing" | "treatAsEmpty" ) This only applies when parsing, maybe names should reflect that also? Further, "treatAsMissing" would imply that a default value was never used when parsing, as they are only used when the representation is empty. I think we can do away with the SDE clause for "treatAsMissing". The clause is only needed for "treatAsEmpty". IBM DFDL does implement nillable processing, including use of ES as nil literal value. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/05/2019 21:23 Subject: Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings Under testing with the EDIFACT schema (from DFDLSchemas on github) against new code in daffodil, I see that my proposal was not sufficient. Steve Hanson stated that IBM DFDL current behavior for required empty strings includes "An empty occurrence with no default gives a Processing Error." I misinterpreted this. I was thinking required occurrence of an array element (as in with index <= minOccurs). But this should not be interpreted that narrowly, but any required occurrence at all including scalar elements. The EDIFACT schema depends on this behavior and backtracking driven by it, in order to work. So my suggestion for new properties to control this is revised to: dfdl:emptyElementPolicy enum with values noEmptyElements - matches current IBM DFDL behavior where * required elements without default values that are empty (specifically which satisfy the empty syntax - defined below) always cause Processing Errors. ** If a default value is specified that is provided as the value instead. When a default value is specified, then implementations that don't support default values when parsing must issue a runtime SDE here, not a processing error. * optional elements which satisfy the empty syntax are not added to the infoset. Defaulting is never considered. emptyElements - matches current description in the DFDL spec where * required elements: if the string/hexBinary satisfies the empty syntax then required elements are created with an empty string or empty hexBinary as their value. If a default value is specified that is substituted as the value instead. When a default value is specified, then implementations that don't support default values when parsing must issue a SDE here, not a processing error. * optional elements: if the string/hexBinary satisfies the empty syntax, and emptyValueDelimiterPolicy is not 'none' then an empty string (or hexbinary) is added to the infoset. If emptyValueDelimiterPolicy is 'none', nothing is added to the infoset. The term "satisfy the empty syntax" means what is found in the data stream may require initiator and/or terminator depending on emptyValueDelimiterPolicy, but if that is 'none' then this is satisfied just by empty string (or no bytes for hexBinary). Having said the above, I believe we also have to consider nillable elements. There are two topics: 1) defaulting to nilled - For the case of a nillable element, where the data syntax does NOT match the nil representation, then in the above anywhere a default value is specified, and there is behavior associated with that, well if the element is nillable, and dfdl:useNilAsDefault='true' is specified, then the element is default valued to being nilled. When nillable and dfdl:useNilAsDefault='true' is specified, then implementations that don't support defaulting to nilled when parsing must issue an SDE here, not a processing error. That takes care of the defaulting aspect of nillables. The second topic is: 2) nillable, and dfdl:nilValue contains %ES; as one of the possible nil representations. Hence, there is the possibility of empty string (or empty hexBinary) matching the nil representation. I think the DFDL spec is clear here that if the data stream satisfies the nil syntax, then required or optional, you get a nilled element, period. Does IBM DFDL implement that behavior? If so great. If not I think we may have to amend the above description of noEmptyElements case for dfdl:emptyElementPolicy to specify the special cases. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Sun, Apr 28, 2019 at 9:36 AM Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: One clarification: is the IBM DFDL behavior the same for empty hexBinary elements as it is for text strings? I'm going to suggest we need a policy property e.g., dfdl:emptyElementPolicy which is an enum with at least these options: noOptionalEmptyElements - matches current IBM DFDL behavior optionalEmptyElementsWithSyntax - matches current description in the DFDL spec where initiator and/or terminator found triggers creation of an empty string value. (Daffodil implements this.) This would apply (I think) to both types xs:string ad xs:hexBinary I'm open to suggestions for better naming for the property and the property values, but these are the two settings we need I think. I do believe that the latter optionalEmptyElementsWithSyntax behavior is what the DFDL spec describes, and is most consistent given the available properties such as emptyValueDelimiterPolicy. We can make implementation of optionalEmptyElementsWithSyntax a DFDL optional language feature, thereby avoiding issues of conformance with the DFDL standard. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Apr 5, 2019 at 12:43 PM Steve Hanson <smh(a)uk.ibm.com> wrote: Daffodil to perform identical tests but the belief is that they implement the spec as published (except maybe for one bug with default values for strings). So there is a mis-match between Daffodil and IBM DFDL. It sounds like a new property is going to be needed which toggles the way that empty strings are handled. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" < michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 03/04/2019 12:04 Subject: Action 306 - IBM DFDL behaviour when parsing empty strings 306 Confirm IBM DFDL behaviour when parsing empty strings (Steve) 7/8: IBM DFDL has not fully implemented the behaviour changes arising from action 140 with respect to empty string elements. Daffodil is about to do so. IBM DFDL users have complained about lack of defaults when parsing but other than that appear happy. Are the rules in the spec for empty strings over complicated? Steve to document the behaviour for IBM DFDL to inform the discussion. ... 1/11: In progress - there are a lot of subtle scenarios 15/11: Not discussed ... 7/2/19: No further progress Some progress :) 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the Infoset. IBM DFDL behaviour: Required. IBM DFDL does not implement default values when parsing, so an empty occurrence with a default value gives an SDE (to prevent backtracking). An empty occurrence with no default gives a Processing Error. If you need to add an empty string to the infoset, you can add default=""(when default values implemented, of course). Optional. IBM DFDL adds nothing to the infoset regardless of presence of initiator and/or terminator. No way to get empty string into the infoset. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then an empty string (type xs:string), empty hexBinary (type xs:hexBinary), or default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) IBM DFDL behaviour: Required. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). Optional. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). So ... The spec today is consistent in one way, in that for both complex & string elements a) a required empty occurrence always adds to the infoset; & b) an optional empty occurrence adds to the infoset if initiator/terminator present; & c) an optional empty occurrence does not add to the infoset if no initiator/terminator present. If the simple string behaviour was to change to match IBM DFDL then that consistency is lost, but the string behaviour then matches that for other simple types. Section 9.4.2.2 disappears as the behaviour is same as 9.4.2.1. Section 9.4.2.3 becomes as below. We lose the ability to get an empty string into the infoset for an optional string with initiator/terminator. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then a default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) We also need to be sure that any other implementations have not yet implemented the current spec behaviour. Need to check with DFDL4S and IBM TPF. To be discussed on next WG call ... Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings
by Steve Hanson 08 May '19

08 May '19

Daffodil to perform identical tests but the belief is that they implement the spec as published (except maybe for one bug with default values for strings). So there is a mis-match between Daffodil and IBM DFDL. It sounds like a new property is going to be needed which toggles the way that empty strings are handled. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 03/04/2019 12:04 Subject: Action 306 - IBM DFDL behaviour when parsing empty strings 306 Confirm IBM DFDL behaviour when parsing empty strings (Steve) 7/8: IBM DFDL has not fully implemented the behaviour changes arising from action 140 with respect to empty string elements. Daffodil is about to do so. IBM DFDL users have complained about lack of defaults when parsing but other than that appear happy. Are the rules in the spec for empty strings over complicated? Steve to document the behaviour for IBM DFDL to inform the discussion. ... 1/11: In progress - there are a lot of subtle scenarios 15/11: Not discussed ... 7/2/19: No further progress Some progress :) 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the Infoset. IBM DFDL behaviour: Required. IBM DFDL does not implement default values when parsing, so an empty occurrence with a default value gives an SDE (to prevent backtracking). An empty occurrence with no default gives a Processing Error. If you need to add an empty string to the infoset, you can add default=""(when default values implemented, of course). Optional. IBM DFDL adds nothing to the infoset regardless of presence of initiator and/or terminator. No way to get empty string into the infoset. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then an empty string (type xs:string), empty hexBinary (type xs:hexBinary), or default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) IBM DFDL behaviour: Required. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). Optional. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). So ... The spec today is consistent in one way, in that for both complex & string elements a) a required empty occurrence always adds to the infoset; & b) an optional empty occurrence adds to the infoset if initiator/terminator present; & c) an optional empty occurrence does not add to the infoset if no initiator/terminator present. If the simple string behaviour was to change to match IBM DFDL then that consistency is lost, but the string behaviour then matches that for other simple types. Section 9.4.2.2 disappears as the behaviour is same as 9.4.2.1. Section 9.4.2.3 becomes as below. We lose the ability to get an empty string into the infoset for an optional string with initiator/terminator. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then a default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) We also need to be sure that any other implementations have not yet implemented the current spec behaviour. Need to check with DFDL4S and IBM TPF. To be discussed on next WG call ... Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 7

Latest OGF DFDL WG Call Minutes
by Steve Hanson 02 May '19

02 May '19

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0