April 2019 - dfdl-wg - lists.ogf.org

clarification needed: unparsing and hidden group choices and choices with dfdl:choiceDispatchKey
by Mike Beckerle 13 Dec '19

13 Dec '19

So interesting issue. Comes up as part of our experience implementing the advanced expression language features: hidden groups, and computed elements. We have hidden groups in DFDL. If a xs:choice is inside a hidden group, then by definition, when unparsing, elements corresponding to some branch of the choice are not in the Infoset. DFDL spec clearly states that dfdl:outputValueCalc is not allowed on the head of a choice branch. But elements inside hidden groups must have dfdl:outputValueCalc, or a default value, otherwise there's no way for them to be filled in as part of the augmented infoset. So the only way a choice can be inside a hidden group is if the head of the choice branches have default values. This is not sufficient. I do not think it was our intention to make hidden groups incapable of holding things complex enough to contain a choice. Rather, the point is to be able to hide things that have excess complexity needed only for parsing/unparsing, so that they need not appear in the final Infoset. So if a choice is allowed inside a hidden group, then a mechanism is needed by which when unparsing a choice branch is selected based on values of other parts of the Infoset, and then the elements of that branch are computed and added to the augmented infoset. Note that without hidden groups, this problem never arises, as the Infoset always has to take a position on choice branches by having an unambiguous element in it. There are 3 cases: 1. unparsing an xs:choice where for parsing, each branch has a discriminator, or is resolved by the simple "first successful parse" rule. 2. unparsing an xs:choice with dfdl:choiceDispatchKey 3. unparsing xs:choice with dfdl:initiatedContent='yes' For all cases, I suggest we must remove the restriction on dfdl:outputValueCalc being disallowed on choice branch head. For (1). First branch where the branch can be constructed successfully is added to the augmented infoset. This is a case where an unparse error is non-fatal, and causes a form of unparser backtracking through alternatives. Branches would be attempted in schema order. For (2) I suggest dfdl:unparseChoiceDispatchKey, evaluated at unparse time only. Produces a choice branch key. That choice branch is then constructed, and default values or dfdl:outputValueCalcs are used to fill in values. (BTW: this is the driving use case we have in Daffodil for this issue.) For (3).... I have no specific solution. I think it may not actually be a separate case and has to be treated as (1) or (2). Comments/Discussion?

2 1

Consolidated Notes including both DFDL WG Call 2018-08-07 along with other recent clarification email threads
by Mike Beckerle 03 Jul '19

03 Jul '19

This message includes two parts. Part 1 is the things we discussed on the DFDL WG Call of 2018-08-07. Part 2 is the other recent emails where the conclusion from the email thread is repeated here for finalization/refinement, and for consolidation so all these changes can be considered together. ================================ Part 1 - Discussed on the call. Re: [DFDL-WG] Clarification discussion points for next call(s) - DFDL Spec issues around nil, empty, normal, absent, defaulting, and separator suppression. Dated August 2nd Conclusions: Section 9.3.1.1 Delete phrase "...this of course implies that....". Note that there is already a correction to create numbered bullets of 3 sentences. The sentence containing this phrase will be bullet #3 of that list. Section 9.4 Item 2 under "For elements and element refs:" Change to: "dfdl:element following property scoping rules, which includes establishing representation as described in Section 9.3.2 and conversion to element type for simple types." Section 9.3.2 The phrase "The first step is to see if the content is trivaill of length zero." Change to: "The first step is to see if the SimpleContent or ComplexContnet region is of length zero as a first approximation." The bullet "delimited => length is zero (delimiter is immediately encountered)" Insert "in scope" after the open parenthesis. Section 9.4.2.3. We agreed that the paragraphs beginning with "For both required and optional..." need to be better tied to the material above. Wording TBD - pending Steve Hanson doing some tests on IBM DFDL. ============================ Part 2 - Below are conclusions from the prior email threads. This is for email review in lieu of discussion on this week's call. Re: [DFDL-WG] clarification: on suppressed ZL string/hexBinary - do we keep variable assignments?Of Aug 6 (last date in the thread) These corrections apply: Sections 9.4.2.2 and 9.4.2.3 The phrase "Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none'[12] <http://daffodil.apache.org/docs/dfdl/#_ftn12>," Change to "Optional occurrence: if dfdl:emptyValueDelimiterPolicy is applicable and is not 'none',...." (retaining the footnote) Section 9.4.2 Before the final phrase "There are three main cases to consider:" Insert this sentence: "The sections below indicate when an item is added to the infoset, and whether it has a default or other value. If there is no processing error then regardless of whether an item is added to the infoset or not, any side-effects due to dfdl:discriminator statements evaluating to true, or dfdl:setVariable statements, are retained." Section 12.2 For property emptyValueDelimiterPolicy, before the phrase "It is a schema definition error if...", insert this sentence: "The value of dfdl:emptyValueDelimiterPolicy should only be checked if there is a dfdl:initiator or dfdl:terminator in scope. If so, and dfdl:emptyValueDelimiterPolicy is not set, it is a schema definition error. If dfdl:initiator is not "" and dfdl:terminator is "" and dfdl:emptyValueDelimiterPolicy is 'terminator' it is a schema definition error. If dfdl:terminator is not "" and dfdl:initiator is " and dfdl:emptyValueDelimiterPolicy is 'initiator' it is a schema definition error." Section 13.16 For property nilValueDelimiterPolicy, before the phrase "It is a schema definition error if...", insert this sentence: "The value of dfdl:nilValueDelimiterPolicy should only be checked if there is a dfdl:initiator or dfdl:terminator in scope. If so, and dfdl:nilValueDelimiterPolicy is not set, it is a schema definition error. If dfdl:initiator is not "" and dfdl:terminator is "" and dfdl:nilValueDelimiterPolicy is 'terminator' it is a schema definition error. If dfdl:terminator is not "" and dfdl:initiator is " and dfdl:nilValueDelimiterPolicy is 'initiator' it is a schema definition error ." Section 9.2.2 The phrase "the occurrence's content in the data..." replace with "the occurrence's SimpleContent or ComplexContent region in the data..." The sentence: "The *empty representation* is special in DFDL, because when parsing it is this condition that can trigger the creation of a default value for an element occurrence." replace with: "The empty representation is special in DFDL because when parsing it it is used to determine when default values are created in the Infoset, and when optional recurring elements are omitted from the Infoset. The empty representation can require initiators or terminators be present so as to enable data formats to explicitly distinguish empty-string/hexBinary values (which might cause default values to be used) from emptiness meaning the absence of any representation." (This is to clarify an error of omission - prior language suggested that EVDP is only relevant when the element has a default value, because only that need was mentioned.) Re: [DFDL-WG] Clarification needed: separator for empty sequence Of Aug 2 Section 14.2 For property dfdl:separator. The sentence: "Separators occur in the data either before, between or after all occurrences of the elements or groups that are the children of the sequence." replaced with "Separators occur in the data either before, between or after all occurrences of *represented* elements (that is, elements without the dfdl:inputValueCalc property) or model groups that are the children of the sequence. Elements with dfdl:inputValueCalc have no representation in the data stream, and so never have separators. Children of a sequence that are model groups are always separated, even if they are empty (meaning have no children of their own - which is allowed for sequence groups), or both the model group child and its contained children occupy zero-length in the data stream." (note: Some of the above is redundant with stipulations in the dfdl:inputValueCalc property description, but I believe it is wise to have this little redundancy.) ====================== These email threads are mentioned here to indicate that they are resolved by one or another of the above corrections: Re: [DFDL-WG] clarification needed - ambiguity about empty string and optional element Of Aug 2 Re: [DFDL-WG] Spec correction ? - Section 9.3.2.1 - second list missing "empty" representationOf Aug 2 ----------- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 7

Re: [DFDL-WG] Action 306 - IBM DFDL behaviour when parsing empty strings
by Steve Hanson 09 May '19

09 May '19

Daffodil to perform identical tests but the belief is that they implement the spec as published (except maybe for one bug with default values for strings). So there is a mis-match between Daffodil and IBM DFDL. It sounds like a new property is going to be needed which toggles the way that empty strings are handled. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 03/04/2019 12:04 Subject: Action 306 - IBM DFDL behaviour when parsing empty strings 306 Confirm IBM DFDL behaviour when parsing empty strings (Steve) 7/8: IBM DFDL has not fully implemented the behaviour changes arising from action 140 with respect to empty string elements. Daffodil is about to do so. IBM DFDL users have complained about lack of defaults when parsing but other than that appear happy. Are the rules in the spec for empty strings over complicated? Steve to document the behaviour for IBM DFDL to inform the discussion. ... 1/11: In progress - there are a lot of subtle scenarios 15/11: Not discussed ... 7/2/19: No further progress Some progress :) 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the Infoset. IBM DFDL behaviour: Required. IBM DFDL does not implement default values when parsing, so an empty occurrence with a default value gives an SDE (to prevent backtracking). An empty occurrence with no default gives a Processing Error. If you need to add an empty string to the infoset, you can add default=""(when default values implemented, of course). Optional. IBM DFDL adds nothing to the infoset regardless of presence of initiator and/or terminator. No way to get empty string into the infoset. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then an empty string (type xs:string), empty hexBinary (type xs:hexBinary), or default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) IBM DFDL behaviour: Required. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). Optional. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). So ... The spec today is consistent in one way, in that for both complex & string elements a) a required empty occurrence always adds to the infoset; & b) an optional empty occurrence adds to the infoset if initiator/terminator present; & c) an optional empty occurrence does not add to the infoset if no initiator/terminator present. If the simple string behaviour was to change to match IBM DFDL then that consistency is lost, but the string behaviour then matches that for other simple types. Section 9.4.2.2 disappears as the behaviour is same as 9.4.2.1. Section 9.4.2.3 becomes as below. We lose the ability to get an empty string into the infoset for an optional string with initiator/terminator. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then a default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) We also need to be sure that any other implementations have not yet implemented the current spec behaviour. Need to check with DFDL4S and IBM TPF. To be discussed on next WG call ... Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 7

Minor clarification needed: kind of error if dfdl:choiceDispatchKey evaluates to an empty string
by Mike Beckerle 30 Apr '19

30 Apr '19

The spec for property dfdl:choiceDispatchKey says "The expression must evaluate to an xs:string which must not be the empty string." It fails to say whether this "must not" is a processing error or schema-definition error. Consistent with delimiter expressions evaluating to empty string, I believe this should be an SDE. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes
by Mike Beckerle 28 Apr '19

28 Apr '19

So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it. I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features. The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as http://www.xml.com/pub/a/2004/07/21/design.html These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself. The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike, > > Your scheme as proposed has a couple of problems. > a) It is possible to create a schema that does not use dfdl:format at all. > Unlikely but possible. With such a schema there is no way to indicate > version. > b) What are the version implications for annotations that have nothing to > do with dfdl:format? Asserts, discriminators, variables. > > Most of the language you use refers to versions of *schemas*. That's not > the same as version of a dfdl:format. Any reason you didn't just change the > DFDL namespace URL to be *http://www.ogf.org/dfdl/dfdl-2.0/* > <http://www.ogf.org/dfdl/dfdl-2.0/>, which must be declared before any > DFDL feature can be used? I think that was always the intent of the DFDL > namespace. > > The key piece of work is validating any proposal against the scoping rules > in section 8 of the spec. > > Regards > > Steve Hanson > > IBM Hybrid Integration, Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 <+44%201962%20815848> > mob:+44-7717-378890 <+44%207717%20378890> > Note: I work Tuesday to Friday > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org > Date: 23/01/2018 16:03 > Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 > features/changes > Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> > ------------------------------ > > > > > There are some features now under discussion which will most likely not be > part of the DFDL v1.0 standard, but a subsequent version 2.0 of the > standard. > > Implementations may have these features prior to the existence of that > draft version of the DFDL standard; hence, a mechanism for indicating the > support, and requirements that implies, on DFDL schemas is needed in order > for these v2.0 features to be introduced. > > The purpose of the versioning mechanism is to provide access to DFDL v2.0 > features, while maintaining also backward compatibility with DFDL v1.0 > schemas. > > For example, new DFDL format properties may be added to the DFDL v2.0 > version; however, those properties are not required to be specified unless > the version of the schema is specifically set to indicate it is using > version 2.0 features. Also, deprecated features should only generate > deprecation warnings if the user is requesting version 2.0 of DFDL, and > should be silent otherwise. > > Proposal: > > New attribute on the dfdl:format annotation: DFDLVersion with valid values > "1", "2". All values are reserved for future use. > > The DFDLVersion, if unspecified, defaults to "1" which means the > implementation is of the DFDL v1.0 specification. > > If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL > may be enforced on the schema, and schema definition errors will result if > version 2.0 requirements are not met. DFDL Version 1.0 features that are > deprecated will generate warnings. > > Example: > > <dfdl:format DFDLVersion="2" .../> > > Example: > > <dfdl:defineFormat name="myV2Format"> > <dfdl:format DFDLVersion="2" .../> > </dfdl:defineFormat> > > The DFDLVersion is not a format property and is not scoped. It can be > added to a named format definition which is then referenced in the usual > manner for named format definitions. > > Backward Compatibility Requirement > > DFDL Implementations that provide version 2.0 functionality must also > provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends > version 1.0. > > Mixed Version Schemas > > If a schema consists of multiple schema documents, and their respective > dfdl:format annotations do not all contain the same value for the > DFDLVersion attribute, then the schema is said to be "mixed version". This > is allowed. > > A named dfdl:format carrying DFDLVersion "2" may not be referenced from a > DFDL Schema document having DFDLVersion "1". > > The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or > lacking the DFDLVersion attribute may be referenced from a DFDL Schema > having DFDLVersion="2". > > There are really 3 ways that DFDL version 2.0 can differ from version 1.0, > and these are that new features are added, old features are deprecated, or > old features are removed. This latter is quite undesirable, but will be > considered a possibility for purpose of discussion this versioning system. > > These will commonly be in combination such as if an old feature is > deprecated with the replacement feature being added. > > These will now be discussed separately, with the emphasis on how mixed > version schemas must work. > > New Features Added > > Each new required DFDL property introduced for version 2.0 will be defined > along with what is called its "version 1.0 default value". When a mixed > version schema exists, the DFDL v1.0 parts of it behave as if any required > DFDLVersion 2 properties have their version 1.0 default value defined, at > top level of the schema (as if defined with that default value in each > DFDLVersion 1 schema document's dfdl:format annotation). > > Features Removed > > Incompatibilities are to be avoided wherever possible, and may not be > needed at all. However, there is still the possibility that some feature > will want to be entirely removed and replaced by something better, and in > that case the versioning scheme needs to be able to support this. > > In that situation, the schema definition error for version 2 is only > generated when a schema has DFDLVersion 2 specified (in dfdl:format > explicitly, or by reference to a named format carrying or referencing > DFDLVersion 2). Mixed schemas that contain schema documents entirely using > version 1 of DFDL do not generate this schema definition error. > > Features deprecated > > Deprecation warnings are generated only when a DFDLVersion="2" attribute > is present or is incorporated by reference (to a named format). A version 1 > schema that is incorporated with a version 2 DFDL schema but is not > modified in any way, does not generate deprecation warnings about version > 1.0 features that have been deprecated in version 2.0. > > (Most likely deprecations would be revising property names or value enums > for clarity.) > > Implementations may provide mechanisms for suppressing warnings. These are > implementation dependent. > > Version 1.0 Existing Deprecations and Compatibility > > During the process of standardization for DFDL v1.0 a number of properties > names were changed. DFDL version 1.0 implementations accept both the old > and new property names. > (Example: dfdl:separatorSuppressionPolicy, which replaced an older > property name.) DFDL schemas that specify DFDLVersion="2" will not accept > the older property names, and it is a schema definition error if the old > (pre DFDL v1.0) property names are used. > > Properly constructed mixed version schemas can still depend on the older > property names in the version 1.0 parts of the schema. > > ------------------------------ > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > *www.tresys.com* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ…> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the *OGF Intellectual Property Policy* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5…> > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=lDIa8xvu2n-e8 > PXWCJmajfYaXESMqvvOWpg1HMWeuGQ&s=hdO_VQ_p-1f7JfZPcSTPR1S_Y1a > nSBXJWarqPdJZdPc&e= > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >

2 6

Latest OGF DFDL WG Call Minutes
by Steve Hanson 05 Apr '19

05 Apr '19

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 05 Apr '19

05 Apr '19

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 05 Apr '19

05 Apr '19

1 0

Re: [DFDL-WG] Byte-order-marks - was: Re: Part 1 - Re: Action 307 - Demonstrate implementation interoperability
by Steve Hanson 05 Apr '19

05 Apr '19

Agreed that BOM support would be dropped from DFDL 1.0 via erratum. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/04/2019 10:44 Subject: Re: Byte-order-marks - was: Re: Part 1 - Re: [DFDL-WG] Action 307 - Demonstrate implementation interoperability Hi Mike Found the thread on BOMs, so ignore my earlier email. I am leaning towards deprecation on the following grounds: - Only one customer that I know of ever requested BOM processing for non-XML data (in 2010, for MRM, before IBM DFDL available) - BOM processing only applies to the message as a whole, not to any embedded Unicode fragments, so support is selective anyway - It is possible to model an optional BOM and use it to set a user-defined encoding variable which is then used by the rest of the schema I have a schema that models UTF16 BOM and it successfully parses and unparses the 3 variants fine (no BOM present, BOM for BE present, BOM for LE present). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 16/10/2018 14:57 Subject: Byte-order-marks - was: Re: Part 1 - Re: [DFDL-WG] Action 307 - Demonstrate implementation interoperability To avoid changing behavior then, we will probably need a property to turn on/off the BOM behavior which strips/generates BOM. All implementations of DFDL that implement text (IBM, and Daffodil) currently do not treat BOMs specially currently, neither stripping them nor generating them. I suggest we need byteOrderMarkPolicy="use/ignore", with "ignore" meaning that the BOM is just treated as a character. Implementation of byteOrderMarkPolicy="use" would be an optional feature of DFDL. That way both IBM DFDL and Daffodil can be compliant without implementing this. (I'd like everything we've collectively been able to live without thus far, that isn't needed for interoperability testing, to ultimately get onto the optional features list.) Or we can simply deprecate the functionality in the spec and say BOMs must be modeled, and just strike the stuff from the next draft, and post an example on how to model BOMs. It sounds heavy handed, but nobody has asked for this feature (on the Daffodil project), and it was put into the DFDL spec way back in the early days when we expected BOMs to be popular, but they never caught on. However, I'll acknowledge that IBM would be in a better position to decide whether this feature is needed, given more users in Asia and other places where UTF-16 may be more popular. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Oct 10, 2018 at 5:24 AM Steve Hanson <smh(a)uk.ibm.com> wrote: I think the main thing with BOMs is that when support is added by an implementation, it should not break existing behaviour for a document that starts with a BOM. So if a user had a schema that explicitly modelled the BOM, or was treating BOM as a character so it appeared in the infoset, then a BOM aware implementation should not suddenly change that. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/10/2018 14:35 Subject: Re: Part 1 - Re: [DFDL-WG] Action 307 - Demonstrate implementation interoperability Very helpful Steve H., , thanks. re: UTF-8 and BOM, for UTF-8, the BOM can be viewed as "just a character", same as it is in UTF-16BE and UTF-16LE. Only utf-16 unadorned has to actually look at, and in theory strip the BOM if found. Nobody is implementing this, and it's not clear it matters much. Today I know that Daffodil just treats UTF-16 as meaning UTF-16BE. Hence, I suggest we consider just making BOM processing optional in DFDL and also make utf-16 (unadorned) optional - takes one small issue off of being "standard compliant". This leaves the question of what does "utf-16" unadorned do, and the answer I think is supposed to be guided by BOM, but if that is unimplemented then the behavior is "implementation defined" i.e., non-portable. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Oct 9, 2018 at 6:25 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Mike, responses in-line below. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/10/2018 23:00 Subject: Part 1 - Re: [DFDL-WG] Action 307 - Demonstrate implementation interoperability I'm going to reply to this in a few parts. With respect to: - dfdl:binaryBooleanTrueRep with value empty string - dfdl:assert on global element and simple type - dfdl:discriminator on global element and simple type - Multiple xs:appinfo elements within each xs:annotation element I think these are minor non-compliances with the DFDL spec, and for interoperability testing we can just revise schemas under test to not use these constructs. SMH: Agree. With respect to: - When parsing, the distinction between an element being 'missing', having an 'empty representation' and having an 'absent representation', is not in accordance with the specification. I think time will tell here, that is, there's nothing we can anticipate having to do because of this as yet. If this non-compliance does not cause interoperability problems for realistic and published DFDL schemas then I wouldn't worry about it. Like IBM DFDL, Daffodil does not implement default values during parsing, and that's a likely area where this issue of missing/empty/absent has effect on behavior. It is quite possible that despite this lack of conformance to the DFDL spec., interoperability testing would be successful. SMH: IBM DFDL gives a runtime SDE when parsing if it a zero-length representation is found for an occurrence AND the element has a default value That prevents a behaviour change when support for default values when parsing is implemented. Suggest Daffodil does same if it does not do so already. With that in place, I think we are ok. With respect to: - When encoding is 'UTF-8' or 'UTF-16', byte order marks are not processed Daffodil also does not implement byte-order-mark processing. We can dodge this issue entirely if we make the UTF-16 charset (specifically UTF-16 without the BE or LE suffix) encoding an optional DFDL feature. That effectively makes byte-order-mark processing also an optional feature, and then both IBM DFDL and Daffodil would be compliant and interoperable. SMH: UTF8 can also have a BOM so that does not solve the problem entirely. Needs some more thought. With respect to: - dfdl:encodingErrorPolicy "replace" This one is harder. Daffodil doesn't implement encodingErrorPolicy='error' so we have no common ground here for interoperability testing. Making the entire encodingErrorPolicy property optional - meaning behavior in the presence of encoding errors is implementation specified - that's super undesirable to me. I suspect that implementing encodingErrorPolicy 'error' will be necessary for Daffodil. If we do that then IBM DFDL can continue to document the lack of this missing required feature of DFDL, or we can make 'replace' optional in the spec., or IBM could implement 'replace'. SMH: This is top of the list of missing features for IBM DFDL. I have asked in the past if this could be added as it's technically a regression when compared to IIB's older text/binary parser (MRM). I will ask again. Additional Non-portable/Problematic Required Features I did an analysis of all DFDL properties, and those that must be implemented to meet the minimum functionality that is not optional for a DFDL implementation per Section 21 of the spec. Starting from a list of all DFDL properties, I eliminated any specific to unparsing, and then any that aren't relevant given something optional in Section 21. Here are the remaining properties I found. Restrictions on what values of these properties are mentioned where their full functionality is considered optional: length - integer values only lengthKind - explicit, implicit only lengthUnits - bytes or characters only representation - binary only byteOrder alignment - number or 'implicit' alignmentUnits - bytes only fillByte leadingSkip trailingSkip encoding - 'UTF-8'', 'UTF-16', 'UTF-16BE', 'UTF-16LE', 'ASCII', and 'ISO-8859-1' encodingErrorPolicy - (Already discussed above, so not further discussed in this section) utf16Width - because UTF-16 is allowed for encoding, 'variable' is problematic. textPadKind textTrimKind textStringJustification textStringPadCharacter binaryNumberRep - binary only binaryFloatRep - ieee only binaryBooleanTrueRep binaryBooleanFalseRep - IBM DFDL doesn't allow empty string for this. (Minor.) binaryCalendarRep - binarySeconds, binaryMillseconds only binaryCalendarEpoch occursCountKind - fixed only occursCount - integer only Looking at this list, there is only 1 additional issue to portability/interoperability this raises today given what I know about the Daffodil implementation and the IBM implementation. Issue: utf16Width='variable' This issue can be addressed with a minor change to the DFDL specification. When the type is xs:string, lengthUnits is 'characters', then the length in characters should take surrogate-pairs found in the UTF-16 data, and count those as occupying 1 character. This utf16Width='variable' feature of DFDL should be optional, as Java JVM-based implementations will find this extremely difficult to support, since JVM standard string representations cannot represent individual characters with code points greater than 0xFFFF occupying 1 location in a string. Daffodil does not implement this 'variable' behavior, and we have no good pathway to do so. Hence, prefer to change the DFDL spec to make this 'variable' optional. Only 'fixed' would be required. I could support deprecating the whole property even. SMH: This is already captured by action 290, which is waiting for me to do some tests with IBM DFDL which claims to have implemented this. Issue: lengthUnits='characters' and variable-width charset encodings I believe this is required behavior. I also believe the lack of support for this is missing from IBM's list of non-compliances. I recall discussion that IBM DFDL requires a fixed width encoding in this situation where lengthUnits is 'characters'. (Please correct me if I am wrong.) I suggest making this combination an optional feature of the DFDL spec., would resolve the issue. This complex feature was added to support naive data format conversions where data originally had ascii encoding and lengthUnits 'bytes' is changed to 'utf-8' with lengthUnits 'characters'. This is a rational way to modernize a data format adding internationalization capability. It however requires a significant change in runtime behavior because utf-8 characters occupy between 1 and 4 bytes per character. SMH: IBM DFDL certainly supports lengthUnits="characters" and encoding="UTF-8", which is an example of this. Optional Features that are Partially Implemented The bigger set of concerns for interoperability is the behavior of a DFDL processor for features that are optional by strict interpretation of Section 21, but are implemented by a specific DFDL implementation, but the implementation is partial. This is the subject of other email messages however. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 11, 2018 at 11:33 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Action 307 was raised recently and first task is for implementations to identify which core spec behaviour is not implemented. IBM DFDL The following is the list of DFDL 1.0 spec core features that IBM DFDL does not yet implement. - dfdl:encodingErrorPolicy "replace" - dfdl:binaryBooleanTrueRep with value empty string - dfdl:assert on global element and simple type - dfdl:discriminator on global element and simple type - Multiple xs:appinfo elements within each xs:annotation element - When parsing, the distinction between an element being 'missing', having an 'empty representation' and having an 'absent representation', is not in accordance with the specification. - When encoding is 'UTF-8' or 'UTF-16', byte order marks are not processed The above lists are derived from information at https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… and are those that apply to core spec features. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL Working Group call
by Steve Hanson 05 Apr '19

05 Apr '19

1 0