July 2018 - dfdl-wg - lists.ogf.org

clarification needed: unparsing and hidden group choices and choices with dfdl:choiceDispatchKey
by Mike Beckerle 13 Dec '19

13 Dec '19

So interesting issue. Comes up as part of our experience implementing the advanced expression language features: hidden groups, and computed elements. We have hidden groups in DFDL. If a xs:choice is inside a hidden group, then by definition, when unparsing, elements corresponding to some branch of the choice are not in the Infoset. DFDL spec clearly states that dfdl:outputValueCalc is not allowed on the head of a choice branch. But elements inside hidden groups must have dfdl:outputValueCalc, or a default value, otherwise there's no way for them to be filled in as part of the augmented infoset. So the only way a choice can be inside a hidden group is if the head of the choice branches have default values. This is not sufficient. I do not think it was our intention to make hidden groups incapable of holding things complex enough to contain a choice. Rather, the point is to be able to hide things that have excess complexity needed only for parsing/unparsing, so that they need not appear in the final Infoset. So if a choice is allowed inside a hidden group, then a mechanism is needed by which when unparsing a choice branch is selected based on values of other parts of the Infoset, and then the elements of that branch are computed and added to the augmented infoset. Note that without hidden groups, this problem never arises, as the Infoset always has to take a position on choice branches by having an unambiguous element in it. There are 3 cases: 1. unparsing an xs:choice where for parsing, each branch has a discriminator, or is resolved by the simple "first successful parse" rule. 2. unparsing an xs:choice with dfdl:choiceDispatchKey 3. unparsing xs:choice with dfdl:initiatedContent='yes' For all cases, I suggest we must remove the restriction on dfdl:outputValueCalc being disallowed on choice branch head. For (1). First branch where the branch can be constructed successfully is added to the augmented infoset. This is a case where an unparse error is non-fatal, and causes a form of unparser backtracking through alternatives. Branches would be attempted in schema order. For (2) I suggest dfdl:unparseChoiceDispatchKey, evaluated at unparse time only. Produces a choice branch key. That choice branch is then constructed, and default values or dfdl:outputValueCalcs are used to fill in values. (BTW: this is the driving use case we have in Daffodil for this issue.) For (3).... I have no specific solution. I think it may not actually be a separate case and has to be treated as (1) or (2). Comments/Discussion?

2 1

Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes
by Mike Beckerle 28 Apr '19

28 Apr '19

So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it. I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features. The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as http://www.xml.com/pub/a/2004/07/21/design.html These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself. The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike, > > Your scheme as proposed has a couple of problems. > a) It is possible to create a schema that does not use dfdl:format at all. > Unlikely but possible. With such a schema there is no way to indicate > version. > b) What are the version implications for annotations that have nothing to > do with dfdl:format? Asserts, discriminators, variables. > > Most of the language you use refers to versions of *schemas*. That's not > the same as version of a dfdl:format. Any reason you didn't just change the > DFDL namespace URL to be *http://www.ogf.org/dfdl/dfdl-2.0/* > <http://www.ogf.org/dfdl/dfdl-2.0/>, which must be declared before any > DFDL feature can be used? I think that was always the intent of the DFDL > namespace. > > The key piece of work is validating any proposal against the scoping rules > in section 8 of the spec. > > Regards > > Steve Hanson > > IBM Hybrid Integration, Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 <+44%201962%20815848> > mob:+44-7717-378890 <+44%207717%20378890> > Note: I work Tuesday to Friday > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org > Date: 23/01/2018 16:03 > Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 > features/changes > Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> > ------------------------------ > > > > > There are some features now under discussion which will most likely not be > part of the DFDL v1.0 standard, but a subsequent version 2.0 of the > standard. > > Implementations may have these features prior to the existence of that > draft version of the DFDL standard; hence, a mechanism for indicating the > support, and requirements that implies, on DFDL schemas is needed in order > for these v2.0 features to be introduced. > > The purpose of the versioning mechanism is to provide access to DFDL v2.0 > features, while maintaining also backward compatibility with DFDL v1.0 > schemas. > > For example, new DFDL format properties may be added to the DFDL v2.0 > version; however, those properties are not required to be specified unless > the version of the schema is specifically set to indicate it is using > version 2.0 features. Also, deprecated features should only generate > deprecation warnings if the user is requesting version 2.0 of DFDL, and > should be silent otherwise. > > Proposal: > > New attribute on the dfdl:format annotation: DFDLVersion with valid values > "1", "2". All values are reserved for future use. > > The DFDLVersion, if unspecified, defaults to "1" which means the > implementation is of the DFDL v1.0 specification. > > If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL > may be enforced on the schema, and schema definition errors will result if > version 2.0 requirements are not met. DFDL Version 1.0 features that are > deprecated will generate warnings. > > Example: > > <dfdl:format DFDLVersion="2" .../> > > Example: > > <dfdl:defineFormat name="myV2Format"> > <dfdl:format DFDLVersion="2" .../> > </dfdl:defineFormat> > > The DFDLVersion is not a format property and is not scoped. It can be > added to a named format definition which is then referenced in the usual > manner for named format definitions. > > Backward Compatibility Requirement > > DFDL Implementations that provide version 2.0 functionality must also > provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends > version 1.0. > > Mixed Version Schemas > > If a schema consists of multiple schema documents, and their respective > dfdl:format annotations do not all contain the same value for the > DFDLVersion attribute, then the schema is said to be "mixed version". This > is allowed. > > A named dfdl:format carrying DFDLVersion "2" may not be referenced from a > DFDL Schema document having DFDLVersion "1". > > The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or > lacking the DFDLVersion attribute may be referenced from a DFDL Schema > having DFDLVersion="2". > > There are really 3 ways that DFDL version 2.0 can differ from version 1.0, > and these are that new features are added, old features are deprecated, or > old features are removed. This latter is quite undesirable, but will be > considered a possibility for purpose of discussion this versioning system. > > These will commonly be in combination such as if an old feature is > deprecated with the replacement feature being added. > > These will now be discussed separately, with the emphasis on how mixed > version schemas must work. > > New Features Added > > Each new required DFDL property introduced for version 2.0 will be defined > along with what is called its "version 1.0 default value". When a mixed > version schema exists, the DFDL v1.0 parts of it behave as if any required > DFDLVersion 2 properties have their version 1.0 default value defined, at > top level of the schema (as if defined with that default value in each > DFDLVersion 1 schema document's dfdl:format annotation). > > Features Removed > > Incompatibilities are to be avoided wherever possible, and may not be > needed at all. However, there is still the possibility that some feature > will want to be entirely removed and replaced by something better, and in > that case the versioning scheme needs to be able to support this. > > In that situation, the schema definition error for version 2 is only > generated when a schema has DFDLVersion 2 specified (in dfdl:format > explicitly, or by reference to a named format carrying or referencing > DFDLVersion 2). Mixed schemas that contain schema documents entirely using > version 1 of DFDL do not generate this schema definition error. > > Features deprecated > > Deprecation warnings are generated only when a DFDLVersion="2" attribute > is present or is incorporated by reference (to a named format). A version 1 > schema that is incorporated with a version 2 DFDL schema but is not > modified in any way, does not generate deprecation warnings about version > 1.0 features that have been deprecated in version 2.0. > > (Most likely deprecations would be revising property names or value enums > for clarity.) > > Implementations may provide mechanisms for suppressing warnings. These are > implementation dependent. > > Version 1.0 Existing Deprecations and Compatibility > > During the process of standardization for DFDL v1.0 a number of properties > names were changed. DFDL version 1.0 implementations accept both the old > and new property names. > (Example: dfdl:separatorSuppressionPolicy, which replaced an older > property name.) DFDL schemas that specify DFDLVersion="2" will not accept > the older property names, and it is a schema definition error if the old > (pre DFDL v1.0) property names are used. > > Properly constructed mixed version schemas can still depend on the older > property names in the version 1.0 parts of the schema. > > ------------------------------ > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > *www.tresys.com* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ…> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the *OGF Intellectual Property Policy* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5…> > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=lDIa8xvu2n-e8 > PXWCJmajfYaXESMqvvOWpg1HMWeuGQ&s=hdO_VQ_p-1f7JfZPcSTPR1S_Y1a > nSBXJWarqPdJZdPc&e= > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >

2 6

Action 292 - Write up of hexBinary with lengthUnits 'bits'
by Mike Beckerle 20 Nov '18

20 Nov '18

The proposed change is to allow lengthUnits 'bits' for hexBinary data. This turned out to be more complex to describe than I originally suspected because of the need to deal with XSD minLength and maxLength facets, which are always measured in bytes. Those in conjunction with dfdl:lengthKind='explicit' and dfdl:lengthUnits='bits' create some minor complexities. The below changes match the Daffodil implementation of this proposed feature. These sentences in the description of dfdl:lengthUnits in Section 12.3 must change. - 'bits' may only be used for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt, and xs:unsignedLong simple types with binary representation. - 'bytes' must be used for type xs:hexBinary. The text should read - 'bits' may only be used for xs:hexBinary and for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt, and xs:unsignedLong simple types with binary representation. Later in the section, the paragraph: When unparsing a simple element with binary representation, then for hexBinary the length is the number of bytes in the infoset value padded to the XSD minLength facet value using dfdl:fillByte, and for the other types the length is the minimum number of bytes to represent the value and any sign. Must change to: When unparsing a simple element with binary representation, then for types other than hexBinary the length is the minimum number of bytes to represent the value and any sign. For type hexBinary when the dfdl:lengthUnits is 'bytes' then the length is the number of bytes in the infoset value padded to the XSD minLength facet value using dfdl:fillByte. For type hexBinary when the dfdl:lengthUnits is 'bits': - First the data is padded to XSD minLength bytes as if the dfdl:lengthUnits was 'bytes'. - When dfdl:lengthKind is other than 'explicit', the length is the number of bytes times 8. - When the dfdl:lengthKind is 'explicit' then the value is further padded or truncated to fit the target length, in bits. - if the data does not have sufficient bytes to supply the target length in bits it is a processing error. - if the data is longer than the minimum number of bytes needed to supply the target length in bits, it is a processing error. - If the explicit length in bits is not a multiple of 8, then the final byte is only partially unparsed according to the current dfdl:byteOrder. Section 12.3.7.2.7 Length of Binary Opaque Elements, the first sentence must be modified from: "The dfdl:lengthUnits property must be 'bytes'. It is a schema definition error otherwise." to "The dfdl:lengthUnits property must be 'bytes' or 'bits'. It is a schema definition error otherwise. Note that even when the dfdl:lengthUnits property is 'bits', the values of the XSD minLength and XSD maxLength facets are still always interpreted as constraining the length in units of bytes. That's the end of the actual proposed language. Note about alternatives: I considered the alternative to make it a schema definition error when dfdl:lengthUnits is 'bits' type is 'xs:hexBinary', and the XSD:minLength or XSD:maxLength facets are defined. I decided to go with the description above to support the use case where a data item is, in hex, some number of bytes long for a valid XML infoset, but the representation is explicitly a partial byte smaller. E.g., in a hexBinary with 8 bytes, but fewer than 64 bits in the representation. Ex: 63 bits as the explicit target length, so 1 bit will be unused from the xs:hexBinary logical value, but the above rules insure no more than 7 bits go unused from the final byte of the hexBinary logical value. This is trying to be consistent with the notion that we do not truncate data to fit into the available length except for xs:string when properties explicitly allow it. It is also trying to be consistent with our treatment of binary integers where a xs:long value may be output into an element having room for any number of bits, and any extra bits in the logical value are ignored. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Clarification discussion points for next call(s) - DFDL Spec issues around nil, empty, normal, absent, defaulting, and separator suppression.
by Mike Beckerle 02 Aug '18

02 Aug '18

I'm trying to tighten up my understanding of section 9 materials around establishing representation, known to/not-to exist, and defaulting, particularly as they interact with separators and separator suppresssion for "absent" representations. Section 9.2.5 This sentence The nil representation can be a zero-length representation if dfdl:nilValue is "%ES;", and there is no framing or framing is suppressed by dfdl:nilValueDelimiterPolicy. Should say ".... if dfdl:nilValue is a list containing either %ES; alone, or %WSP*; alone, and ...." This matches existing erratum 5.32. It's just another place that needs the same update. Sections 9.2.1 to 9.2.4 do not say what happens if when trying to establish the representation, an assertion failure occurs. In particular, can an assertion failure cause establishing of nil, empty, or normal represntation to fail, resulting in absent representation? For example, if an assert requires the length of the value to be greater than 1 character, then a zero-length string cannot be the normal representation. But do we parse to nil representation and run assertions - and if fail, parse for empty representation, then re-run assertions, and if fail parse for normal representation, then re-run assertions, and if fail and the rep was "trivially ZL", then it is absent? Section 9.2.3 doesn't say what happens if a type conversion error happens when trying to establish Normal representation. E.g., if a non-defaultable text integer is parsed from a zero-length string. T A section 9.2.5 should be added describing "No representation" - Computed elements (dfdl:inputValueCalc) have no representation at all. Not zero length, not anything. They have no implications for parsing of, or unparsing of, delimiters of any kind. Section 9.3 Section 9.3.1.1 says establishing known-to-exist, no processing error can occur. For example, an element can have zero-length representation if it is nillable and nilValue=%ES; and there are no delimiters nor framing. But an assert can fail for this by specifically excluding dfdl:valueLength(.) eq 0. Or an element of type string can have zero length, but an assert can insist the string contain 1 or more characters. Section 9.3.1.3 says known not to exist if the occurence is Missing, which means absent representation is one way. So can an assert that causes nil, empty, and normal representation to fail (assuming asserts are evaluated and contribute to that decision about representation) can cause the occurrence to be absent; hence, missing. It also says a processing error when parsing the component means it is known not to exist. Section 9.3.2 Establishing Representation Section 9.3.2.1 Simple Element (1) already has an erratum allowing WSP* alone as well as ES. Does not say whether (2) empty representation - qualified by "empty representation must be able to be of zero-length". (3) normal representation - does not say if asserts can fail and prevent this zero-length from being acceptable. (actually this point about assertions applies to all of 1, 2, and 3. Section 9.3.2.2 Complex Element If a complex type element is lengthKind 'delimited' do we still have to recurse the type before deciding if it is trivially zero length, or can we look at the data stream without recursing in? Section 9.3.2 5th bullet says that we can look for if a delimiter is immediately encountered, and does not say this is for simple types only. A complex type element can have empty representation if the content region is empty and the delimiters with EVDP specify what is found in the data stream. Absent representation for a complex element can only occur if the representation is zero length after recursing through the type tree. This implies that if EVDP indicates delimiters for empty value, then ZL means absent. Section 9.3.3 StopValue seems like it has a point of uncertainty for every occurrance. The fact that a stop value must exist doesn't mean there are no points of uncertainty. It is uncertain if the logical value will be the stop value or not. But another way of thinking about it is that the stopvalue parser does not need to establish points for backtracking. All elements MUST succeed until it parses a successful stop value, and if any failure occurs we backtrack the entire array, not just an element. Use of stopValue with type xs:string creates lots of ambiguities. E.g., ZL can be a valid normal representation, but the stop value may be "stop", i.e., non-ZL. In that case, since all elements are optional according to minOccurs 0, then when a ZL is parsed is the optional element suppressed? Or not - meaning you get an array full of empty string "normal" values? Section 9.4.2 Says a complex type must have descended into the type and returned with no processing error, but does not say whether processing errors signaled by asserts on simple type elements also disable empty representation from being established. Section 9.4.2.2 Says if EVDP is not none, can an assert insist on something that subverts establishment of empty representation, such as that the length is > 0? Or the assert can test something orthogonal - entirely unrelated like some variable is set to a certain value? E.g., <defineVariable name="disallowEmptyValues" type="xs:boolean".../> Then an assert on the element says { if ($disallowEmptyValues) then false else true }. For optional occurrence, if EVDP is not none, then empty representation is established by the presence of some positive syntax - the representation is not ZL. However, this says a empty string or empty hexbinary becomes the value, not the default value of the element. Is that correct? I would think having positive syntax that matches the empty representation would satisfy known-to-exist, and then that would trigger assigning the default value. However, I suppose the rationale is that such an empty value for an optional element means there will be no defaulting, hence, the empty representation corresponds to empty string or empty hexBinary as "normal" representation. The suggestion to use an assert that checks a non-zero minLength facet only makes sense if the processing error will cause the end of the array (occursCountKind parsed or implicit). If the OCK is such that there is no point of uncertainty, then this processing error would cause the whole array-element to fail. That is, the assert suggested here really does too much to be used only to filter empty strings/hexBinary from going into the infoset. Or does a ZL failure for a delimited simpleType value, where the text is ZL, but the type conversion fails, or an assert fails, does that create Absent representation resulting in no empty string going into the infoset? I suspect this issue is tied up with separator suppression policy and when a ZL thing is suppressed, the separator absorbed, and nothing goes into the infoset. 9.4.2.3 - suggests that processors must keep track of the "all empty flag" for every infoset node and recursively all child nodes. This section should say that a complex type has empty representation if it is known to exist, and the position in the data doesn't change after a recursive traversal. But this last paragraph of the section contradicts what is said in the 2nd sentence of the section (maybe). The point the example is making - the principle it is illustrating, does not seem to be explicilty stated. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Minor spec correction - 9.3.1.1 missing numbering of 3 items.
by Mike Beckerle 02 Aug '18

02 Aug '18

In section 9.3.1.1, after the word "determination:" the next 3 lines should be numbered list items. This may have been reported before. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 1

clarification needed - ambiguity about empty string and optional element
by Mike Beckerle 02 Aug '18

02 Aug '18

Consider this data of 4 characters: foo; Consider this schema where the default format is the basic general set of text-oriented defaults. <xs:element name="ex_infix" dfdl:lengthKind="implicit"> <xs:complexType> <xs:sequence dfdl:separator=";" dfdl:separatorSuppressionPolicy="anyEmpty" dfdl:separatorPosition="infix"> <xs:element name="x" type="xs:string" dfdl:lengthKind="delimited"/> <xs:element name="y" type="xs:string" minOccurs="0" dfdl:lengthKind="delimited" dfdl:occursCountKind="implicit"/> </xs:sequence> </xs:complexType> </xs:element> This is in a current Daffodil unit test, and produces this infoset: <ex_infix><x>foo</x><y/></ex_infix> That is, an empty string element is created for element 'y'. I'd like to know what IBM DFDL produces as the infoset for this example. I believe the DFDL spec is actually self-contradictory and so ambiguous here about what is the right behavior. - DFDL Spec 14.2.1 description of anyEmpty: "...any occurrences that have zero length representation MAY be omitted from the data, along with their associated separator." - Note that it says "may", not "must be". So anyEmpty is "lax" in insisting that the zero-length elements aren't present. - This doesn't clarify anything for us. But it admits the possibility that the ";" separator appears even if the 'y' element occurrence is determined to not exist. - DFDL Spec 9.3.1.1 says an element is known to exist if it has the nil, empty, or normal representation - In the example, element 'y' is zero-length which is either empty or normal representation since a string can have "" (empty string) as a value. - Since the 'y' element decl does not specify a XSD default value, the concept of 'empty' and defaulting doesn't apply here, so a zero-length string is a normal representation, and according to this section, it is known-to-exist. - This contradicts 9.4.2.2 below. - DFDL Spec 9.3.1.3 says "Note: based on the above, when processing a sequence for which a separator is defined, the presence of a match in the data for the separator is not sufficient to cause the parser to determine that an associated component is known-to-exist." It then refers you to 14.2.1 - I don't think this changes anything. Again it just admits that the separator ";" can appear even without the following element. I.e., I think it just allows for lax processing of excess separators. - DFDL Spec 9.4.2 Element Defaults When Parsing - Subsection 9.4.2.2 Simple element (xs:string or xs:hexBinary) (Emphasis below is mine) - Here's the excerpted text: - - "Required occurrence:* If the element has a default value* then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none'[12] <http://daffodil.apache.org/docs/dfdl/#_ftn12> then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, *otherwise nothing is added to the Infoset. * Note: *To prevent unwanted empty strings *or empty hexBinary values from being added to the Infoset, use XSD minLength > '0' and a dfdl:assert that uses the dfdl:checkConstraints() function, to raise a processing error." - Note that the language states "if the element has a default value" - which denotes that the section is dealing with both defaultable AND non-defaultable elements, and is not exclusively discussing defaultable elements as the title of 9.4.2 would imply. - The second statement is about optional occurrences, and it does not qualify what it says on defaultable element or not. Hence, I read the "nothing is added to the infoset" as applies whether or not the element is defaultable. So a zero length (ZL) string is never going to create an empty-string value for an optional element. - However, this contradicts the note about preventing unwanted empty strings. That note is only sensible if optional elements of zero-length will get added to the infoset and extra steps are required to force a facet check to prevent them. Unless I'm missing another place in the DFDL spec that clarifies this, I think we need to revise this area to make things clearer. But first we have to pick which is the intended semantics. In the example above, which infoset is the one we want: <ex_infix><x>foo</x><y/></ex_infix> (empty string as normal representation takes priority over optionality) or <ex_infix><x>foo</x></ex_infix> (optionality takes priority over empty string as normal representation) Either way I think this change is needed: - Section 9.4.2 - change section title to "Element Defaults and Optionality When Parsing" But a bunch of other clarifications are also needed. Today Daffodil 2.1.0 implements the first behavior. <ex_infix><x>foo</x><y/></ex_infix> with the empty 'y' element. What does IBM DFDL do? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Clarification needed: separator for empty sequence
by Mike Beckerle 02 Aug '18

02 Aug '18

I believe Daffodil has an incorrect behavior in the way it treats separators today, but I want to clarify things relative to the DFDL spec before fixing it. Consider this element: <xs:element name="NS_13"> <xs:complexType> <xs:sequence dfdl:separator="," dfdl:separatorPosition="infix" dfdl:separatorSuppressionPolicy="trailingEmptyStrict"> <xs:sequence> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert>{fn:true()}</dfdl:assert> </xs:appinfo> </xs:annotation> </xs:sequence> <xs:element name="e2" type="xs:int" minOccurs="1" maxOccurs="unbounded" dfdl:textNumberPattern="#####" dfdl:occursCountKind="implicit" /> </xs:sequence> </xs:complexType> </xs:element> The outermost sequence has infix separator. It's content begins with another sequence, but this sequence is empty, having only assert statements in it. Question: does this empty sequence cause a separator to be inserted after it? E.g., Should this data parse ",1,2,3" and is that initial comma required? An argument can be made that such empty sequences can be detected by a DFDL implementation and treated as having "no representation" akin to how an element with dfdl:inputValueCalc is treated. Such elements are invisible as far as the representation is concerned. They cause no separators in sequences, no alignment regions, etc. In that case the data would not require nor accept the initial comma. But the DFDL spec is not clear on whether empty sequences are treated in this way, so I am assuming they are not treated specially, so the comma is required because all model groups are treated as required regardless of whether they have zero-length representations. Is that correct? If so then daffodil has a bug, because it does NOT put a separator in for this today. What if the empty sequence carries a dfdl:initiator="A" annotation? Such a sequence is still empty in the XSDL sense of empty sequence, but clearly has a representation that is not zero length. In that case I think the data has to be "A,1,2,3" so that there is a separator after the empty sequence's non-ZL representation. I think this is not controversial. Other variations: What if the empty sequence contains elements with dfdl:inputValueCalc only. So there is a content model, but it is all non-represented elements. Would it still be a "DFDL empty sequence" or "DFDL non-represented sequence"? What if the sequence isn't empty, but contains only "optional" elements. In that case, is the whole sequence "optional" and so the comma becomes sensitive to the dfdl:separatorSuppressionPolicy? I guess this means a model group inherits the optionality/required-ness of its contents, unless it has its own required framing. I believe the simplest thing is to require the comma here. This is, however, a backward incompatible change to Daffodil to conform, so I want to be sure this is correct. The element declaration can be rewritten so that the empty sequence comes before the separated sequence. Presumably the reason someone would insert an empty sequence like this is to get the assert statement executed at the beginning of the sequence, not afterwards. However, if the sequence has separators, then you can't just insert an initial empty sequence to carry that assert without requiring a separator. The element can be rewritten: <xs:element name="NS_13"> <xs:complexType> <xs:sequence > <xs:sequence> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert>{fn:true()}</dfdl:assert> </xs:appinfo> </xs:annotation> </xs:sequence> <xs:sequence dfdl:separator="," dfdl:separatorPosition="infix" dfdl:separatorSuppressionPolicy="trailingEmptyStrict"> <xs:element name="e2" type="xs:int" maxOccurs="unbounded" dfdl:textNumberPattern="#####" /> </xs:sequence> </xs:sequence> </xs:complexType> </xs:element> This achieves the desired behavior where the assertion executes first, but does not cause a separator to be needed, and does so regardless of the treatment of empty sequences. Ultimately, we need a clarification that states either: 1) empty sequences are considered represented terms that are required, and hence, they imply framing such as alignment and presence of separators in separated sequences even if they have zero length. or 2) Introduce concept "DFDL non-represented sequence". These are sequences that have no framing, no delimiters, and have empty content model, or only elements with dfdl:inputValueCalc or other DFDL non-represented sequences in their content model (recursively). They have no representation, so imply no alignment, no need of separators, etc. A group ref (hidden or not) to a DFDL non-represented sequence is also a DFDL non-represented sequence. Note that these could be generalized to choice groups also. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Spec correction ? - Section 9.3.2.1 - second list missing "empty" representation
by Mike Beckerle 02 Aug '18

02 Aug '18

In section 9.3.2.1, there are two numbered lists. The first numbered list should qualify its mention of "empty representation" with "(when the emptyValueDelimiterPolicy and initiator and terminator policies are defined such that zero-length is allowed as the empty representation.)" The second numbered list should mention nil (literal), nil (logical), empty, and normal representations. It is missing discussion of non-zero-length "empty" representations (due to emptyValueDelimiterPolicy not none, and initiator and terminator defined such that the empty representation is non-zero length). Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Clarification needed: stricter separator suppresssion policy
by Mike Beckerle 02 Aug '18

02 Aug '18

I have been trying to rationalize the definitions in the DFDL spec for separated sequences. I cannot find a way to express something very simple: A repeating element with minOccurs non-zero-length elements with separators required, and up to maxOccurs (or unbounded) non-zero-length elements with separators are allowed. This means separators can never be adjacent anywhere in the corresponding data stream (except if escaped, or hidden inside say a fixed-length string inside a complex type element). Adjacent delimiters would be a parse error. I expected this to be occursCountKind 'implicit' with separatorSuppressionPolicy 'never', but that appears to mean that maxOccurs must be bounded and there are always exactly maxOccurs separators, the latter of which (maxOccurs - minOccurs of them) can be empty strings, meaning optional elements will not be created for them. All the other 3 separator suppression policies absorb adjacent separators, except for trailingEmptyStrict doesn't absorb them at the end of the group. There doesn't seem to be a way to be strict about the format and speculatively parse only non-zero-length elements requiring each optional occurance to appear with associated separator. I.e., no trailing adjacent separators, and no adjacent separators in the middle or beginning either. Are we missing separatorSuppressionPolicy='neverEmpty' or 'anyEmptyStrict' perhaps? Comments? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

clarification: on suppressed ZL string/hexBinary - do we keep variable assignments?
by Mike Beckerle 02 Aug '18

02 Aug '18

In some situations we parse and get a successful zero-length parse for a string or hexBinary. But because the occurrence is optional, we do NOT add an element to the infoset. In that case, what happens to side-effects that occurred during the successful parse. There are two possible kinds of side-effects. Variables can be set, and a discriminator can be set to true. It seems to me that if a discriminator is set, then that *must* be preserved, and in that case it would seem the variable settings should be retained as well. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 6