August 2018 - dfdl-wg - lists.ogf.org

Minor spec correction - 9.3.1.1 missing numbering of 3 items.
by Mike Beckerle 02 Aug '18

02 Aug '18

In section 9.3.1.1, after the word "determination:" the next 3 lines should be numbered list items. This may have been reported before. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 1

Incorrect property name in section 12.3.6
by Steve Hanson 02 Aug '18

02 Aug '18

12.3.6 contains the following bullet: �h ��bytes�� if parent is a choice with dfdl:choiceKind ��explicit��; This should be: �h ��bytes�� if parent is a choice with dfdl:choiceLengthKind ��explicit��; Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

clarification needed - ambiguity about empty string and optional element
by Mike Beckerle 02 Aug '18

02 Aug '18

Consider this data of 4 characters: foo; Consider this schema where the default format is the basic general set of text-oriented defaults. <xs:element name="ex_infix" dfdl:lengthKind="implicit"> <xs:complexType> <xs:sequence dfdl:separator=";" dfdl:separatorSuppressionPolicy="anyEmpty" dfdl:separatorPosition="infix"> <xs:element name="x" type="xs:string" dfdl:lengthKind="delimited"/> <xs:element name="y" type="xs:string" minOccurs="0" dfdl:lengthKind="delimited" dfdl:occursCountKind="implicit"/> </xs:sequence> </xs:complexType> </xs:element> This is in a current Daffodil unit test, and produces this infoset: <ex_infix><x>foo</x><y/></ex_infix> That is, an empty string element is created for element 'y'. I'd like to know what IBM DFDL produces as the infoset for this example. I believe the DFDL spec is actually self-contradictory and so ambiguous here about what is the right behavior. - DFDL Spec 14.2.1 description of anyEmpty: "...any occurrences that have zero length representation MAY be omitted from the data, along with their associated separator." - Note that it says "may", not "must be". So anyEmpty is "lax" in insisting that the zero-length elements aren't present. - This doesn't clarify anything for us. But it admits the possibility that the ";" separator appears even if the 'y' element occurrence is determined to not exist. - DFDL Spec 9.3.1.1 says an element is known to exist if it has the nil, empty, or normal representation - In the example, element 'y' is zero-length which is either empty or normal representation since a string can have "" (empty string) as a value. - Since the 'y' element decl does not specify a XSD default value, the concept of 'empty' and defaulting doesn't apply here, so a zero-length string is a normal representation, and according to this section, it is known-to-exist. - This contradicts 9.4.2.2 below. - DFDL Spec 9.3.1.3 says "Note: based on the above, when processing a sequence for which a separator is defined, the presence of a match in the data for the separator is not sufficient to cause the parser to determine that an associated component is known-to-exist." It then refers you to 14.2.1 - I don't think this changes anything. Again it just admits that the separator ";" can appear even without the following element. I.e., I think it just allows for lax processing of excess separators. - DFDL Spec 9.4.2 Element Defaults When Parsing - Subsection 9.4.2.2 Simple element (xs:string or xs:hexBinary) (Emphasis below is mine) - Here's the excerpted text: - - "Required occurrence:* If the element has a default value* then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none'[12] <http://daffodil.apache.org/docs/dfdl/#_ftn12> then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, *otherwise nothing is added to the Infoset. * Note: *To prevent unwanted empty strings *or empty hexBinary values from being added to the Infoset, use XSD minLength > '0' and a dfdl:assert that uses the dfdl:checkConstraints() function, to raise a processing error." - Note that the language states "if the element has a default value" - which denotes that the section is dealing with both defaultable AND non-defaultable elements, and is not exclusively discussing defaultable elements as the title of 9.4.2 would imply. - The second statement is about optional occurrences, and it does not qualify what it says on defaultable element or not. Hence, I read the "nothing is added to the infoset" as applies whether or not the element is defaultable. So a zero length (ZL) string is never going to create an empty-string value for an optional element. - However, this contradicts the note about preventing unwanted empty strings. That note is only sensible if optional elements of zero-length will get added to the infoset and extra steps are required to force a facet check to prevent them. Unless I'm missing another place in the DFDL spec that clarifies this, I think we need to revise this area to make things clearer. But first we have to pick which is the intended semantics. In the example above, which infoset is the one we want: <ex_infix><x>foo</x><y/></ex_infix> (empty string as normal representation takes priority over optionality) or <ex_infix><x>foo</x></ex_infix> (optionality takes priority over empty string as normal representation) Either way I think this change is needed: - Section 9.4.2 - change section title to "Element Defaults and Optionality When Parsing" But a bunch of other clarifications are also needed. Today Daffodil 2.1.0 implements the first behavior. <ex_infix><x>foo</x><y/></ex_infix> with the empty 'y' element. What does IBM DFDL do? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Clarification needed: separator for empty sequence
by Mike Beckerle 02 Aug '18

02 Aug '18

I believe Daffodil has an incorrect behavior in the way it treats separators today, but I want to clarify things relative to the DFDL spec before fixing it. Consider this element: <xs:element name="NS_13"> <xs:complexType> <xs:sequence dfdl:separator="," dfdl:separatorPosition="infix" dfdl:separatorSuppressionPolicy="trailingEmptyStrict"> <xs:sequence> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert>{fn:true()}</dfdl:assert> </xs:appinfo> </xs:annotation> </xs:sequence> <xs:element name="e2" type="xs:int" minOccurs="1" maxOccurs="unbounded" dfdl:textNumberPattern="#####" dfdl:occursCountKind="implicit" /> </xs:sequence> </xs:complexType> </xs:element> The outermost sequence has infix separator. It's content begins with another sequence, but this sequence is empty, having only assert statements in it. Question: does this empty sequence cause a separator to be inserted after it? E.g., Should this data parse ",1,2,3" and is that initial comma required? An argument can be made that such empty sequences can be detected by a DFDL implementation and treated as having "no representation" akin to how an element with dfdl:inputValueCalc is treated. Such elements are invisible as far as the representation is concerned. They cause no separators in sequences, no alignment regions, etc. In that case the data would not require nor accept the initial comma. But the DFDL spec is not clear on whether empty sequences are treated in this way, so I am assuming they are not treated specially, so the comma is required because all model groups are treated as required regardless of whether they have zero-length representations. Is that correct? If so then daffodil has a bug, because it does NOT put a separator in for this today. What if the empty sequence carries a dfdl:initiator="A" annotation? Such a sequence is still empty in the XSDL sense of empty sequence, but clearly has a representation that is not zero length. In that case I think the data has to be "A,1,2,3" so that there is a separator after the empty sequence's non-ZL representation. I think this is not controversial. Other variations: What if the empty sequence contains elements with dfdl:inputValueCalc only. So there is a content model, but it is all non-represented elements. Would it still be a "DFDL empty sequence" or "DFDL non-represented sequence"? What if the sequence isn't empty, but contains only "optional" elements. In that case, is the whole sequence "optional" and so the comma becomes sensitive to the dfdl:separatorSuppressionPolicy? I guess this means a model group inherits the optionality/required-ness of its contents, unless it has its own required framing. I believe the simplest thing is to require the comma here. This is, however, a backward incompatible change to Daffodil to conform, so I want to be sure this is correct. The element declaration can be rewritten so that the empty sequence comes before the separated sequence. Presumably the reason someone would insert an empty sequence like this is to get the assert statement executed at the beginning of the sequence, not afterwards. However, if the sequence has separators, then you can't just insert an initial empty sequence to carry that assert without requiring a separator. The element can be rewritten: <xs:element name="NS_13"> <xs:complexType> <xs:sequence > <xs:sequence> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert>{fn:true()}</dfdl:assert> </xs:appinfo> </xs:annotation> </xs:sequence> <xs:sequence dfdl:separator="," dfdl:separatorPosition="infix" dfdl:separatorSuppressionPolicy="trailingEmptyStrict"> <xs:element name="e2" type="xs:int" maxOccurs="unbounded" dfdl:textNumberPattern="#####" /> </xs:sequence> </xs:sequence> </xs:complexType> </xs:element> This achieves the desired behavior where the assertion executes first, but does not cause a separator to be needed, and does so regardless of the treatment of empty sequences. Ultimately, we need a clarification that states either: 1) empty sequences are considered represented terms that are required, and hence, they imply framing such as alignment and presence of separators in separated sequences even if they have zero length. or 2) Introduce concept "DFDL non-represented sequence". These are sequences that have no framing, no delimiters, and have empty content model, or only elements with dfdl:inputValueCalc or other DFDL non-represented sequences in their content model (recursively). They have no representation, so imply no alignment, no need of separators, etc. A group ref (hidden or not) to a DFDL non-represented sequence is also a DFDL non-represented sequence. Note that these could be generalized to choice groups also. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Spec correction ? - Section 9.3.2.1 - second list missing "empty" representation
by Mike Beckerle 02 Aug '18

02 Aug '18

In section 9.3.2.1, there are two numbered lists. The first numbered list should qualify its mention of "empty representation" with "(when the emptyValueDelimiterPolicy and initiator and terminator policies are defined such that zero-length is allowed as the empty representation.)" The second numbered list should mention nil (literal), nil (logical), empty, and normal representations. It is missing discussion of non-zero-length "empty" representations (due to emptyValueDelimiterPolicy not none, and initiator and terminator defined such that the empty representation is non-zero length). Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Clarification needed: stricter separator suppresssion policy
by Mike Beckerle 02 Aug '18

02 Aug '18

I have been trying to rationalize the definitions in the DFDL spec for separated sequences. I cannot find a way to express something very simple: A repeating element with minOccurs non-zero-length elements with separators required, and up to maxOccurs (or unbounded) non-zero-length elements with separators are allowed. This means separators can never be adjacent anywhere in the corresponding data stream (except if escaped, or hidden inside say a fixed-length string inside a complex type element). Adjacent delimiters would be a parse error. I expected this to be occursCountKind 'implicit' with separatorSuppressionPolicy 'never', but that appears to mean that maxOccurs must be bounded and there are always exactly maxOccurs separators, the latter of which (maxOccurs - minOccurs of them) can be empty strings, meaning optional elements will not be created for them. All the other 3 separator suppression policies absorb adjacent separators, except for trailingEmptyStrict doesn't absorb them at the end of the group. There doesn't seem to be a way to be strict about the format and speculatively parse only non-zero-length elements requiring each optional occurance to appear with associated separator. I.e., no trailing adjacent separators, and no adjacent separators in the middle or beginning either. Are we missing separatorSuppressionPolicy='neverEmpty' or 'anyEmptyStrict' perhaps? Comments? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

clarification: on suppressed ZL string/hexBinary - do we keep variable assignments?
by Mike Beckerle 02 Aug '18

02 Aug '18

In some situations we parse and get a successful zero-length parse for a string or hexBinary. But because the occurrence is optional, we do NOT add an element to the infoset. In that case, what happens to side-effects that occurred during the successful parse. There are two possible kinds of side-effects. Variables can be set, and a discriminator can be set to true. It seems to me that if a discriminator is set, then that *must* be preserved, and in that case it would seem the variable settings should be retained as well. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 6

Clarification needed: 14.2.2 Parsing Sequence Groups with Separators - Rep(N)
by Mike Beckerle 01 Aug '18

01 Aug '18

In 14.2.2 the phrase: * Rep(M)* means M occurrences of nil, empty, normal or absent representation. These are optional occurrences so default rules do not apply for empty representations. Begs the question of what happens if the empty representation is not the same as absent representation due to initiators and/or terminators being required by emptyValueDelimiterPolicy. If the empty representation is found, what would a Parser do? These are optional, so defaulting is not applicable. Treat it as "absent" is one possibility, Treat it as "normal" which is likely to result in a parse error, is another. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 1

Clarification: empty sequence - how can it be non-zero content length
by Mike Beckerle 01 Aug '18

01 Aug '18

Section 14.1 states: It is a processing error if the SequenceContent region of an empty sequence has non-zero length when parsing. How is it possible for an empty sequence to have a non-zero length of its content region? I understand it could have delimiters outside the content region, and other framing possibly, but the content region has nothing in it then only annotations can appear. Annotations can be property definitions or can be DFDL statements. None of those can consume representation bits when parsing, or produce representation when unparsing. So I believe this processing error cannot happen. If I am correct, the statement about processing error should be dropped. Also, clearly this statement does not apply to a sequence that carries a dfdl:hiddenGroupRef, since the hidden group can contribute to the content region. I would not consider a sequence carrying hiddenGroupRef to be empty unless the referenced group is also empty. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1