October 2014 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

glossary entry for "fixed length" needed.
by Mike Beckerle 26 Nov '14

26 Nov '14

Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Action 271: semantic of maxoccurs '0'
by Steve Hanson 25 Nov '14

25 Nov '14

271 Use of maxOccurs '0' (All) 2/9: Legal in XSDL and DFDL. One use case in XSDL is when deriving a complex type by restriction, where it is used to indicate that an element in the base type must not appear. Another use case could be if a schema undergoes a version revision that removes elements; perhaps clearer if the removed element gets maxOccurs '0' rather than omitting it. Steve has seen an instance of its use in a DFDL schema. So if DFDL continues to support it, need to document the behaviour for the various occursCountKinds and what happens to separator. Also behaviour of occursCount expression that returns 0. Proposal needed. ... 23/9: No progress IBM has discussed this internally and has the following proposal. Infoset refers to the augmented infoset. OCK maxOccurs '0' Parsing Unparsing fixed No occurrences looked for in the data (1) No occurrences looked for in the infoset or written (2) implicit No occurrences looked for in the data (1) No occurrences looked for in the infoset or written (2) expression occursCount occurrences expected. If any are found it is a validation error. Any number of occurrences expected in the infoset and written. If any are found it is a validation error. parsed Any number of occurrences expected in the data. If any are found it is a validation error. Any number of occurrences expected in the infoset and written. If any are found it is a validation error. stopValue Any number of occurrences expected in the data. If any are found it is a validation error. Any number of occurrences expected in the infoset and written. If any are found it is a validation error. OCK Parsing and occursCount '0' Unparsing and no occurrences in infoset expression No occurrences looked for in the data. (1) No occurrences written. (2) Notes (1) If sequence has a separator, no separator is looked for in the data when parsing. (2) If sequence has a separator, no separator is output when unparsing. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 4

Fw: Fw: Action 260
by Steve Hanson 18 Nov '14

18 Nov '14

IBM has discussed this issue at some length internally, and come to the conclusion that the optional and array elements in a sequence should follow the separator suppression policy (SSP), and not have an implied SSP which is at odds with the sequence's SSP. In the table below, cells marked with a cross imply a schema definition error, and cells marked ok imply that there is a behaviour of the element for that OCK which is in keeping with the SSP of the sequence. SSP (1) OCK fixed implicit expression parsed stopValue (2) never ok ok ok x (5) ok (6) trailingEmpty trailingEmptyStrict ok (3) ok ok (4) x (5) ok (6) anyEmpty ok (3) ok ok (4) ok ok Notes: (1) SSP property applies only to an ordered sequence. An unordered sequence assumes 'anyEmpty' (as all optional/array elements must be 'parsed') (2) Missing restriction - for 'stopValue' the dfdl:stopValue property must not include empty string. (3) maxOccurs provides count, so nothing is eligible for suppression, so SSP has no practical effect (same as a required element) (4) infoset provides count, so nothing is eligible for suppression, so SSP has no practical effect (same as a required element) (5) 'parsed' only makes sense with 'anyEmpty' (6) Because a stop value must appear, and from (2) empty string is not allowed, SSP has no practical effect. The issue of maxOccurs = '0' is discussed in a separate email. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/10/2014 14:40 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 29/08/2014 21:21 Subject: Re: [DFDL-WG] Fw: Action 260 I reviewed this. It looks good to me. The note at the bottom that we don't say what happens on a zero-trip I.e., a represented element, but where occursCount evaluates to 0, is a useful clarification also. Do we want to create an erratum for this? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Thu, Aug 28, 2014 at 10:13 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Please review for Tuesday's WG call ... Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2014 15:02 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 06/08/2014 13:50 Subject: Fw: [DFDL-WG] Action 260 So my suggestion below, to wrap the array in a sequence, does not work; it just moves the problem down into the new sequence. After much deliberation, we think that the definitions of Positional sequence and Non-positional sequence should not be viewed as driving the behaviour of a sequence, but simply as the resultant characteristics of a sequence that has certain properties. That leaves modellers free to mix occursCountKinds, as in Tim's example. No need for any new SDE scenarios. Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. In DFDL, a sequence is considered positional if it contains only required elements and/or optional and array elements that have dfdl:occursCountKind 'implicit', 'fixed' or 'expression', and it has dfdl:separatorSuppressionPolicy 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Often the components of such a sequence have an initiator. Such sequences sometimes allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing might need to be employed by to identify each occurrence. In DFDL, a sequence is non-positional if it contains any optional or array elements that have dfdl:occursCountKind 'parsed' or 'stopValue', and/or it has dfdl:separatorSuppressionPolicy 'anyEmpty'. See parallel email for action 261 that ensures 'expression' behaves itself. One behaviour that is missing from the spec. For a sequence with separators, what is expected in the data stream if occursCount = 'fixed' / 'implicit' and maxOccurs = '0', or occursCountKind = 'expression' and occursCount evaluates to 0 ? We believe that no separator should be expected when parsing and none output when unparsing (same behaviour as inputValueCalc). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/08/2014 12:42 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 30/06/2014 10:04 Subject: Re: [DFDL-WG] Action 260 You would wrap the array and it's count in a sequence. Then the 'count+array' is treated as a single entity as far as the parent sequence is concerned. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 26/06/2014 20:06 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org Before we settle one way or the other, I would like the following data format to be taken into consideration. chars,5,A,B,C,D,E,integers,1,2,3 chars,3,C,,,integers,2,10,11 I am assuming that the occursCountKind for the arrays is 'expression' and the occursCount refers to the integer field that precedes the array. In order to represent the empty strings on the second line it is essential to specify SSP as 'trailingEmpty' or 'never'. If we disallow the combination of ock='expression' and SSP='trailingEmpty' then how would this format be modelled? regards, Tim Kimber, Technical Lead for IBM Integration Bus Healthcare Pack Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 25/06/2014 16:25 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org I prefer choice (a) for two reasons * It is more restrictive and therefore more conservative (preserving freedom to change in future if needed) * If a user has a positional data format, you don't want them to even have to understand the concept of speculation in order to model their data. So choice (a) allows a simpler description that doesn't need to introduce the notion that the parser might be speculation. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Jun 25, 2014 at 5:20 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: 260 Positional and non-positional sequences (All) 10/6: Spec defines the above but also allows different occursCountKinds within the same sequence which may have different (implied) separatorSuppressionPolicy, which results in a sequence which is a mixture of both. Should this be allowed? If so what are the rules? Can certain combinations be disallowed? 17/6: IBM have discussed internally and will submit a proposal. In the spec we define Positional Sequence and Non-Positional Sequence: Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. A positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Typically the components of such a sequence have an initiator. Such sequences allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing is employed by the parser to identify each occurrence. A non-positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'anyEmpty'. The problem is that the setting of dfdl:separatorSuppressionPolicy is only examined for child elements with dfdl:occursCountKind 'implicit'. For other dfdl:occursCountKinds, there is the concept of an 'implied' dfdl:separatorSuppressionPolicy: When dfdl:occursCountKind is 'fixed' then ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'expression' ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'parsed' ... the implied behaviour is 'anyEmpty'. When dfdl:occursCountKind is 'stopValue' ...the implied behaviour is 'anyEmpty'. So if a Positional sequence as defined above contains children with dfdl:occursCountKind 'parsed' or 'stopValue' then surely it is no longer a Positional sequence. A solution to this is to prevent the appearance of certain values of dfdl:occursCountKind within a Positional sequence. However, precisely which values to outlaw is subject to interpretation of the phrase "Each occurrence in the sequence can be identified by its position in the data". Is this intended to mean: a) an observer of the raw data can identify an occurrence of an element in the sequence solely by counting separators => SDE if 'parsed', 'stopValue' or 'expression' ** appeared in a Positional sequence; ** Although 'expression' would appear to be like 'fixed' it actually breaks a) so must be included in the SDE list. or b) a parser does not have to speculate to identify an occurrence of an element in the sequence => SDE only if 'parsed' appeared in a Positional sequence. Note that it is possible to wrap a 'parsed' etc element in a local sequence or another element to avoid an SDE. But this could still be seen as a violation of a) if the separators of both are the same, as the observer can not count the separators. So should the rule be applied recursively, ie, a Positional sequence can not contain a non-Positional sequence unless the separators are different? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

do we allow indexing of non-array, non-optional
by Mike Beckerle 17 Nov '14

17 Nov '14

I don't recall whether we decided this matter or not. I would search for it myself, but I tried and failed to find anything. This is a hard topic to do searching on... no good keywords. If element 'e' has maxOccurs = 1, minOccurs = 1 (or neither are mentioned), is e[1] a valid expression? what about e[../some/pathExp/here]? If element 'e' has minOccurs= 0 maxOccurs = 1, is e[1] a valid expression, or do you access an optional element just by ../e ? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 4

Trimming of a text number that's all zeros when the number pattern has a sign char at the end
by Andrew Edwards 03 Nov '14

03 Nov '14

Hi all, I've hit an interesting case revolving around trimming and number patterns that doesn't seem quite sane to me. Consider an element with the following properties: textTrimKind='padChar' textNumberPadCharacter='0' textNumberPattern='0000+;0000-' So we have the sign character at the end of the representation. Now, imagine that the data being parsed is "0000+". The relevant rules from the DFDL specification are: Section 13.2 on textTrimKind When 'padChar', the element is trimmed of the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter, dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element. Section 13.6 on textNumberPadCharacter When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Section 13.6.1 Describes all of the pattern syntax. In our hypothetical case, the content region is not all zeros, as it ends in '+'. This means that the rule in section 13.6 does not apply and we only apply the rule in 13.2. This results in us trimming away all of the zeros and ending up with '+'. This then doesn't parse as a number. The problem seems to be that the rule in Section 13.6 doesn't take into account that the suffix of the pattern can result in text in the content region that isn't part of the digits of the number. Should the rule under section 13.56 be something more like this... When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, or the SimleContent region consists of a string of '0' characters followed by non-digit characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Thoughts? Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

elementFormDefault and the form attribute on element declarations
by Mike Beckerle 03 Nov '14

03 Nov '14

Neither the term "elementFormDefault" nor the term "unqualified" appear in the DFDL Spec. So we're silent about whether these are allowed or not in DFDL schemas. We should specify what of this XML Schema behavior is allowed, which I believe means add a line to each of the lists in section 5.1 of things used and not-used/reserved from XML Schema. FYI: Today the Daffodil implementation implements elementFormDefault on the xs:schema but not the form attribute on element declarations. We have tests that exercise elementFormDefault in qualified and unqualified and not specified at all (which defaults to "unqualified" behavior in XSD, so we're consistent with that.) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Action 274 - on qnames in path expressions in DFDL
by Mike Beckerle 03 Nov '14

03 Nov '14

I did some experiments with the XML validation built into eclipse, and the key constraint, which has paths (in the selector and field attributes) that refer to other parts of the XML infoset/document and so involve the names of elements. I also have been reading the XPath 2.0 specs including the formal semantics. To my interpretation XPath 2.0 is pretty explicit: An element or type QName consisting only of a local part NCName expands to the default element/type namespace and the local part. They don't qualify this in any way that I can find for the names used in path steps. The formal semantics is pretty hard to understand, but this detail seems to be swept under the rug. There would need to be specific treatment of this in the naming environment, but I cannot find this. There is discussion of name resolution in some of their judgements, but nothing I can see that creates an exception for names in path steps. (Called name tests) What I was looking for was language that specifies something about a non-prefixed name in a path step and that it is NOT augmented by the default namespace. I could find no such exception. But if I am right and this detail isn't there, i.e., If there is no such exception, then the implementation of XPath in eclipse (which is xerces I believe) is incorrect in the way it resolves names that appear in path steps. Here's how it works according to eclipse: In xpath expressions in the selectors and fields of unique and key constraints, xmlns (the default namespace) is not considered as implicitly augmenting unqualified names that appear in path steps. An identifier in a path step that has no prefix (e.g., 'foo' in the path './p1:bar/foo/p2:baz') is always interpreted as identifying either 1) a child element defined by a local element declaration with form "unqualified". or 2) this corner case: a child element defined by an element reference to a global element declaration but only when that global element declaration has no effective target namespace. (Use of 'effective' here rules out an element decl inside a schema document that itself has no target namespace but is included into a schema that has one. Such an element declaration effectively has a target namespace, so this corner case doesn't apply.) Case (2) here is a rather obscure corner case, but there nevertheless. In no case (that I could construct) does the name in a path step get augmented by a schema's default namespace (defined by xmlns="...whatever..."). The above is true for names without prefix appearing in path steps of expressions. The QNames that are used in an XML schema to reference a global declaration or definition (in the ref of an element ref, type attribute in a type reference, base in a simple type derivation, etc.) are different. Those names DO pickup a default namespace if one is defined and they have no explicit prefix themselves. This creates a corner case for these references (to global decls/defs) which is if a definition/declaration appears in no effective target namespace, then there is no way to reference it at all from a schema that has a default namespace definition surrounding the point of reference. There is no syntactic way to turn-off use of a default namespace. (The semantics of xpath allows for a "none" default namespace definition, but there is no xml syntax for expressing explicitly this "none". It is simply what you have if there is no prefix and no default namespace). You can change which namespace the default is, but there's no way to say explicitly "I meant this QName reference to be to something not in any namespace. Unless/until we can find language in some XML Schema spec indicating how path step names (name tests) are specifically resolved, then the above is just "how one system does it." But we may end up having to follow precedent here from implementations. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 4

clarification: fn:exactly-one function
by Mike Beckerle 03 Nov '14

03 Nov '14

According to the DFDL spec, fn:exactly-one returns "True if the provided sequence contains exactly one node/value," which suggests that the function would return false if the sequence contains > or < 1 node/value. The XPath description states that fn:exactly-one "returns $arg if it contains exactly one item. Otherwise, raises an error." It seems of value to have a true/false result, because something that raises an error like this can only be used as an assertion effectively. On the other hand, that's not what XPath does. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 2

Fw: Action 258 - data where inactive escape character is to be retained
by Steve Hanson 31 Oct '14

31 Oct '14

On last WG call there was still some concern about the names used. We will go with the proposal unless better names are suggested. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 31/10/2014 11:18 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 24/10/2014 13:22 Subject: Re: Fw: [DFDL-WG] Action 258 - data where inactive escape character is to be retained Proposal updated below to reflect recent WG calls - changes in red type. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 28/08/2014 15:47 Subject: Fw: [DFDL-WG] Action 258 - data where inactive escape character is to be retained Strawman proposal for action 258. 258 Consider allowing more flexible escapeCharacter schemes (Mike) 6/5: Motivated by example of an escape character which is active when in front of an in-scope delimiter, but not when in front of another character. 20/5: Can't model Mike's example with current facilities, but Mike's example is a generalisation of a particular MITRE example. Do we really need this? Jonathan to follow up. 3/6: Closed. Jonathan has provided the background to the MITRE example which was really about initiators and terminators. The generalised use case is perhaps speculative, so it was agreed not to change the DFDL spec to handle this unless a concrete use case emerges. 17/6: Re-opened. vCard 3.0 (http://tools.ietf.org/html/rfc2426) is an example of a format that exhibits the need for this. Need a proposal to handle this case, and which fits in with the existing extraEscapedCharacters and escapeEscapeCharacter property. Noted that using lengthKind 'pattern' is sometimes a way of working round this kind of thing. ... 15/7: No progress 22/7: Steve has started to write up a proposal. ... 26/8: No further progress 2/9: Strawman proposal sent by Steve for comment. Concern over name of new property. Review for next week. ... 16/9: No further progress 23/9: Discussed the naming issue, Steve to send revised proposal. Candidate for deferral to 1.1 ? 14/10: Agreed that it would be good to be able to handle vCard 3.0. With Steve to revise the proposal. New property dfdl:applyEscapeCharacter added. The description of dfdl:escapeKind is updated. No changes to dfdl:generateEscapeBlock but I've added it below by way of comparison. Property Name Description escapeKind Enum Valid values 'escapeCharacter', 'escapeBlock' The type of escape mechanism defined in the escape scheme When 'escapeCharacter': On unparsing a single character of the data is escaped by adding a dfdl:escapeCharacter or dfdl:escapeEscapeCharacter immediately before it. The characters to escape are determined by property dfdl:escapeCharacterPolicy. On parsing any in-scope terminating delimiter encountered in the data is not interpreted as such when it is immediately preceded by the dfdl:escapeCharacter (when not itself preceded by the dfdl:escapeEscapeCharacter). Occurrences of the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data as determined by property dfdl:escapeCharacterPolicy unless the dfdl:escapeCharacter is preceded by the dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter does not precede the dfdl:escapeCharacter, respectively. When 'escapeBlock': On unparsing the entire data are escaped by adding dfdl:escapeBlockStart to the beginning and dfdl:escapeBlockEnd to the end of the data. The data is either always escaped or escaped when needed as specified by dfdl:generateEscapeBlock. If the data is escaped and contains the dfdl:escapeBlockEnd then first character of each appearance of the dfdl:escapeBlockEnd is escaped by the dfdl:escapeEscapeCharacter. On parsing the dfdl:escapeBlockStart string must be the first characters in the (trimmed) data in order to activate the escape scheme. The dfdl:escapeBlockStart string is removed from the beginning of the data. Until a matching dfdl:escapeBlockEnd string (that is, one not preceded by the dfdl:escapeEscapeCharacter) is found in the data, any in-scope terminating delimiter encountered in the data is not interpreted as such, and any dfdl:escapeEscapeCharacters are removed when they precede an dfdl:escapeBlockEnd string. The matching dfdl:escapeBlockEnd string is removed from the data.. The matching dfdl:escapeBlockEnd does not have to be the last characters in the (trimmed) data in order to de-activate the escape scheme. A dfdl:escapeBlockStart occurring anywhere in the data other than the first characters has no significance. Annotation: dfdl:escapeScheme escapeCharacterPolicy Enum Valid values 'all', 'delimiters' Controls when escape characters are removed during parsing, and output during unparsing, when dfdl:escapeKind is 'escapeCharacter'. When 'all': During unparsing the following are escaped as described in dfdl:escapeKind when they are in the data. · Any in-scope terminating delimiter by escaping its first character. · dfdl:escapeCharacter (escaped by dfdl:escapeEscapeCharacter) · any dfdl:extraEscapedCharacters During parsing, occurrences of dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are interpreted and removed from the data as described in dfdl:escapeKind. When 'delimiters': During unparsing the following are escaped as described in dfdl:escapeKind when they are in the data. · Any in-scope terminating delimiter by escaping its first character. · dfdl:escapeCharacter (escaped by dfdl:escapeEscapeCharacter) During parsing, occurrences of dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are interpreted and removed from the data as described in dfdl:escapeKind, except that dfdl:escapeCharacter is only removed when it immediately precedes an in-scope terminating delimiter. Annotation: dfdl:escapeScheme generateEscapeBlock Enum Valid values 'always', 'whenNeeded' Controls when escaping is used on unparsing when dfdl:escapeKind is 'escapeBlock'. If 'always' then escaping always occurs as described in dfdl:escapeKind. If 'whenNeeded' then escaping occurs as described in dfdl:escapeKind when the data contains any of the following: · any in-scope terminating delimiter · dfdl:escapeBlockStart at the start of the data · any dfdl:extraEscapedCharacters Annotation: dfdl:escapeScheme Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2014 15:10 ----- From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/06/2014 17:28 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained I think I have come across a concrete example of this. It's from a standard called vCard 3.0 (RFC 2426 - http://tools.ietf.org/html/rfc2426) A backslash is used to a) escape itself; b) escape in-scope delimiters; c) indicate an embedded linefeed. The backslash would need removing for a) and b) but not for c). ESCAPED-CHAR = "\\" / "\;" / "\," / "\n" / "\N") ; \\ encodes \, \n or \N encodes newline ; \; encodes ;, \, encodes , Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 04/06/2014 08:53 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained WG call 3rd June: The generalised use case is perhaps speculative, so it was agreed not to change the DFDL spec to handle this unless a concrete use case emerges. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 20/05/2014 18:11 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained Sent by: dfdl-wg-bounces(a)ogf.org All, I’ll chime in with an observation and few more details on how Roger Costello got around a similar problem. Observation An escape block allows the escapeBlockEnd to be escaped with the escapeEscapeCharacter, while allowing the escapeEscapeCharacter itself to appear in the data without any special semantics as long as it is NOT followed by escapeBlockEnd. (From section 13.2.1 of the spec: “On parsing the dfdl:escapeBlockStart is removed from the beginning of the data and dfdl:escapeBlockEnd is removed from end of the data and any dfdl:escapeEscapeCharacters are removed when they precede a dfdl:escapeBlockEnd.”) This is really similar to the problem that Mike posed to the group below, where an escape character is sometimes an escape character but sometimes isn’t. While an escape block might not be suitable in all circumstances, the original problem that sparked Mike’s post was amenable to using an escape block, and that is how Roger Costello got around the problem. Some more details Roger Costello was using a quotation mark (“) as the initiator and terminator for quoted values in a data format. In this format, quotation marks can be escaped with a backslash (\); however, within a quoted string, the data could have a backslash as a normal data character (e.g. \n, representing two characters, not a single newline character). Roger posed his challenge to the Daffodil team, and then Mike created the example below to demonstrate the problem to the DFDL WG. In contrast to Mike’s example, Roger was having the problem with initiators and terminators, not a separator. At the time, we thought that that an escape block couldn’t be applied to the data format in question, so Mike may have altered the problem in order to prevent an escape block from clouding the issue as posed to the WG. It turns out, after more analysis, that an escape block could be used, and that solved the problem: escapeBlockStart=”"” escapeBlockEnd=”"” escapeEscapeCharacter=”\” Closing Observations In general, DFDL supports two different escape schemes with different behavior for the escape character. * When escapeKind=”escapeCharacter”, the escape character is always an escape character. * When escapeKind=”escapeBlock”, the escape character (escapeEscapeCharacter) is only an escape character in front of escapeBlockEnd. In this case, we were able to use an escape block to model the data format. While there may be a data format that has a character that is sometimes an escape character and sometimes isn’t, without a real world example, I echo Mike’s hesitance to add this feature to DFDL. HTH, Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Mike Beckerle Sent: Friday, May 02, 2014 2:47 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] data where inactive escape character is to be retained We have data that has ; as separator and has fields that look like this: abcd \; efgh That's a single field. The backslash escapes the ; so that the data is abcd ; efgh. This same data set also has abcd \n efgh Here the backslash precedes an ordinary non-delimiter. The data is supposed to be abcd \n efgh. That is, this data set requires the backslash to be retained in the data when it is not preceding the start of a delimiter. Am I missing something or is it impossible to model this? It would seem there needs to be a flag to indicate whether the escape characters that don't actually escape a delimiter are to be retained or not. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0