- dfdl-wg - lists.ogf.org

do we allow indexing of non-array, non-optional
by Mike Beckerle 17 Nov '14

17 Nov '14

I don't recall whether we decided this matter or not. I would search for it myself, but I tried and failed to find anything. This is a hard topic to do searching on... no good keywords. If element 'e' has maxOccurs = 1, minOccurs = 1 (or neither are mentioned), is e[1] a valid expression? what about e[../some/pathExp/here]? If element 'e' has minOccurs= 0 maxOccurs = 1, is e[1] a valid expression, or do you access an optional element just by ../e ? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 4

Re: [DFDL-WG] referencing local element declarations from path expressions between two schema documents
by Mike Beckerle 17 Nov '14

17 Nov '14

Roger, I am struggling to defend the semantics you suggest below, because I cannot find anything in the XSD spec that specifically says how path steps are interpreted. In the absence of that, other members of the DFDL workgroup argue that path steps are just references to elements, and the default xmlns definition should work exactly as it works for any QName. So if the default namespace is defined, then a name with no prefix is by default interpreted relative to that definition. Hence, you have to do this: <xs:schema xmlns="BB"... > .... <xs:key name="myKey"> <xs:selector path="a:foo/bar" xmlns=""/> ... That is to say, if you wish to refer to locally declared elements with form unqualified, you cannot have a default namespace in effect, and the above provides the way you work around this, which is you explicitly reset the default namespace to "" meaning no-namespace. So is there anything you know of in the XSD spec that clarifies this, or is this only clarified based on precedent of how it works in Xerces XSD validator, but also Schematron, XSLT, etc. I have struggled to interpret the formal semantics document about XSD. My read of it did not find clarification of this point, but I confess to not understanding it sufficiently to be sure. Thanks in advance ...mike beckerle Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Wed, Sep 17, 2014 at 8:30 AM, Costello, Roger L. <costello(a)mitre.org> wrote: > Hi Mike, > > > > 1. In XSD only the global stuff can be referenced. So this is not allowed: > > > > <xs:element ref="bb:b1/c1"> > > > > 2. When using xs:key the XPath expression in @path must have a namespace > prefix even if the targetNamespace is the > > default namespace. Eg. *bk:Book. *If the schema had instead set > elementFormDefault="unqualified" then the XPath expressions would not be > namespace-qualified. > > > > 3. If you embed Schematron in an XSD file, during processing the > Schematron processor will extract out the Schematron and generate an XSLT > program. At that point you are in the world of XSLT and ordinary XPath > rules apply. So if BB.xsd has declared a global element {BB}b1 which > contains {}c1 then the XPath expression to reference c1 in an instance > document is this: bb:b1/c1 (where bb has been bound to BB in the > Schematron/XSLT code). It’s irrelevant to Schematron/XSLT whether the XML > Schemas have/don’t have a default namespace declaration. > > > > /Roger > > > > *From:* Mike Beckerle [mailto:mbeckerle.dfdl@gmail.com] > *Sent:* Wednesday, September 17, 2014 4:37 AM > *To:* dfdl-wg(a)ogf.org; Costello, Roger L. > *Subject:* referencing local element declarations from path expressions > between two schema documents > > > > > > I am looking for clarification about how path steps are to be interpreted > with respect to the issue of qualified/unqualified element names, and > default or no-default namespaces. > > If there is a precedent we can follow set by other tools which embed paths > into XSD (e.g., schematron?) or even just the selector/paths things in XSD > key/uniqueness constraints then I'd like to understand it. > > These are some of the conundrums: > > > Two Schema Files: A, and B > > Schema file A has target namespace AA, and binds the AA namespace to > prefix aa, and the BB namespace to prefix bb > > Schema file B has target namespace BB, and the same prefix bindings. > > Schema B has an element named b1 which has two local element declarations > inside it named c1 and c2. > > Schema A has an element named a1 which contains an element reference to > bb:b1. > > a1 has a DFDL annotation (e.g., dfdl:assert) which has a path expression > in it. this path contains bb:b1/c1 > > When is the path correct? > > 1a) when schema B has elementFormDefault='unqualified' (that's the > default), and schema A does NOT have a default namespace. Why? the default > namespace - if it existed for schema A - would implicitly qualify the c1 > step of the path, but since c1 is a local element in schema B with element > form unqualified, it does not require (and cannot have) a namespace > specified in path steps for c1. > > 1b) when schema B has elementFormDefault='qualified', and schema A has > xmlns='BB'. Why? Because in this case the path bb:b1/c1 is equivalent to > bb:b1/bb:c1, and since c1 will have a namespace-qualified name in schema B, > that path will properly describe the path to the c1 element. > > 1c) Variation: when schema B has no target namespace (which I believe > makes elementFormDefault irrelevant), and schema A has no default > namespace. Because then c1 can't be qualfied as it lives in no namespace, > and schema A has no implied qualifiers. > > > > When is the path incorrect? > > 2a) when schema A has xmlns="BB". Why? Because the default namespace means > the bb:b1/c1 is equivalent to {BB}b1/{BB}c1, but there is no {BB}/c1 since > c1 is just a local element declaration, and B has elementFormDefault > unqualified. > > 2b) when schema B has elementFormDefault='qualified', and schema A has no > default namespace. In that case the step c1 needs a namespace qualifier > either by default, or explicitly via a prefix, but it doesn't have one. > > So I am interested in opinions on whether the above analysis is correct or > not, and especially in precedent for how this stuff works in other tools. > > I have made a big assumption above which is that if xmlns="BB", then any > path step inside something carrying that default binding is interpreted as > qualified by that namespace. I picked this because it seemed reasonable, > and it seems one can always modify one's schema to properly address > something in another schema without having to change that other schema. > I.e., it seems to have good composition properties. > > Thanks for any thoughts. > > > > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > www.tresys.com > > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the OGF Intellectual Property Policy > <http://www.ogf.org/About/abt_policies.php> > > >

1 0

OGF DFDL WG Call Agenda 2014-11-18
by Steve Hanson 17 Nov '14

17 Nov '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13374?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Minutes 2014-11-11
by Steve Hanson 12 Nov '14

12 Nov '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13373?download= ** Updated DFDL 1.0 specification published at http://www.ogf.org/documents/GFD.207.pdf. ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] elementFormDefault and the form attribute on element declarations
by Steve Hanson 12 Nov '14

12 Nov '14

WG agreed that elementFormDefault and form attributes will be mentioned explicitly in section 5.1. http://redmine.ogf.org/issues/241 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 03/11/2014 14:57 Subject: Re: [DFDL-WG] elementFormDefault and the form attribute on element declarations Section 5.1 of the spec has 'Standard XSDL namespace management' in its list of what a DFDL schema consists of. Section 5.1 does not go on explicitly to exclude 'elementFormDefault' attribute or 'form' attribute, so my reading is that both these are therefore allowed. We could expand what is meant by 'Standard XSDL namespace management'. But if we do that we should make sure that we do so in any other areas that are a bit vague. For example, we disallow complex type derivation and substitution groups so does this imply that 'final' and 'block' on complex types and elements are prohibited regardless of setting? For example, we disallow identity constraints but does that mean that 'id' can never be used on schema objects? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 15/10/2014 23:25 Subject: [DFDL-WG] elementFormDefault and the form attribute on element declarations Sent by: dfdl-wg-bounces(a)ogf.org Neither the term "elementFormDefault" nor the term "unqualified" appear in the DFDL Spec. So we're silent about whether these are allowed or not in DFDL schemas. We should specify what of this XML Schema behavior is allowed, which I believe means add a line to each of the lists in section 5.1 of things used and not-used/reserved from XML Schema. FYI: Today the Daffodil implementation implements elementFormDefault on the xs:schema but not the form attribute on element declarations. We have tests that exercise elementFormDefault in qualified and unqualified and not specified at all (which defaults to "unqualified" behavior in XSD, so we're consistent with that.) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: minor/typographical - fn prefix namespace not in spec
by Steve Hanson 12 Nov '14

12 Nov '14

WG agreed that the namespace URI needs adding to section 23 and linking to the fn: prefix. http://redmine.ogf.org/issues/240 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 12/11/2014 12:04 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 28/10/2014 17:41 Subject: [DFDL-WG] minor/typographical - fn prefix namespace not in spec Sent by: dfdl-wg-bounces(a)ogf.org The URI corresponding to the xpath functions namespace http://www.w3.org/2005/xpath-functions is not in the DFDL spec. I am assuming where we are borrowing the exact XPath function behavior the namespace is the xpath functions URI, not some DFDL-specific URI. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] clarification: fn:exactly-one function
by Steve Hanson 12 Nov '14

12 Nov '14

WG agreed that fn:exactly-one() would have its description updated to match XPath 2.0 spec. Perhaps also worth updating fn:exists() and fn:empty() so their descriptions match XPath 2.0 spec. http://redmine.ogf.org/issues/239 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 03/11/2014 14:08 Subject: Re: [DFDL-WG] clarification: fn:exactly-one function The main purpose envisaged for fn:exactly-one() was as a guard around off-the-shelf XPath processors. In that scenario, throwing an error is probably required, as what would you do otherwise? If you wanted to return a true/false value could you just use fn:count($arg) eq 1 instead of fn:exactly-one() ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 30/10/2014 10:00 Subject: [DFDL-WG] clarification: fn:exactly-one function Sent by: dfdl-wg-bounces(a)ogf.org According to the DFDL spec, fn:exactly-one returns "True if the provided sequence contains exactly one node/value," which suggests that the function would return false if the sequence contains > or < 1 node/value. The XPath description states that fn:exactly-one "returns $arg if it contains exactly one item. Otherwise, raises an error." It seems of value to have a true/false result, because something that raises an error like this can only be used as an assertion effectively. On the other hand, that's not what XPath does. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end
by Steve Hanson 12 Nov '14

12 Nov '14

WG accepted the new wording. Tracked by http://redmine.ogf.org/issues/238. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 11/11/2014 16:53 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end One further modification as we are missing a case when there is nothing but zeroes in the content region... When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, or in all of the SimpleContent region being trimmed, then the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 18:10 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Hi Steve Yep - I agree with your new definition of the rule taking into account justification-independence and quoted text. I'm not sure I like the nested if-else in the new text though, as it took a couple of reads to understand. How about the text below? It splits out the second 'if/else', which I find easier to understand. When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Cheers, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Steve Hanson/UK/IBM To: Andrew Edwards/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 15:39 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Andy I agree that the existing words do not cover all the scenarios. Your proposed words are on the right track but only cover left trimming a right justified text number. We need something that is independent of justification and can handle patterns where there is quoted text as well as signs. One can envisage some bizarre scenarios. Eg, Text number pattern is "#0'000'" - an attempt to divide by 1000 using the pattern. DFDL parser would trim everything except 1 zero which would not match the pattern which expects at least 3 zeros. Trimming happens before pattern is looked at so I don't think we could cater for this (if we even wanted to). Perhaps we should say: When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then if the SimpleContent region is trimmed so that the removal of a '0' character leaves the next character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. That means that "000,000,123" would end up as "0,000,123" instead of ",000,123" today and "0000.025" would end up as "0.025" instead of ".025" today but I think that is good. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM@IBMGB To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 30/10/2014 16:43 Subject: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Sent by: dfdl-wg-bounces(a)ogf.org Hi all, I've hit an interesting case revolving around trimming and number patterns that doesn't seem quite sane to me. Consider an element with the following properties: textTrimKind='padChar' textNumberPadCharacter='0' textNumberPattern='0000+;0000-' So we have the sign character at the end of the representation. Now, imagine that the data being parsed is "0000+". The relevant rules from the DFDL specification are: Section 13.2 on textTrimKind When 'padChar', the element is trimmed of the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter, dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element. Section 13.6 on textNumberPadCharacter When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Section 13.6.1 Describes all of the pattern syntax. In our hypothetical case, the content region is not all zeros, as it ends in '+'. This means that the rule in section 13.6 does not apply and we only apply the rule in 13.2. This results in us trimming away all of the zeros and ending up with '+'. This then doesn't parse as a number. The problem seems to be that the rule in Section 13.6 doesn't take into account that the suffix of the pattern can result in text in the content region that isn't part of the digits of the number. Should the rule under section 13.56 be something more like this... When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, or the SimleContent region consists of a string of '0' characters followed by non-digit characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Thoughts? Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Fw: Action 258 - data where inactive escape character is to be retained
by Steve Hanson 12 Nov '14

12 Nov '14

Proposal below accepted. Issue raised: http://redmine.ogf.org/issues/237 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 31/10/2014 11:20 Subject: Fw: [DFDL-WG] Action 258 - data where inactive escape character is to be retained On last WG call there was still some concern about the names used. We will go with the proposal unless better names are suggested. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 31/10/2014 11:18 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 24/10/2014 13:22 Subject: Re: Fw: [DFDL-WG] Action 258 - data where inactive escape character is to be retained Proposal updated below to reflect recent WG calls - changes in red type. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 28/08/2014 15:47 Subject: Fw: [DFDL-WG] Action 258 - data where inactive escape character is to be retained Strawman proposal for action 258. 258 Consider allowing more flexible escapeCharacter schemes (Mike) 6/5: Motivated by example of an escape character which is active when in front of an in-scope delimiter, but not when in front of another character. 20/5: Can't model Mike's example with current facilities, but Mike's example is a generalisation of a particular MITRE example. Do we really need this? Jonathan to follow up. 3/6: Closed. Jonathan has provided the background to the MITRE example which was really about initiators and terminators. The generalised use case is perhaps speculative, so it was agreed not to change the DFDL spec to handle this unless a concrete use case emerges. 17/6: Re-opened. vCard 3.0 (http://tools.ietf.org/html/rfc2426) is an example of a format that exhibits the need for this. Need a proposal to handle this case, and which fits in with the existing extraEscapedCharacters and escapeEscapeCharacter property. Noted that using lengthKind 'pattern' is sometimes a way of working round this kind of thing. ... 15/7: No progress 22/7: Steve has started to write up a proposal. ... 26/8: No further progress 2/9: Strawman proposal sent by Steve for comment. Concern over name of new property. Review for next week. ... 16/9: No further progress 23/9: Discussed the naming issue, Steve to send revised proposal. Candidate for deferral to 1.1 ? 14/10: Agreed that it would be good to be able to handle vCard 3.0. With Steve to revise the proposal. New property dfdl:applyEscapeCharacter added. The description of dfdl:escapeKind is updated. No changes to dfdl:generateEscapeBlock but I've added it below by way of comparison. Property Name Description escapeKind Enum Valid values 'escapeCharacter', 'escapeBlock' The type of escape mechanism defined in the escape scheme When 'escapeCharacter': On unparsing a single character of the data is escaped by adding a dfdl:escapeCharacter or dfdl:escapeEscapeCharacter immediately before it. The characters to escape are determined by property dfdl:escapeCharacterPolicy. On parsing any in-scope terminating delimiter encountered in the data is not interpreted as such when it is immediately preceded by the dfdl:escapeCharacter (when not itself preceded by the dfdl:escapeEscapeCharacter). Occurrences of the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data as determined by property dfdl:escapeCharacterPolicy unless the dfdl:escapeCharacter is preceded by the dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter does not precede the dfdl:escapeCharacter, respectively. When 'escapeBlock': On unparsing the entire data are escaped by adding dfdl:escapeBlockStart to the beginning and dfdl:escapeBlockEnd to the end of the data. The data is either always escaped or escaped when needed as specified by dfdl:generateEscapeBlock. If the data is escaped and contains the dfdl:escapeBlockEnd then first character of each appearance of the dfdl:escapeBlockEnd is escaped by the dfdl:escapeEscapeCharacter. On parsing the dfdl:escapeBlockStart string must be the first characters in the (trimmed) data in order to activate the escape scheme. The dfdl:escapeBlockStart string is removed from the beginning of the data. Until a matching dfdl:escapeBlockEnd string (that is, one not preceded by the dfdl:escapeEscapeCharacter) is found in the data, any in-scope terminating delimiter encountered in the data is not interpreted as such, and any dfdl:escapeEscapeCharacters are removed when they precede an dfdl:escapeBlockEnd string. The matching dfdl:escapeBlockEnd string is removed from the data.. The matching dfdl:escapeBlockEnd does not have to be the last characters in the (trimmed) data in order to de-activate the escape scheme. A dfdl:escapeBlockStart occurring anywhere in the data other than the first characters has no significance. Annotation: dfdl:escapeScheme escapeCharacterPolicy Enum Valid values 'all', 'delimiters' Controls when escape characters are removed during parsing, and output during unparsing, when dfdl:escapeKind is 'escapeCharacter'. When 'all': During unparsing the following are escaped as described in dfdl:escapeKind when they are in the data. · Any in-scope terminating delimiter by escaping its first character. · dfdl:escapeCharacter (escaped by dfdl:escapeEscapeCharacter) · any dfdl:extraEscapedCharacters During parsing, occurrences of dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are interpreted and removed from the data as described in dfdl:escapeKind. When 'delimiters': During unparsing the following are escaped as described in dfdl:escapeKind when they are in the data. · Any in-scope terminating delimiter by escaping its first character. · dfdl:escapeCharacter (escaped by dfdl:escapeEscapeCharacter) During parsing, occurrences of dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are interpreted and removed from the data as described in dfdl:escapeKind, except that dfdl:escapeCharacter is only removed when it immediately precedes an in-scope terminating delimiter. Annotation: dfdl:escapeScheme generateEscapeBlock Enum Valid values 'always', 'whenNeeded' Controls when escaping is used on unparsing when dfdl:escapeKind is 'escapeBlock'. If 'always' then escaping always occurs as described in dfdl:escapeKind. If 'whenNeeded' then escaping occurs as described in dfdl:escapeKind when the data contains any of the following: · any in-scope terminating delimiter · dfdl:escapeBlockStart at the start of the data · any dfdl:extraEscapedCharacters Annotation: dfdl:escapeScheme Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2014 15:10 ----- From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/06/2014 17:28 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained I think I have come across a concrete example of this. It's from a standard called vCard 3.0 (RFC 2426 - http://tools.ietf.org/html/rfc2426) A backslash is used to a) escape itself; b) escape in-scope delimiters; c) indicate an embedded linefeed. The backslash would need removing for a) and b) but not for c). ESCAPED-CHAR = "\\" / "\;" / "\," / "\n" / "\N") ; \\ encodes \, \n or \N encodes newline ; \; encodes ;, \, encodes , Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 04/06/2014 08:53 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained WG call 3rd June: The generalised use case is perhaps speculative, so it was agreed not to change the DFDL spec to handle this unless a concrete use case emerges. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 20/05/2014 18:11 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained Sent by: dfdl-wg-bounces(a)ogf.org All, I’ll chime in with an observation and few more details on how Roger Costello got around a similar problem. Observation An escape block allows the escapeBlockEnd to be escaped with the escapeEscapeCharacter, while allowing the escapeEscapeCharacter itself to appear in the data without any special semantics as long as it is NOT followed by escapeBlockEnd. (From section 13.2.1 of the spec: “On parsing the dfdl:escapeBlockStart is removed from the beginning of the data and dfdl:escapeBlockEnd is removed from end of the data and any dfdl:escapeEscapeCharacters are removed when they precede a dfdl:escapeBlockEnd.”) This is really similar to the problem that Mike posed to the group below, where an escape character is sometimes an escape character but sometimes isn’t. While an escape block might not be suitable in all circumstances, the original problem that sparked Mike’s post was amenable to using an escape block, and that is how Roger Costello got around the problem. Some more details Roger Costello was using a quotation mark (“) as the initiator and terminator for quoted values in a data format. In this format, quotation marks can be escaped with a backslash (\); however, within a quoted string, the data could have a backslash as a normal data character (e.g. \n, representing two characters, not a single newline character). Roger posed his challenge to the Daffodil team, and then Mike created the example below to demonstrate the problem to the DFDL WG. In contrast to Mike’s example, Roger was having the problem with initiators and terminators, not a separator. At the time, we thought that that an escape block couldn’t be applied to the data format in question, so Mike may have altered the problem in order to prevent an escape block from clouding the issue as posed to the WG. It turns out, after more analysis, that an escape block could be used, and that solved the problem: escapeBlockStart=”"” escapeBlockEnd=”"” escapeEscapeCharacter=”\” Closing Observations In general, DFDL supports two different escape schemes with different behavior for the escape character. * When escapeKind=”escapeCharacter”, the escape character is always an escape character. * When escapeKind=”escapeBlock”, the escape character (escapeEscapeCharacter) is only an escape character in front of escapeBlockEnd. In this case, we were able to use an escape block to model the data format. While there may be a data format that has a character that is sometimes an escape character and sometimes isn’t, without a real world example, I echo Mike’s hesitance to add this feature to DFDL. HTH, Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Mike Beckerle Sent: Friday, May 02, 2014 2:47 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] data where inactive escape character is to be retained We have data that has ; as separator and has fields that look like this: abcd \; efgh That's a single field. The backslash escapes the ; so that the data is abcd ; efgh. This same data set also has abcd \n efgh Here the backslash precedes an ordinary non-delimiter. The data is supposed to be abcd \n efgh. That is, this data set requires the backslash to be retained in the data when it is not preceding the start of a delimiter. Am I missing something or is it impossible to model this? It would seem there needs to be a flag to indicate whether the escape characters that don't actually escape a delimiter are to be retained or not. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end
by Andrew Edwards 11 Nov '14

11 Nov '14

One further modification as we are missing a case when there is nothing but zeroes in the content region... When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, or in all of the SimpleContent region being trimmed, then the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 18:10 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Hi Steve Yep - I agree with your new definition of the rule taking into account justification-independence and quoted text. I'm not sure I like the nested if-else in the new text though, as it took a couple of reads to understand. How about the text below? It splits out the second 'if/else', which I find easier to understand. When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Cheers, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Steve Hanson/UK/IBM To: Andrew Edwards/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 15:39 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Andy I agree that the existing words do not cover all the scenarios. Your proposed words are on the right track but only cover left trimming a right justified text number. We need something that is independent of justification and can handle patterns where there is quoted text as well as signs. One can envisage some bizarre scenarios. Eg, Text number pattern is "#0'000'" - an attempt to divide by 1000 using the pattern. DFDL parser would trim everything except 1 zero which would not match the pattern which expects at least 3 zeros. Trimming happens before pattern is looked at so I don't think we could cater for this (if we even wanted to). Perhaps we should say: When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then if the SimpleContent region is trimmed so that the removal of a '0' character leaves the next character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. That means that "000,000,123" would end up as "0,000,123" instead of ",000,123" today and "0000.025" would end up as "0.025" instead of ".025" today but I think that is good. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM@IBMGB To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 30/10/2014 16:43 Subject: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Sent by: dfdl-wg-bounces(a)ogf.org Hi all, I've hit an interesting case revolving around trimming and number patterns that doesn't seem quite sane to me. Consider an element with the following properties: textTrimKind='padChar' textNumberPadCharacter='0' textNumberPattern='0000+;0000-' So we have the sign character at the end of the representation. Now, imagine that the data being parsed is "0000+". The relevant rules from the DFDL specification are: Section 13.2 on textTrimKind When 'padChar', the element is trimmed of the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter, dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element. Section 13.6 on textNumberPadCharacter When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Section 13.6.1 Describes all of the pattern syntax. In our hypothetical case, the content region is not all zeros, as it ends in '+'. This means that the rule in section 13.6 does not apply and we only apply the rule in 13.2. This results in us trimming away all of the zeros and ending up with '+'. This then doesn't parse as a number. The problem seems to be that the rule in Section 13.6 doesn't take into account that the suffix of the pattern can result in text in the content region that isn't part of the digits of the number. Should the rule under section 13.56 be something more like this... When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, or the SimleContent region consists of a string of '0' characters followed by non-digit characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Thoughts? Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0