November 2012 - dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Clarification needed: pad/trim and delimited length
by Steve Hanson 12 Nov '12

12 Nov '12

Decided on last DFDL WG call to leave the behaviour as currently specified, as it is possible to code the current behaviour. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 26/10/2012 13:39 Subject: Re: [DFDL-WG] Clarification needed: pad/trim and delimited length Mike For escape blocks, the escape start/end character must be the first/last character in the text. The order of processing stated in the spec was an attempt to handle the situation where the escape start/end character was not the first/last character in the text due to padding. So for examples like: Variable length: "aaa,aaaa" ,"bbbbbbb" ,"ccccccc" Fixed length: "aaa,aaaa" "bbbbbbb" "ccccccc" >From an email discussion several years ago, in the answer to Alan's question "Should we only look for escapeStartString at the beginning of the data " Mike you said: "I'd prefer that we respect them anywhere, but canonical form when generated is at the beginning of the data. However, if we want to be more restrictive/conservative for v1.0 I'm fine with that." I tested IBM DFDL's implementation. The delimited example above (left justified) worked ok - the parser recognised the start quote and switched on escaping, correctly escaped the comma, then found the end quote and switched off escaping, then found the delimiter . With trimKind 'none' it issued an error to the effect that there was text between quote and next delimiter. With trimKind 'padChar' it worked ok and trimmed off the pad before going on to remove the quotes. However when the scenario was right-justified, it got it wrong, which I think is your point. The above order of processing leads to the following behaviour when trimKind is 'padChar'. Let's say I am exporting CSV data from Excel: Data: xx<sp><sp> Infoset: xx Data: "x,x<sp><sp>" Infoset: xx,<sp><sp> Explanation: The second data is same as the first except I have added in a comma, which causes Excel to escape with quotes in its normal way. The trimming takes place before escapes removed, so the first data loses the spaces while the second keeps them in the infoset. I don't think this is what a user would expect. (Note that Excel escapes the whole field). Seems to me there are competing requirements here, need to decide whether they all need to be satisfied by DFDL 1.0. Several possibilities to consider, here's some for starters: - Keep current rule but only allow it with left-justified fields - Keep current rule and trim 'as you go' rather than after extracting the data - Extend current rule and trim before and after escape character removal - Change rule to trim after escape character removal and handle leading/trailing text via delimiters - Change rule to trim after escape character removal and allow %WSP; etc in escape block start/end strings Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 25/10/2012 13:45 Subject: [DFDL-WG] Clarification needed: pad/trim and delimited length Sent by: dfdl-wg-bounces(a)ogf.org The spec says that pad characters are removed before escape scheme processing. However, in delimited context, I can't even determine the length of the field to trim off the padding unless I can do the escape scheme processing. This is either a Chicken-Egg, or the algorithm for parsing is substantially more complex due to padding. Comments? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

regex dialect considered too loose in spec
by Mike Beckerle 10 Nov '12

10 Nov '12

In a recent review meeting, a lot of discomfort was expressed by otherwise enthusiastic DFDL supporters, about our vagueness about regex-dialect accepted by DFDL. What regex dialect is used by each of the IBM implementations (C/Java)? Are they compatible? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

OGF DFDL WG Call Minutes 2012-11-05
by Steve Hanson 05 Nov '12

05 Nov '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12961?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2012-11-05
by Steve Hanson 05 Nov '12

05 Nov '12

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12961?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (5 Nov 16:00 GMT in IBM Hursley DE2K09)
by Steve Hanson 02 Nov '12

02 Nov '12

Rescheduled: OGF DFDL Working Group weekly call 05/11/2012 - (Repeats) Chair: Steve Hanson/UK/IBM Sent By: Steve Hanson/UK/IBM@IBMGB Location: IBM Hursley DE2K09 Steve Hanson/UK/IBM@IBMGB Steve Hanson has rescheduled this meeting. You have not yet responded. Repeats This entry repeats Required: mbeckerle.dfdl(a)gmail.com, Stephanie Fetzer/Charlotte/IBM@IBMUS, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: a.mumfor(a)radium.ncsc.mil, boyd.fletcher(a)us.army.mil, bradley.r.sexton(a)gmail.com, dfdl-wg(a)ogf.org, jcranford(a)mitre.org, marcos.bento(a)deimos.com.pt, remcgrat(a)illinois.edu OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0

Fw: clarification needed: assert evaluation order and arrays
by Steve Hanson 02 Nov '12

02 Nov '12

Hi Mike I have checked with our resident XPath expert, and you are correct. The self-axis refers to the current occurrence. Sorry to mislead you. I think we need a short discussion about what checkConstraints() can and can't check against. I'll add to agenda for next call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/11/2012 09:19 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 01/11/2012 18:34 Subject: Re: [DFDL-WG] clarification needed: assert evaluation order and arrays Do I understand this right? I have always thought that "." refers to the current occurrence (i.e., the [dfdl:occursIndex()] is implied if the current elemetn is an array). Hence, if I have an array of a complex type with children named a and b, then "./a" is a valid expression referring to the current instance's child a. Similarly, if I type "../bar", and my parent is an array, then ".." moves me up to the current instance within that array. Upward paths are unambiguous. Current node paths are unambiguous. Only downward paths that encounter an array would need indexing brackets. I think lots of things break if the above isn't the rule. E.g., you can't take an element and add minOccurs/maxOccurs to it to make it repeat without all the paths written on it being broken. If I am right, then there is no problem with making dfdl:checkConstraints(.) take the node, ask if it is an array, and if so compare its occursIndex to maxOccurs. I do take your point that the secondary check for minOccurs has to be expressed differently with a separate assert. The fact that dfdl:occursCount(path) takes a node-set-valued path is .... well really annoying, but inevitable. We need to watch for these exceptions. On Thu, Nov 1, 2012 at 1:39 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: You can't do that with one assert I'm afraid. The main issue is that a path location must always return exactly one item, so it's use within the context of an array must be in conjunction with a predicate, which must refer to a single instance. That's what dfdl:occursIndex() is for. To check that an array meets its bounds, you need a separate assert that uses the dfdl:count() function. You'd have to hard-code the min/max values, and place it on a containing object. <sequence> <annotation><appinfo...> <dfdl:assert>{ dfdl:occursCount(./foo) ge 5 and dfdl:occursCount(./foo) le 10 }</dfdl:assert> </appinfo></annotation> <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.[dfdl:occursIndex()]) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> </sequence> I think that means our definition of dfdl:checkConstraints() in the spec is wrong. Table 34 and Section 5.2 together imply that minOccurs and maxOccurs are used in checkConstraints. I think that simply doesn't work. It should only be using 'fixed' and the facets (I think 'default' is pointless as well). Worth noting that when a path is used in the context dfdl:occursCount(<path>) then it is not an error if more than one item is returned, and it should not be wrapped with fn:exactly-one(). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 01/11/2012 16:45 Subject: [DFDL-WG] clarification needed: assert evaluation order and arrays Sent by: dfdl-wg-bounces(a)ogf.org Question: If I write <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> I have two sources of constraints. One is the pattern, the other the min/max occurs. Does that one assertion calling dfdl:checkConstraints mean both will be checked? That is, one check occurring as each element is parsed, and the other at the end of the array? Assume I am not using any validation option, so the DFDL processor would not otherwise check the max/minOccurs because occursCountKind is parsed. Will the checkConstraints fail as soon as we parse the 11th element (is it checking the min/max occurs for each element occurrence as it is parsed), or do we parse as many as we can, and fail only when we check and find out that the entire array has 36 elements that were successfully parsed? What I would like the above to mean is this: 1) as each element occurrence is parsed we check the pattern and parse error (assertion failed) if there is no match. 2) Also after a successful parse of an occurrence, we check that the index is <=10, and parse-error (assertion failed) if not. 3) at the end of the array, we check that the number of occurrences is >= 5. If not we get a parse error (assertion failed). Comments? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2012-10-30 - resending
by Steve Hanson 02 Nov '12

02 Nov '12

** Resending as some maintenance on Redmine changed all the doc ids! ** Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/8145?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

clarification needed: assert evaluation order and arrays
by Mike Beckerle 01 Nov '12

01 Nov '12

Question: If I write <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> I have two sources of constraints. One is the pattern, the other the min/max occurs. Does that one assertion calling dfdl:checkConstraints mean both will be checked? That is, one check occurring as each element is parsed, and the other at the end of the array? Assume I am not using any validation option, so the DFDL processor would not otherwise check the max/minOccurs because occursCountKind is parsed. Will the checkConstraints fail as soon as we parse the 11th element (is it checking the min/max occurs for each element occurrence as it is parsed), or do we parse as many as we can, and fail only when we check and find out that the entire array has 36 elements that were successfully parsed? What I would like the above to mean is this: 1) as each element occurrence is parsed we check the pattern and parse error (assertion failed) if there is no match. 2) Also after a successful parse of an occurrence, we check that the index is <=10, and parse-error (assertion failed) if not. 3) at the end of the array, we check that the number of occurrences is >= 5. If not we get a parse error (assertion failed). Comments? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance)
by Mike Beckerle 01 Nov '12

01 Nov '12

I'll write this up like an errata, but this is for discussion of whether we believe this is clear and complete. ------------------------------------- *Glossary*: DFDL Statements are the annotation elements dfdl:assert, dfdl:discriminator, dfdl:setVariable, and dfdl:newVariableInstance. *Errata*: Locations where DFDL Statements are allowed to appear are extended to also include Global Element Declarations, and on Simple Type Definitions. *Errata*: Clarification about discriminators: Discriminators exclude Assertions even when combining across references. Beyond the stipulation that there can be only one dfdl:discriminator at any annotation point of the DFDL schema, there are further constraints. A single dfdl:discriminator annotation may appear on an element reference, or on the global element declaration it refers to, or on the simple type appearing immediately within or referenced from the global element declaration. But only one of those places. In addition, if a discriminator occupies one of those places, then no dfdl:assert annotations may appear in any of those locations. A dfdl:discriminator annotation may appear on a group reference or on the model group within the global group definition it refers to. But only one of those places, and similarly, if a discriminator appears in any of those places, then no dfdl:assert annotations may appear in any of those locations. *Errata*: Clarification about the execution order of DFDL Statements when they appear on an element reference or element declaration. DFDL Statement annotations for a given schema component are executed as follows: 1) all relevant DFDL statement annotations are gathered to form a single list which preserves schema-definition order. - For a simpleType definition, the DFDL statement annotations found immediately on it are kept in schema-definition order, and are appended to the end of a list of those from any base simpleType definition. - For an element declaration having simple type, the DFDL statement annotations found immediately on the declaration are appended to the end of the list of those from its simple type. - For an element reference, DFDL statement annotations found immediately on the element reference are appended to the end of the list of those from the global element declaration it references. 2) given the combined list, the annotations are executed as follows: 1. before any parsing of the element, a dfdl:discriminator with testKind="pattern" is executed. 2. if there is no discriminator, then before any parsing of the element, all dfdl:asserts (there could be several) with testKind="pattern" are executed in the order they appear in the list of DFDL statements. 3. The element itself is parsed, or its inputValueCalc property is evaluated to create its value. 4. all newVariableInstance annotations are executed and new variables are placed into scope for the duration of these remaining steps. The statements are executed in the order they appear in the list of DFDL statements. 5. all setVariable annotations are executed. The statements are executed in the order they appear in the list of DFDL statements. 6. if a discriminator is present it is executed 7. if no discriminator is present, then assert annotations can be present, and they are executed. If there are multiple assert annotations the statements are executed in the order they appear in the list of DFDL statements. If the element reference or local element declaration is an array, then this evaluation is repeated for each occurrence of the array. *Discussion: * The above allows the default expressions associated with any statement to refer to the value of the element itself as "." However, there's this anomaly of syntax where things don't seem right: <dfdl:newVariableInstance ref="foo:bar" default="{....some expression ...}"/> creates a new variable for the scope of the entity it annotates. It's clear what this means if this annotation is placed on a sequence or choice. For the children of that sequence/choice the new variable instance is in effect. On a simpleType or element having simple type, similarly it is clear (in that case it's a very local variable, just for the expressions in other newVariableInstance statemsnts, setVariableStatements, and discriminators and assertions). The rub: on an element declaration (or reference) when there is a complex type, its not clear. <element name="foo"> <annotation><appinfo ....> <dfdl:newVariableInstance......>  </appinfo></annotation> <complexType> <sequence> <annotation><appinfo ....> <dfdl:assert>... according to rules above this cannot refer to the newVariableInstance...</dfdl:assert> </appinfo></annotation> .... </sequence> </complexType> </element> The above rules say the newVariableInstance isn't evaluated until AFTER the element is parsed, so the assert down inside on the sequence will NOT see this new variable even tho textually, the newVariableInstance annotation looks like it would be in scope over the sequence. Possible ways to avoid this oddity without messing up evaluation order: allow newVariableInstance only on simpleType, element declarations and element references having simpleType, group references, and sequence/choice. Disallow them on element declarations of complex type or on element references to those. The schema author can always introduce an extra tier of sequence to provide the exact behavior they need, and this otherwise error-prone issue can be avoided.

4 12

proposal: DFDL needs additional function dfdl:characterCode
by Mike Beckerle 01 Nov '12

01 Nov '12

An important use case for DFDL is converting legacy data to/from XML. XML 1.0 disallows a bunch of string characters. If the data contains those characters, then the question arises of what to turn them into that both preserves information content, but also is legal in XML so that you can convert the DFDL infoset into XML without violating XML's constraints. The natural thing to do is create an element containing the character code of the illegal character, as an integer. E.g., character code U+0001 would become an element. Such as: <ccode>1</ccode>. This could be done using a hidden element that is a string, and the element ccode above would have an inputValueCalc that converts the offending character of that string into an integer. But we need a function dfdl:characterCode(str, pos) : int The arguments would be a string, and a position (base 1) within that string, and the return result would be the character code of the character in the string at that position. If pos is out of the bounds of the string (i.e., is negative, 0, or too large), then a processing error would occur. For unparsing the inverse function would also be needed: dfdl:character(intArg) : string. This would return a string containing one character whose codepoint is the intArg. Example Consider this data: 123<0>456<1>789<2>123l where <0> means just one character with codepoint 0, etc. In hex that would be 313233 00 343536 01 373839 02 313233 The best I can think of for modeling this while preserving all information would end up with XML looking like this: <nonXMLString> <fragment><stringData>123</stringData></fragment> <fragment><nonXMLChar><charCode>0</charCode></nonXMLChar></fragment> <fragment><stringData>456</stringData></fragment> <fragment><nonXMLChar><charCode>1</charCode></nonXMLChar></fragment> <fragment><stringData>789</stringData></fragment> <fragment><nonXMLChar><charCode>2</charCode></nonXMLChar></fragment> <fragment><stringData>123</stringData></fragment> </nonXMLString> So our nonXMLString is of a type which is array of fragment, a fragment is a choice of either (legal XML) stringData, or a nonXMLChar. The nonXMLChar has a child element because it will need to convert to from a string so will use inputValueCalc and outputValueCalc to do so, so it needs to be a sequence so that it can have the other hidden elements needed to pull this off. stringData would have lengthKind="pattern" and a pattern that allows any sequence of XML-allowed characters. nonXMLChar would have a hidden first child element of type string of explicit length 1 with an assertion that the string match a pattern that is any of the illegal characters (but just one of them). The charCode child element would inputValueCalc to get the character code of the character. For 8 bit encodings it would be ok as a table lookup in XPath, but for unicode..... we'd need a function that returns a character code. If you just have one embedded illegal character, like NUL, then you could just model it as a separator, which would simplify things considerably (and is possible in a someday XML 1.1 future since NUL is then the only disallowed character.) But for XML 1.0's illegal characters, we need to be able to convert to/from some non-string representation if we are to preserve information content. Hence we need these additional functions. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

4 10