- dfdl-wg - lists.ogf.org

Fw: clarification needed: assert evaluation order and arrays
by Steve Hanson 14 Nov '12

14 Nov '12

Agreed on DFDL WG call that checkConstraints() will only look at facets and the fixed attribute. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 14/11/2012 11:04 ----- From: Steve Hanson/UK/IBM To: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com>, Cc: Tim Kimber/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 02/11/2012 14:51 Subject: Fw: [DFDL-WG] clarification needed: assert evaluation order and arrays Hi Mike I have checked with our resident XPath expert, and you are correct. The self-axis refers to the current occurrence. Sorry to mislead you. I think we need a short discussion about what checkConstraints() can and can't check against. I'll add to agenda for next call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/11/2012 09:19 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 01/11/2012 18:34 Subject: Re: [DFDL-WG] clarification needed: assert evaluation order and arrays Do I understand this right? I have always thought that "." refers to the current occurrence (i.e., the [dfdl:occursIndex()] is implied if the current elemetn is an array). Hence, if I have an array of a complex type with children named a and b, then "./a" is a valid expression referring to the current instance's child a. Similarly, if I type "../bar", and my parent is an array, then ".." moves me up to the current instance within that array. Upward paths are unambiguous. Current node paths are unambiguous. Only downward paths that encounter an array would need indexing brackets. I think lots of things break if the above isn't the rule. E.g., you can't take an element and add minOccurs/maxOccurs to it to make it repeat without all the paths written on it being broken. If I am right, then there is no problem with making dfdl:checkConstraints(.) take the node, ask if it is an array, and if so compare its occursIndex to maxOccurs. I do take your point that the secondary check for minOccurs has to be expressed differently with a separate assert. The fact that dfdl:occursCount(path) takes a node-set-valued path is .... well really annoying, but inevitable. We need to watch for these exceptions. On Thu, Nov 1, 2012 at 1:39 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: You can't do that with one assert I'm afraid. The main issue is that a path location must always return exactly one item, so it's use within the context of an array must be in conjunction with a predicate, which must refer to a single instance. That's what dfdl:occursIndex() is for. To check that an array meets its bounds, you need a separate assert that uses the dfdl:count() function. You'd have to hard-code the min/max values, and place it on a containing object. <sequence> <annotation><appinfo...> <dfdl:assert>{ dfdl:occursCount(./foo) ge 5 and dfdl:occursCount(./foo) le 10 }</dfdl:assert> </appinfo></annotation> <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.[dfdl:occursIndex()]) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> </sequence> I think that means our definition of dfdl:checkConstraints() in the spec is wrong. Table 34 and Section 5.2 together imply that minOccurs and maxOccurs are used in checkConstraints. I think that simply doesn't work. It should only be using 'fixed' and the facets (I think 'default' is pointless as well). Worth noting that when a path is used in the context dfdl:occursCount(<path>) then it is not an error if more than one item is returned, and it should not be wrapped with fn:exactly-one(). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 01/11/2012 16:45 Subject: [DFDL-WG] clarification needed: assert evaluation order and arrays Sent by: dfdl-wg-bounces(a)ogf.org Question: If I write <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> I have two sources of constraints. One is the pattern, the other the min/max occurs. Does that one assertion calling dfdl:checkConstraints mean both will be checked? That is, one check occurring as each element is parsed, and the other at the end of the array? Assume I am not using any validation option, so the DFDL processor would not otherwise check the max/minOccurs because occursCountKind is parsed. Will the checkConstraints fail as soon as we parse the 11th element (is it checking the min/max occurs for each element occurrence as it is parsed), or do we parse as many as we can, and fail only when we check and find out that the entire array has 36 elements that were successfully parsed? What I would like the above to mean is this: 1) as each element occurrence is parsed we check the pattern and parse error (assertion failed) if there is no match. 2) Also after a successful parse of an occurrence, we check that the index is <=10, and parse-error (assertion failed) if not. 3) at the end of the array, we check that the number of occurrences is >= 5. If not we get a parse error (assertion failed). Comments? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Expressions too restrictive? Example from IPFIX format
by Mike Beckerle 13 Nov '12

13 Nov '12

I'm looking at RFC 5101 and RFC 5102 These describe a dense binary file format for observing network flows. The application is related to network security. This format has a 'meta' structure to it that I'm not sure how to deal with in DFDL currently. Here's the problem as succinctly as I can make it. One configures some network information capture tools to capture some information. This is flexible, so what is captured can be quite variable. The resulting data stream contains first sets of templates, then sets of data records, then more templates, then more records, and so on. The templates each are identified by an ID number which is an integer from 256 to 65534. A template then contains a count of how many fields are in the template, and then a field descriptor for each field, which includes the length of the field in octets. Each data record set begins with the ID of its template, then a total length in octets for the set, then the records, which are just field, field, field, each as described by the associated template with no record separators or anything. Now, I can model a template in DFDL as an array of field descriptors. I can also model a data record set as a templateID and an array of data records, and each data record as an array of fields, where the number of field occurrences is given in the template, and each field is a byte array of occurs count given in the corresponding template's field descriptor for that occursIndex. The one thing I can't figure out how to do is to create an XPath to the right template given the template ID in the data record set header. The problem, is that the TemplateID is not an array index. The templates might have arbitrary ID numbers. They might not be say, 256, 257, etc. in order. They could be scattered, etc. The standard only requires that they are unique IDs. So the set of templates is truly a set with these identifiers. So I need a way to write an XPath that would choose the template from the set, whose ID matches a particular integer value from the data record set's header. Right now I think our XPath subset doesn't allow this. We can only index arrays with integers, and we have no searching capability that processes a set of nodes to identify a node having any particular value or characteristic. So: are we being too strict here. Should we allow somewhat more complex predicates? such as { ...../template[idField eq $templateID] } (For speed reasons, I might not do that lookup each time I parse a data record. I might pre-fill an array of 65535 structures by way of inputValueCalc so that I can actually use the template IDs as indexes into that array. But either way I need the select from matching set capability.) In general, templates can be interleaved in IPFix in that the requirement is just that they are transmitted before data record sets that reference them. So in general, an application that reads IPFix data cannot say, first scrape off all the templates and generate a schema for them, and then use that schema to parse. The late availability of the templates is something that is inherent in the format. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

OGF DFDL WG Call Agenda 2012-11-13
by Steve Hanson 12 Nov '12

12 Nov '12

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12965?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Clarification needed: pad/trim and delimited length
by Steve Hanson 12 Nov '12

12 Nov '12

Decided on last DFDL WG call to leave the behaviour as currently specified, as it is possible to code the current behaviour. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 26/10/2012 13:39 Subject: Re: [DFDL-WG] Clarification needed: pad/trim and delimited length Mike For escape blocks, the escape start/end character must be the first/last character in the text. The order of processing stated in the spec was an attempt to handle the situation where the escape start/end character was not the first/last character in the text due to padding. So for examples like: Variable length: "aaa,aaaa" ,"bbbbbbb" ,"ccccccc" Fixed length: "aaa,aaaa" "bbbbbbb" "ccccccc" >From an email discussion several years ago, in the answer to Alan's question "Should we only look for escapeStartString at the beginning of the data " Mike you said: "I'd prefer that we respect them anywhere, but canonical form when generated is at the beginning of the data. However, if we want to be more restrictive/conservative for v1.0 I'm fine with that." I tested IBM DFDL's implementation. The delimited example above (left justified) worked ok - the parser recognised the start quote and switched on escaping, correctly escaped the comma, then found the end quote and switched off escaping, then found the delimiter . With trimKind 'none' it issued an error to the effect that there was text between quote and next delimiter. With trimKind 'padChar' it worked ok and trimmed off the pad before going on to remove the quotes. However when the scenario was right-justified, it got it wrong, which I think is your point. The above order of processing leads to the following behaviour when trimKind is 'padChar'. Let's say I am exporting CSV data from Excel: Data: xx<sp><sp> Infoset: xx Data: "x,x<sp><sp>" Infoset: xx,<sp><sp> Explanation: The second data is same as the first except I have added in a comma, which causes Excel to escape with quotes in its normal way. The trimming takes place before escapes removed, so the first data loses the spaces while the second keeps them in the infoset. I don't think this is what a user would expect. (Note that Excel escapes the whole field). Seems to me there are competing requirements here, need to decide whether they all need to be satisfied by DFDL 1.0. Several possibilities to consider, here's some for starters: - Keep current rule but only allow it with left-justified fields - Keep current rule and trim 'as you go' rather than after extracting the data - Extend current rule and trim before and after escape character removal - Change rule to trim after escape character removal and handle leading/trailing text via delimiters - Change rule to trim after escape character removal and allow %WSP; etc in escape block start/end strings Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 25/10/2012 13:45 Subject: [DFDL-WG] Clarification needed: pad/trim and delimited length Sent by: dfdl-wg-bounces(a)ogf.org The spec says that pad characters are removed before escape scheme processing. However, in delimited context, I can't even determine the length of the field to trim off the padding unless I can do the escape scheme processing. This is either a Chicken-Egg, or the algorithm for parsing is substantially more complex due to padding. Comments? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

regex dialect considered too loose in spec
by Mike Beckerle 10 Nov '12

10 Nov '12

In a recent review meeting, a lot of discomfort was expressed by otherwise enthusiastic DFDL supporters, about our vagueness about regex-dialect accepted by DFDL. What regex dialect is used by each of the IBM implementations (C/Java)? Are they compatible? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

OGF DFDL WG Call Minutes 2012-11-05
by Steve Hanson 05 Nov '12

05 Nov '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12961?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2012-11-05
by Steve Hanson 05 Nov '12

05 Nov '12

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12961?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (5 Nov 16:00 GMT in IBM Hursley DE2K09)
by Steve Hanson 02 Nov '12

02 Nov '12

Rescheduled: OGF DFDL Working Group weekly call 05/11/2012 - (Repeats) Chair: Steve Hanson/UK/IBM Sent By: Steve Hanson/UK/IBM@IBMGB Location: IBM Hursley DE2K09 Steve Hanson/UK/IBM@IBMGB Steve Hanson has rescheduled this meeting. You have not yet responded. Repeats This entry repeats Required: mbeckerle.dfdl(a)gmail.com, Stephanie Fetzer/Charlotte/IBM@IBMUS, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: a.mumfor(a)radium.ncsc.mil, boyd.fletcher(a)us.army.mil, bradley.r.sexton(a)gmail.com, dfdl-wg(a)ogf.org, jcranford(a)mitre.org, marcos.bento(a)deimos.com.pt, remcgrat(a)illinois.edu OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0

Fw: clarification needed: assert evaluation order and arrays
by Steve Hanson 02 Nov '12

02 Nov '12

Hi Mike I have checked with our resident XPath expert, and you are correct. The self-axis refers to the current occurrence. Sorry to mislead you. I think we need a short discussion about what checkConstraints() can and can't check against. I'll add to agenda for next call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/11/2012 09:19 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 01/11/2012 18:34 Subject: Re: [DFDL-WG] clarification needed: assert evaluation order and arrays Do I understand this right? I have always thought that "." refers to the current occurrence (i.e., the [dfdl:occursIndex()] is implied if the current elemetn is an array). Hence, if I have an array of a complex type with children named a and b, then "./a" is a valid expression referring to the current instance's child a. Similarly, if I type "../bar", and my parent is an array, then ".." moves me up to the current instance within that array. Upward paths are unambiguous. Current node paths are unambiguous. Only downward paths that encounter an array would need indexing brackets. I think lots of things break if the above isn't the rule. E.g., you can't take an element and add minOccurs/maxOccurs to it to make it repeat without all the paths written on it being broken. If I am right, then there is no problem with making dfdl:checkConstraints(.) take the node, ask if it is an array, and if so compare its occursIndex to maxOccurs. I do take your point that the secondary check for minOccurs has to be expressed differently with a separate assert. The fact that dfdl:occursCount(path) takes a node-set-valued path is .... well really annoying, but inevitable. We need to watch for these exceptions. On Thu, Nov 1, 2012 at 1:39 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: You can't do that with one assert I'm afraid. The main issue is that a path location must always return exactly one item, so it's use within the context of an array must be in conjunction with a predicate, which must refer to a single instance. That's what dfdl:occursIndex() is for. To check that an array meets its bounds, you need a separate assert that uses the dfdl:count() function. You'd have to hard-code the min/max values, and place it on a containing object. <sequence> <annotation><appinfo...> <dfdl:assert>{ dfdl:occursCount(./foo) ge 5 and dfdl:occursCount(./foo) le 10 }</dfdl:assert> </appinfo></annotation> <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.[dfdl:occursIndex()]) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> </sequence> I think that means our definition of dfdl:checkConstraints() in the spec is wrong. Table 34 and Section 5.2 together imply that minOccurs and maxOccurs are used in checkConstraints. I think that simply doesn't work. It should only be using 'fixed' and the facets (I think 'default' is pointless as well). Worth noting that when a path is used in the context dfdl:occursCount(<path>) then it is not an error if more than one item is returned, and it should not be wrapped with fn:exactly-one(). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 01/11/2012 16:45 Subject: [DFDL-WG] clarification needed: assert evaluation order and arrays Sent by: dfdl-wg-bounces(a)ogf.org Question: If I write <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> I have two sources of constraints. One is the pattern, the other the min/max occurs. Does that one assertion calling dfdl:checkConstraints mean both will be checked? That is, one check occurring as each element is parsed, and the other at the end of the array? Assume I am not using any validation option, so the DFDL processor would not otherwise check the max/minOccurs because occursCountKind is parsed. Will the checkConstraints fail as soon as we parse the 11th element (is it checking the min/max occurs for each element occurrence as it is parsed), or do we parse as many as we can, and fail only when we check and find out that the entire array has 36 elements that were successfully parsed? What I would like the above to mean is this: 1) as each element occurrence is parsed we check the pattern and parse error (assertion failed) if there is no match. 2) Also after a successful parse of an occurrence, we check that the index is <=10, and parse-error (assertion failed) if not. 3) at the end of the array, we check that the number of occurrences is >= 5. If not we get a parse error (assertion failed). Comments? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2012-10-30 - resending
by Steve Hanson 02 Nov '12

02 Nov '12

** Resending as some maintenance on Redmine changed all the doc ids! ** Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/8145?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0