- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2014-02-11
by Steve Hanson 12 Feb '14

12 Feb '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13207?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2014-02-11
by Steve Hanson 10 Feb '14

10 Feb '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13207?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Minutes 2014-02-05
by Steve Hanson 06 Feb '14

06 Feb '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13205?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario)
by Steve Hanson 05 Feb '14

05 Feb '14

Thanks Tim, all good points. Comments to your comments. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 05/02/2014 11:01 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) A couple of comments below. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 05/02/2014 10:50 Subject: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org 248 Discriminators and potential points of uncertainty (Steve) 28/1: Steve to write up a proposal to prevent a discriminator from behaving in a non-obvious manner when used with a potential point of uncertainty that turns out not to be an actual point of uncertainty. 5/2: With Steve I started on this by reading section 9.3.3 on points of uncertainty, which lists the potential PoUs. Here's the list to save getting the spec out. 1. An xs:choice branch 2. All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') 3. An optional xs:element 4. An array xs:element 5. All xs:elements in an xs:sequence containing one or more floating xs:elements. The section then looks at each in turn and gives the circumstances when it is an actual PoU or not. As currently written, it is only 3 and 4 where a potential PoU might not be an actual PoU. For 1, 2 and 5 it says they are always actual PoUs. But I'm not sure that's correct. A deeper analysis of what is actually going on with 1, 2 and 5 says to me that there are times when there might not be an actual PoU. 1. Given that there is no concept in DFDL of optional choice branches, then if the last branch is reached then there is no longer a PoU. It must be that branch else it is a processing error. TK: I think of it slightly differently. It is a PoU, even if the branch is the only remaining branch. If we say that the final choice branch is not a PoU then diagnostics become confused - the parser reports the error code as 'error while parsing root/choice/lastBranch/field1' when the correct error code would be 'none of the branches of root/choice were found in the data'. SMH: I see your point. My thinking was that choices have finite branches and a choice is (1,1). If I have got to the last branch then I am not one of the other branches so I must be this one. If there is any other possibility then the model is missing a branch, even if it is just one that contains an empty sequence with an assert {fn:false()}. In practice of course users forget to add that last branch (there's no XSDL equivalent to the 'default' branch of a switch/case statement), so yes they could end up with an unclear diagnostic. 2. There can come a point in an unordered sequence when all that can be encountered is one element, and if that is (1,1) then there is no longer a PoU. TK: It's still a PoU. The specification says that occursCountKind is 'parsed' for all members of an unordered group, so min/maxOccurs do not come into play. SMH: Interesting. The spec says that if a member is optional or an array then it must be 'parsed'. If it is (1,1) though it does not have an occursCountKind. The specific case I was thinking of is when all members are (1,1), so when you have one element to go there is no PoU. However, the rewrite into a repeating choice has the effect of making everything 'parsed', which is really the point you are making. So I agree with you, it is easier to say that everything is an actual PoU else it complicates the rewrite semantic. 5. If all floating elements are (1,1) and all are encountered, then from that point on there are no longer any PoUs due to floating elements. TK: I suspect that floating elements are somewhat like unordered branches - most users will not want min/maxOccurs to affect the parsing of the group. Schema validation ( or more complex validation applied in the receiving application ) will deal with non-conformances. SMH: Possibly yes. With something like X12 NTE segments, that is the case. But we don't express the floating semantic as a rewrite of the whole sequence like we do for unordered, it's more of a per element thing. And if that is done dynamically as we go through the sequence, having no PoU can result. I'd like us to get straight on this before I proceed with the action proper. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/02/2014 10:12 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 27/01/2014 17:39 Subject: Fw: Thoughts on a discriminator scenario Been thinking some more on the discriminator scenario below that I mailed out before xmas, and discussing it with the IBM DFDL team. The 'confusing' aspect of the behaviour is that a discriminator within a potential PoU will act on a higher level PoU if the potential PoU is not an actual PoU. In the example, the array element 'Type1' is not an actual PoU for occurrence 1, only for occurrences 2+. So when the discriminator fires for occurrence 1 it will resolve a higher level unresolved PoU if one exists. Perhaps the spec should say that a discriminator can't 'leak' beyond the potential PoU that encloses it ? If so, then for occurrence 1 the discriminator has no effect, and only has an effect for occurrences 2+. This makes for more predictable and robust schemas. We'd need to go through spec section 9.3.3 carefully to see if this does not break any of the potential PoUs that are listed. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/01/2014 09:55 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 20/12/2013 13:20 Subject: Thoughts on a discriminator scenario Take the following schema (simplified) for element Type1 (1,10) being a loop for elements A,B,C. Type 1 does not have an initiator so I need to use a discriminator to establish the existence of an occurrence of Type1 so that incorrect backtracking does not occur after an error. Because occursCountKind is 'implicit', the 1st occurrence is not a point of uncertainty so the discriminator acts instead on any enclosing point of uncertainty, but for 2nd and subsequent occurrences it acts on Type1. That is all working as designed, but I think users will the 1st occurrence behaviour a bit confusing. There are workarounds to avoid the problem, eg, use occursCountKind 'parsed' or split Type1 into two as (1,1) and (0,9). I think this is worth documenting in a tutorial as this is quite subtle stuff. <xs:element name="Type1" maxOccurs="10" dfdl:occursCountKind="implicit"> <dfdl:discriminator test="{fn:exists(A)}" /> <xs:complexType> <xs:sequence> <xs:element name="A" dfdl:initiator="A:" ... /> <xs:element name="B" dfdl:initiator="B:" ... /> <xs:element name="C" dfdl:initiator="C:"... /> </xs:sequence> </xs:complexType> Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 248 (was Thoughts on a discriminator scenario)
by Steve Hanson 05 Feb '14

05 Feb '14

248 Discriminators and potential points of uncertainty (Steve) 28/1: Steve to write up a proposal to prevent a discriminator from behaving in a non-obvious manner when used with a potential point of uncertainty that turns out not to be an actual point of uncertainty. 5/2: With Steve I started on this by reading section 9.3.3 on points of uncertainty, which lists the potential PoUs. Here's the list to save getting the spec out. 1. An xs:choice branch 2. All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') 3. An optional xs:element 4. An array xs:element 5. All xs:elements in an xs:sequence containing one or more floating xs:elements. The section then looks at each in turn and gives the circumstances when it is an actual PoU or not. As currently written, it is only 3 and 4 where a potential PoU might not be an actual PoU. For 1, 2 and 5 it says they are always actual PoUs. But I'm not sure that's correct. A deeper analysis of what is actually going on with 1, 2 and 5 says to me that there are times when there might not be an actual PoU. 1. Given that there is no concept in DFDL of optional choice branches, then if the last branch is reached then there is no longer a PoU. It must be that branch else it is a processing error. 2. There can come a point in an unordered sequence when all that can be encountered is one element, and if that is (1,1) then there is no longer a PoU. 5. If all floating elements are (1,1) and all are encountered, then from that point on there are no longer any PoUs due to floating elements. I'd like us to get straight on this before I proceed with the action proper. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/02/2014 10:12 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 27/01/2014 17:39 Subject: Fw: Thoughts on a discriminator scenario Been thinking some more on the discriminator scenario below that I mailed out before xmas, and discussing it with the IBM DFDL team. The 'confusing' aspect of the behaviour is that a discriminator within a potential PoU will act on a higher level PoU if the potential PoU is not an actual PoU. In the example, the array element 'Type1' is not an actual PoU for occurrence 1, only for occurrences 2+. So when the discriminator fires for occurrence 1 it will resolve a higher level unresolved PoU if one exists. Perhaps the spec should say that a discriminator can't 'leak' beyond the potential PoU that encloses it ? If so, then for occurrence 1 the discriminator has no effect, and only has an effect for occurrences 2+. This makes for more predictable and robust schemas. We'd need to go through spec section 9.3.3 carefully to see if this does not break any of the potential PoUs that are listed. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/01/2014 09:55 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 20/12/2013 13:20 Subject: Thoughts on a discriminator scenario Take the following schema (simplified) for element Type1 (1,10) being a loop for elements A,B,C. Type 1 does not have an initiator so I need to use a discriminator to establish the existence of an occurrence of Type1 so that incorrect backtracking does not occur after an error. Because occursCountKind is 'implicit', the 1st occurrence is not a point of uncertainty so the discriminator acts instead on any enclosing point of uncertainty, but for 2nd and subsequent occurrences it acts on Type1. That is all working as designed, but I think users will the 1st occurrence behaviour a bit confusing. There are workarounds to avoid the problem, eg, use occursCountKind 'parsed' or split Type1 into two as (1,1) and (0,9). I think this is worth documenting in a tutorial as this is quite subtle stuff. <xs:element name="Type1" maxOccurs="10" dfdl:occursCountKind="implicit"> <dfdl:discriminator test="{fn:exists(A)}" /> <xs:complexType> <xs:sequence> <xs:element name="A" dfdl:initiator="A:" ... /> <xs:element name="B" dfdl:initiator="B:" ... /> <xs:element name="C" dfdl:initiator="C:"... /> </xs:sequence> </xs:complexType> Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Agenda 2014-02-05
by Steve Hanson 05 Feb '14

05 Feb '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13205?download= Note non-standard day (Wed) and time (3pm UK) Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Progressing DFDL WG Action 066
by Steve Hanson 04 Feb '14

04 Feb '14

To discuss as part of action 066. a) IBM has shared the IBM DFDL error messages. They use an IBM-defined error number but eventually this should be replaced/augmented by an OGF DFDL WG defined code. See Tim's email below, which describes how the XSDL spec does this. b) It was noted that the IBM and Daffodil tdml runners should be converged. First step towards this is for IBM to get permission to download the Daffodil tests. Steve has kicked this off today. Perhaps action 066 should be closed and more specific actions spawned for the above? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/06/2012 09:38 ----- From: Tim Kimber/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Steve Hanson/UK/IBM@IBMGB Date: 13/06/2012 21:42 Subject: Re: TDML question Hi Mike, Fair question. IBM requires all diagnostic messages from its software products to be identified by a unique error code. In the IBM test driver program, it is this code that we check for. The XML Schema specification actually assigns unique strings to the various types of error that can occur ( e.g. cvc-* ). If the DFDL specification did the same then the TDML format would be able to specify that the content of the <errors> tag is a list of defined error codes. For DFDL v1.0, I think implementers are free to use it in a way that fits their own requirements. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 13/06/2012 20:45 Subject: TDML question I'm enhancing the daffodil TDML runner. I want to keep it compatible with the IBM one so that we can use TDML files as an interchange medium for discussing bugs/semantics/etc. I added the error feature where one can put expected errors into the TDML file. I have a question about the <error>...</error> element. Which is this: How is the string contents of these error elements used? I tentatively just have it search the error messages created by the parse for these error strings. If any actual error message contains the error string, then that error "passes". Here's my example, which is parsing a 2 character integer, which will fail because the text is AA, and this is base 10. <ts:testSuite xmlns:ts={ tdml } suiteName="theSuiteName"> <ts:parserTestCase ID="some identifier" name="firstUnitTest" root="data"> <ts:document>AA</ts:document> <ts:errors> <ts:error>convert</ts:error>  <ts:error>xs:int</ts:error>  </ts:errors> </ts:parserTestCase> </ts:testSuite> So my test passes so long as the words "convert" and "xs:int" are found in the error message that is generated. Is this consistent with your TDML file usage? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Spec question: 16.6 Forward Progress Requirement
by Tim Kimber 03 Feb '14

03 Feb '14

Proposed revised wording: 16.6 Forward Progress Requirement When parsing an optional occurrence of an array element, if the element is found to have a zero-length representation and its associated separator region is also zero-length then it is a processing error. An optional occurrence is defined as follows: - any occurrence of an element with occursCountKind='implicit' and index > XSDL minOccurs - any occurrence of an element with occursCountKind='parsed' or 'stopValue' If occursCountKind is 'stopValue' then the element will not be a point of uncertainty and the processing error will invalidate the entire array. For occursCountKind='implicit' or 'parsed' the processing error will terminate the array and will thus avoid either redundant processing or an infinite loop ( when XSDL maxOccurs='unbounded'). Feel free to propose better terminology. I'm not sure whether 'optional occurrence' has been used elsewhere in the specification with a different meaning. The final paragraph is conveying important information but can be reworded if necessary. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 ----- Forwarded by Tim Kimber/UK/IBM on 29/01/2014 14:42 ----- From: Tim Kimber/UK/IBM To: dfdl-wg(a)ogf.org, Date: 28/01/2014 14:24 Subject: Spec question: 16.6 Forward Progress Requirement 16.6 Forward Progress Requirement It is a processing error when an array is potentially unbounded and the position in the data does not move during the parsing of an occurrence of the element including any associated separator. This is to prevent an infinite loop. An array is potentially unbounded if any of the following are true: * dfdl:occursCountKind is 'parsed' or 'stopValue' * dfdl:occursCountKind is 'implicit' and XSDL maxOccurs is 'unbounded' Q1: Is this paragraph intended to a) safely terminate the parsing of the array, allowing the parser to proceed with whatever follows in the DFDL schema or b) cause the parser to discard the entire array along with everything that was parsed within the nearest point of uncertainty? If b) then I think a Schema Definition Error would be more appropriate ( because backing out the entire array implies that none of the occurrences were well-formed and some later branch might be the correct option ). Although... Q2: If the array has occursCountKind='stopValue' then none of the occurrences is an actual point of uncertainty because only stopValue can terminate the array. If paragraph 16.1 is invoked then a processing error will be thrown. But the occurrence is not a point of uncertainty so the entire array will be treated as badly-formed. In other words, behaviour will be as described in option b) above. I think that is correct in this case because the array's stopValue was never found. Q3: Should this infinite-loop detection be performed only after at least minOccurs occurrences of the array have been processed successfully? Until then, I think an empty occurrence should cause the default value to be put into the info set. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Rescheduled: DFDL Working Group Call (5 Feb 15:00 GMT in DE3S09)
by Steve Hanson 29 Jan '14

29 Jan '14

Rescheduled: DFDL Working Group Call 05/02/2014 - (Repeats) Location: DE3S09 Steve Hanson has rescheduled this meeting. You have not yet responded. Repeats This entry repeats Required: jcranford(a)mitre.org, mbeckerle(a)tresys.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org, jonathan.w.cranford(a)gmail.com, mbeckerle.dfdl(a)gmail.com, Stephanie Fetzer/Charlotte/IBM@IBMUS OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

OGF DFDL WG Call Minutes 2014-01-28
by Steve Hanson 29 Jan '14

29 Jan '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13202?download= Note next week's call is Wed 3pm UK (10am Eastern). Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0