May 2014 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

data where inactive escape character is to be retained
by Mike Beckerle 04 Jun '14

04 Jun '14

We have data that has ; as separator and has fields that look like this: abcd \; efgh That's a single field. The backslash escapes the ; so that the data is abcd ; efgh. This same data set also has abcd \n efgh Here the backslash precedes an ordinary non-delimiter. The data is supposed to be abcd \n efgh. That is, this data set requires the backslash to be retained in the data when it is not preceding the start of a delimiter. Am I missing something or is it impossible to model this? It would seem there needs to be a flag to indicate whether the escape characters that don't actually escape a delimiter are to be retained or not. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

3 2

OGF DFDL WG Call Minutes 2014-05-20
by Steve Hanson 30 May '14

30 May '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13249?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

2 1

OGF DFDL WG Call 2014-05-27 - cancelled
by Steve Hanson 27 May '14

27 May '14

Sorry to have to cancel the call, I am unable to make it. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: DFDL Working Group Call
by Steve Hanson 27 May '14

27 May '14

Steve is busy this week with customer issues - canceling

1 0

Action 259 - Consider allowing more flexible escapeBlock schemes
by Steve Hanson 20 May '14

20 May '14

Action 259 was raised last call to decide what to do about the following, as minuted: Steve has an example of an escape block where the escape block end is not at the end of the un-trimmed data. This gives a processing error. Another IBM product accepts this usage. Should DFDL allow this? Or should there be a new escapeKind that allows escapeBlockStart/End anywhere? Tried importing these values from a CSV file into an Excel spreadsheet, a Symphony spreadsheet (ie, successor to 123), and also accessing them via ODBC using a Microsoft driver, to compare with IBM DFDL and IBM Cast Iron behaviour. Test Data IBM DFDL IBM Cast Iron MS Excel Lotus Symphony ODBC 1 This is normal This is normal This is normal This is normal This is normal This is normal 2 "This is OK" This is OK This is OK This is OK This is OK This is OK 3 "This| is expected" This| is expected This| is expected This| is expected This| is expected This| is expected 4 This too "is OK" This too "is OK" This too "is OK" This too "is OK" This too "is OK" This too 5 Even "this" is OK Even "this" is OK Even "this" is OK Even "this" is OK Even "this" is OK Even 6 "This" is NOT OK PARSE FAILED This is NOT OK This is NOT OK This is NOT OK This 7 "This"" is still OK" This" is still OK This" is still OK This" is still OK This" is still OK This" is still OK The data under discussion is 6. It looks like DFDL is out of step with the behaviour of Excel / Symphony spreadsheets, and Cast Iron has adopted that behaviour too. Out of interest I also checked the output behaviour from Excel. That escaped all instances of embedded quotes in the same way as DFDL, so no issues there. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

OGF DFDL WG Call Agenda 2014-05-20
by Steve Hanson 20 May '14

20 May '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13249?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 248 (was Thoughts on a discriminator scenario)
by Steve Hanson 15 May '14

15 May '14

This will be discussed on today's call. Please have a position on the paragraph below that ends 'What do others think?' Thanks Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/05/2014 10:19 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 30/04/2014 12:25 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Tim Responses below. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/04/2014 14:03 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org "2. If a potential point of uncertainty is sometimes an actual point of uncertainty (ock 'implicit') then a discriminator that applies it will only ever resolve, or have no effect on, that point of uncertainty. It never has an effect on any enclosing point of uncertainty." This could be misinterpreted. The discriminator could evaluate to 'false' and thus cause the POI to be resolved negatively ( the component would be 'known not to exist' ) SMH: Agree, and I can improve the words here. 1. and 3. will both apply if an element with ock='fixed' appears as a choice branch. Is the POI always an actual POI or never? SMH: No. There are two independent points of uncertainty, the choice branch and the array. The wording of 3. reads very strangely. 'If a potential point of uncertainty is never an actual point of uncertainty' begs the question 'why is it even a potential point of uncertainty?'. The current wording follows from our definition of the term 'point of uncertainty': "A point of uncertainty occurs in the data stream when there is more than one schema component that might occur at that point." Points of uncertainty can be nested. Any one of the following constructs is a potential point of uncertainty: 1. An xs:choice 2. All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') 3. An optional xs:element 4. An array xs:element. 5. All xs:elements in an xs:sequence containing one or more floating xs:elements. I think this definition is too broad. It forces us to discuss potential POUs that will never be actual POUs according to the first sentence. SMH: Yes it does read a bit strangely, but there's a reason for this. If we said that ock 'fixed', 'expression' or 'stopValue' are never POUs then what does it mean if a discriminator is placed on such an element? A discriminator gets evaluated for each occurrence of an array. For that reason we can not let a discriminator within an array leak beyond the array - regardless of whether it is a POU or not - otherwise what does that mean to enclosing POUs? So even if we said that ock 'fixed', 'expression' or 'stopValue' are never POUs we would still need the spec to state that a discriminator never leaks beyond them. I think it is clearer to say that a discriminator never leaks beyond a potential POU and keep the existing definition. What do others think? regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/04/2014 11:44 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org 248 Discriminators and potential points of uncertainty (Steve) 28/1: Steve to write up a proposal to prevent a discriminator from behaving in a non-obvious manner when used with a potential point of uncertainty that turns out not to be an actual point of uncertainty. 5/2: Steve sent an email to check whether choice branches, unordered elements and floating elements should always be actual points of uncertainty, as there are times when there is no uncertainty, eg, last choice branch; all floating elements found. It was decided that they are always actual points of uncertainty. To do otherwise will complicate implementations and result in fragile schemas. Steve will proceed with the proposal on that basis. Based on the above, which reflects the email discussion below, here is what I propose to resolve this action. 1. If a potential point of uncertainty is always an actual point of uncertainty (choice branch, element in unordered sequence, floating element, ock 'parsed') then a discriminator that applies to it will only ever resolve that point of uncertainty. It never has an effect on any enclosing point of uncertainty. 2. If a potential point of uncertainty is sometimes an actual point of uncertainty (ock 'implicit') then a discriminator that applies it will only ever resolve, or have no effect on, that point of uncertainty. It never has an effect on any enclosing point of uncertainty. 3. If a potential point of uncertainty is never an actual point of uncertainty (ock 'fixed', 'expression', 'stopValue') then a discriminator that applies to it will never have an effect on that point of uncertainty. Nor does it ever have an effect on any enclosing point of uncertainty. I think 1 and 2 are not controversial, but there is an alternative for 3: 3. If a potential point of uncertainty is never an actual point of uncertainty (ock 'fixed', 'expression', 'stopValue') then a discriminator that applies to it will never have an effect on that point of uncertainty. Instead the discriminator is applied to any enclosing point of uncertainty. The alternative means that changing an element from (say) ock 'parsed' to ock 'expression' has the same effect on a discriminator as changing the element to (1,1). The discriminator that applied to it now applies to any enclosing pou. SMH: Afternote: The alternative does not work for the reason given in my reply to Tim above. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 05/02/2014 12:04 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Thanks Tim, all good points. Comments to your comments. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 05/02/2014 11:01 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) A couple of comments below. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 05/02/2014 10:50 Subject: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org 248 Discriminators and potential points of uncertainty (Steve) 28/1: Steve to write up a proposal to prevent a discriminator from behaving in a non-obvious manner when used with a potential point of uncertainty that turns out not to be an actual point of uncertainty. 5/2: With Steve I started on this by reading section 9.3.3 on points of uncertainty, which lists the potential PoUs. Here's the list to save getting the spec out. 1. An xs:choice branch 2. All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') 3. An optional xs:element 4. An array xs:element 5. All xs:elements in an xs:sequence containing one or more floating xs:elements. The section then looks at each in turn and gives the circumstances when it is an actual PoU or not. As currently written, it is only 3 and 4 where a potential PoU might not be an actual PoU. For 1, 2 and 5 it says they are always actual PoUs. But I'm not sure that's correct. A deeper analysis of what is actually going on with 1, 2 and 5 says to me that there are times when there might not be an actual PoU. 1. Given that there is no concept in DFDL of optional choice branches, then if the last branch is reached then there is no longer a PoU. It must be that branch else it is a processing error. TK: I think of it slightly differently. It is a PoU, even if the branch is the only remaining branch. If we say that the final choice branch is not a PoU then diagnostics become confused - the parser reports the error code as 'error while parsing root/choice/lastBranch/field1' when the correct error code would be 'none of the branches of root/choice were found in the data'. SMH: I see your point. My thinking was that choices have finite branches and a choice is (1,1). If I have got to the last branch then I am not one of the other branches so I must be this one. If there is any other possibility then the model is missing a branch, even if it is just one that contains an empty sequence with an assert {fn:false()}. In practice of course users forget to add that last branch (there's no XSDL equivalent to the 'default' branch of a switch/case statement), so yes they could end up with an unclear diagnostic. 2. There can come a point in an unordered sequence when all that can be encountered is one element, and if that is (1,1) then there is no longer a PoU. TK: It's still a PoU. The specification says that occursCountKind is 'parsed' for all members of an unordered group, so min/maxOccurs do not come into play. SMH: Interesting. The spec says that if a member is optional or an array then it must be 'parsed'. If it is (1,1) though it does not have an occursCountKind. The specific case I was thinking of is when all members are (1,1), so when you have one element to go there is no PoU. However, the rewrite into a repeating choice has the effect of making everything 'parsed', which is really the point you are making. So I agree with you, it is easier to say that everything is an actual PoU else it complicates the rewrite semantic. 5. If all floating elements are (1,1) and all are encountered, then from that point on there are no longer any PoUs due to floating elements. TK: I suspect that floating elements are somewhat like unordered branches - most users will not want min/maxOccurs to affect the parsing of the group. Schema validation ( or more complex validation applied in the receiving application ) will deal with non-conformances. SMH: Possibly yes. With something like X12 NTE segments, that is the case. But we don't express the floating semantic as a rewrite of the whole sequence like we do for unordered, it's more of a per element thing. And if that is done dynamically as we go through the sequence, having no PoU can result. I'd like us to get straight on this before I proceed with the action proper. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/02/2014 10:12 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 27/01/2014 17:39 Subject: Fw: Thoughts on a discriminator scenario Been thinking some more on the discriminator scenario below that I mailed out before xmas, and discussing it with the IBM DFDL team. The 'confusing' aspect of the behaviour is that a discriminator within a potential PoU will act on a higher level PoU if the potential PoU is not an actual PoU. In the example, the array element 'Type1' is not an actual PoU for occurrence 1, only for occurrences 2+. So when the discriminator fires for occurrence 1 it will resolve a higher level unresolved PoU if one exists. Perhaps the spec should say that a discriminator can't 'leak' beyond the potential PoU that encloses it ? If so, then for occurrence 1 the discriminator has no effect, and only has an effect for occurrences 2+. This makes for more predictable and robust schemas. We'd need to go through spec section 9.3.3 carefully to see if this does not break any of the potential PoUs that are listed. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/01/2014 09:55 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 20/12/2013 13:20 Subject: Thoughts on a discriminator scenario Take the following schema (simplified) for element Type1 (1,10) being a loop for elements A,B,C. Type 1 does not have an initiator so I need to use a discriminator to establish the existence of an occurrence of Type1 so that incorrect backtracking does not occur after an error. Because occursCountKind is 'implicit', the 1st occurrence is not a point of uncertainty so the discriminator acts instead on any enclosing point of uncertainty, but for 2nd and subsequent occurrences it acts on Type1. That is all working as designed, but I think users find will the 1st occurrence behaviour a bit confusing. There are workarounds to avoid the problem, eg, use occursCountKind 'parsed' or split Type1 into two as (1,1) and (0,9). I think this is worth documenting in a tutorial as this is quite subtle stuff. <xs:element name="Type1" maxOccurs="10" dfdl:occursCountKind="implicit"> <dfdl:discriminator test="{fn:exists(A)}" /> <xs:complexType> <xs:sequence> <xs:element name="A" dfdl:initiator="A:" ... /> <xs:element name="B" dfdl:initiator="B:" ... /> <xs:element name="C" dfdl:initiator="C:"... /> </xs:sequence> </xs:complexType> Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

DFDL schemas for ISO8583 1993 published on GitHub
by Steve Hanson 14 May '14

14 May '14

The repository in DFDLSchemas GitHub which contains DFDL schemas for the ISO 8583 standard has been updated. It now includes schemas for 1993 and 1987 versions of the standard, supporting both packed bitmaps and unpacked bitmaps. The schemas are compatible with IBM DFDL 1.0 and 1.1 releases, as shipped in IBM WebSphere Message Broker 8.0.0.2 onwards and IBM Integration Bus 9.0.0.1 onwards, respectively. Example data streams and XML infosets are included. Full details in the repository readme at https://github.com/DFDLSchemas/ISO8583. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL mini-tutorials
by Cranford, Jonathan W. 13 May '14

13 May '14

DFDL Working Group, Attached are four mini-tutorials written by Avery Bibeau, a high school student. James Garriss of MITRE is mentoring him on a school project which includes writing tutorials for DFDL. The idea is that these mini-tutorials could be folded into tutorials in the future more in line with the existing DFDL tutorials at www.ogf.org/dfdl. Avery is contributing these mini-tutorials to the Working Group with the hope that he would be credited in the tutorials. Any feedback would be appreciated. Also, below is a list of topics he's planning to work on. If you see topics that aren't implemented in either of the implementations, please let us know, as he is only trying to write tutorials on features that he can test in the existing implementations. * asserts * discriminators * input value calculations He had planned to do a mini-tutorial on default values, but had problems testing his examples in both implementations because apparently support for default values is not quite implemented yet. Thanks in advance, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

2 1