July 2013 - dfdl-wg - lists.ogf.org

proposal: add fn:error function and dfdl:hex function to DFDL
by Mike Beckerle 12 Jul '13

12 Jul '13

Proposed: add fn:error function and add dfdl:hex function. DFDL only needs one of the several variations of fn:error fn:error($error as xs:QName?, $description as xs:string) as none dfdl:hex($num as xs:integer) as string The dfdl:hex function inverts the behavior of the dfdl:byte, and related functions (currently also proposed) which take a string specification of a hex value. The argument to dfdl:hex can be a long, unsignedLong, or any subtype thereof. The number of hex digits in the resulting string is a function of the input type. If byte or unsigned byte, exactly 2 hex digits are produced, for short and unsignedShort, 4 hex digits, and so on. The resulting value contains a prefix 'x' character as its first character. So: dfdl:short(dfdl:hex(xs:short(208))) eq xs:short(208) is true, and a corresponding tautology holds for all the other DFDL functions that construct integers from hexadecimal, if you replace the 208 above with a value in range for the corresponding numeric type. Example: This allows a meaningful error message that includes the offending parts of data. fn:error(ex:magic_number, fn:concat("The magic number: 0", dfdl:hex(.), " was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).")) The $error argument (of type QName) is used in an implementation-dependent manner to provide the context of the error. So for example, if namespace prefix 'ex' is bound to 'http://example.com' then the above call might produce: "Error: {http://example.com}magic_number - The magic number: 0xFEFF0000 was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian)." Only the part of the above string after the hyphen is standardized. The hyphen and part before the hyphen is implementation dependent.

3 8

about recursive data structures
by Sampo Syreeni 12 Jul '13

12 Jul '13

The DFDL spec has been growing quite a bit over the past two years or so. Mostly because of of the handling of arcane details. So... Has it also grown in the wider, descriptive (complexity) sense? Can it now also describe e.g. the arbitrary link structure a typical *NIX file system image can contain, with its hard links? If so, I might just now have an application for that. If not, why not? It isn't as though you can't develop clean semantics for that, if only as an option. And it's clearly warranted because formats utilizing such constructs constitute a sizable proportion of data both store and actively passed around. -- Sampo Syreeni, aka decoy - decoy(a)iki.fi, http://decoy.iki.fi/front +358-50-5756111, 025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2

2 1

clarification needed: when expressions are evaluated
by Mike Beckerle 11 Jul '13

11 Jul '13

Suppose I have <dfdl:defineFormat name="myFormat"> <dfdl:format byteOrder="{../bOrder }"/> </dfdl:defineFormat> <dfdl:format encoding="{ ../enc }"/> Is that immediately nonsense, since ../bOrder isn't meaningful at top level where formats are defined? What about ../enc - evaluated at top level or when the encoding property is needed? I.e., I use this format down inside my schema ....<element ref="myElement" dfdl:ref="myFormat"/> Does the relative path ../bOrder get executed relative to myElement? My initial thought was that relative paths aren't usable in top-level definitions, but if that's the case then escapeScheme's, which can only be defined at top level currently, would not have any way to take advantage of the runtime-expression capabilites of escapeChar, escapeEscapeChar, etc. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 1

Fw: What are the consequences of a failed assert?
by Steve Hanson 11 Jul '13

11 Jul '13

Let's say I have an assert with 'fatalError'. What happens if the assert has a processing error when it is being evaluated. Is this still a processing error, or does it get converted into a fatal error? I include the use of fn:error() here as well. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/07/2013 09:59 ----- From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB, Cc: "Cranford, Jonathan W." <jcranford(a)mitre.org>, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 10/07/2013 18:06 Subject: RE: [DFDL-WG] What are the consequences of a failed assert? > What is being asked for here sounds like the ability to throw an error and stop the parser regardless of what points of uncertainty are in scope Exactly! > James, would this 'fatal error' give you the capability you are looking for? Yes, I think it would. I would like to encourage the WG to add this feature if at all possible. From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Monday, July 08, 2013 6:11 AM To: Tim Kimber Cc: Cranford, Jonathan W.; dfdl-wg(a)ogf.org; Garriss Jr., James P. Subject: RE: [DFDL-WG] What are the consequences of a failed assert? Errata 3.4 added the failureType attribute to dfdl:assert so that an assert could report the equivalent of a validation error and carry on ('recoverable error') instead of throwing a 'processing error'. What is being asked for here sounds like the ability to throw an error and stop the parser regardless of what points of uncertainty are in scope. That does sound like a genuinely useful thing to be able to do, and gives dfdl:assert the full range of failure actions. We'd have to create a new error type, say 'fatal error'. It would have the same effect as a 'schema definition error'. James, would this 'fatal error' give you the capability you are looking for? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, "dfdl-wg-bounces(a)ogf.org" <dfdl-wg-bounces(a)ogf.org>, "Garriss Jr., James P." <jgarriss(a)mitre.org>, Steve Hanson/UK/IBM@IBMGB Date: 08/07/2013 09:29 Subject: RE: [DFDL-WG] What are the consequences of a failed assert? It depends on whether there was exactly one point of uncertainty active when the discriminator expression evaluated to 'true'. Points of uncertainty can be nested within one another. A POU can be resolved ( or made 'inactive' as I was trying to hint at ) by a discriminator that evaluates to 'true'. When a processing error occurs, it is caught by the nearest ( most deeply nested ) *active* POU. If there is no active POU then the parser halts and reports the error. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "Garriss Jr., James P." <jgarriss(a)mitre.org>, Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, "dfdl-wg-bounces(a)ogf.org" <dfdl-wg-bounces(a)ogf.org> Date: 05/07/2013 20:31 Subject: RE: [DFDL-WG] What are the consequences of a failed assert? Wouldn’t the use of a discriminator suppress the backtracking behavior in the face of a processing error? That is, if a discriminator is used to confirm that the Content-Type header branch is the correct one, wouldn’t a subsequent assert that causes a processing error force the parser to halt with an error? -Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Garriss Jr., James P. Sent: Friday, July 05, 2013 12:09 PM To: Steve Hanson; Tim Kimber Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Very helpful, guys, as always. Thank you. Question: Is it possible to force the failure of an assert to be a validation error instead of a processing error? Here’s why: All the email headers are modeled as an unordered list. The last item on the list is a catch-all, designed to scoop up all the unexpected headers and hide them from the infoset (via hiddenGroupRef). So if I have an Content-Type header that is obviously invalid: Content-Type: bogus/data The assert fails as expected. But since there is always another choice, the catch-all, the Content-Type header quietly disappears. What I really want is for there to be a validation error for “bogus/data,” so that processing stops! Is that possible? Is that reasonable? From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Steve Hanson Sent: Thursday, July 04, 2013 4:26 AM To: Tim Kimber Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Almost. A Recoverable Error does not cause backtracking. It was added to enable an assert to perform validation that checked the data stream rather than just the infoset. For example, checking the max physical length of a non-string value. So it has the same effect on the parse as a Validation Error. I've corrected your summary. The definition and description of the different errors is covered in the spec by section 2. There have been some errata in this section to clarify the behaviour. Some of the wording has been around since early drafts of the spec, so any suggestions for improvement are welcome. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 03/07/2013 21:51 Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Sent by: dfdl-wg-bounces(a)ogf.org There are four types of error in DFDL: - a Schema Definition Error : The schema itself is not valid. ( at least three kinds: xsd not valid, xsd not in the DFLD subset, DFDL annotations not following the rules ) - A Processing Error : The data cannot be parsed. Or if unparsing, the info set cannot be unparsed. - A Recoverable Error : ( see the Errata ). This is effectively a user-defined form of Validation Error, raised while executing a DFDL assert, - A Validation Error : The info set does not conform to the XSD A Schema Definition Error is immediately fatal. Most of these can be detected by the processor before parsing/unparsing begins. A Processing Error or a Recoverable Error will cause the parser to suppress the error and backtrack to the nearest point of uncertainty. So it will only stop the parse if there are no points of uncertainty currently active. A Recoverable Error does not cause backtracking - the parser continues to parse after reporting the error. A Validation Error is only reported if validation is enabled in the DFDL processor. It does not cause backtracking - the parser continues to parse after reporting the error. That's the gist of it. Further details from other WG members may follow shortly, depending on how accurate I have managed to be. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 03/07/2013 19:06 Subject: [DFDL-WG] What are the consequences of a failed assert? Sent by: dfdl-wg-bounces(a)ogf.org I have an element with an assert, <xsd:element name="Type" dfdl:inputValueCalc="{ fn:lower-case(../MixedCaseType) }"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="The type must match one of the values on the enumerated list."/> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="application"/> <xsd:enumeration value="multipart"/> <xsd:enumeration value="message"/> <xsd:enumeration value="text"/> </xsd:restriction> </xsd:simpleType> </xsd:element> and the assert is failing (as it should in this case!). Parse Error: Assertion failed. The type must match one of the values on the enumerated list. What are the consequences of a failed assert? I have an old version of the spec—is there a place to a get a current, complete copy?—but it says “An unsuccessful dfdl:assert causes a processing error.” 1. What does “processing error” mean in English? 2. Does it mean the input is invalid? 3. Does it mean the processor should stop here and go no further? 4. Does it mean the process should simply ignore the problem and move on to the next item in the schema? TIA -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 7

Implementation limits
by Steve Hanson 11 Jul '13

11 Jul '13

Just checking - did we close on this? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 21/01/2013 18:38 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 19/11/2012 21:08 Subject: Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Do we have a section where this goes? There are lots of implied implementation limits. Maximum string length? Maximum hexbinary length? Maximum number of packed digits in various numbers. Maximum length of a delimiter? Maximum length of a fixed attribute? Maximum length of a pattern regex expression? Maximum length of a regular expression? There's also limits on buffering of data for unparsing, etc. There's also limits on lookahead/speculation. I.e., how far an implementation is willing to speculate forward. etc. For some types these are clear: e.g., binary xs:int is 32 bits max. Do we need a numbered section on implementation specific limits. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 4

Re: [DFDL-WG] What are the consequences of a failed assert?
by Steve Hanson 10 Jul '13

10 Jul '13

Errata 3.4 added the failureType attribute to dfdl:assert so that an assert could report the equivalent of a validation error and carry on ('recoverable error') instead of throwing a 'processing error'. What is being asked for here sounds like the ability to throw an error and stop the parser regardless of what points of uncertainty are in scope. That does sound like a genuinely useful thing to be able to do, and gives dfdl:assert the full range of failure actions. We'd have to create a new error type, say 'fatal error'. It would have the same effect as a 'schema definition error'. James, would this 'fatal error' give you the capability you are looking for? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, "dfdl-wg-bounces(a)ogf.org" <dfdl-wg-bounces(a)ogf.org>, "Garriss Jr., James P." <jgarriss(a)mitre.org>, Steve Hanson/UK/IBM@IBMGB Date: 08/07/2013 09:29 Subject: RE: [DFDL-WG] What are the consequences of a failed assert? It depends on whether there was exactly one point of uncertainty active when the discriminator expression evaluated to 'true'. Points of uncertainty can be nested within one another. A POU can be resolved ( or made 'inactive' as I was trying to hint at ) by a discriminator that evaluates to 'true'. When a processing error occurs, it is caught by the nearest ( most deeply nested ) *active* POU. If there is no active POU then the parser halts and reports the error. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "Garriss Jr., James P." <jgarriss(a)mitre.org>, Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, "dfdl-wg-bounces(a)ogf.org" <dfdl-wg-bounces(a)ogf.org> Date: 05/07/2013 20:31 Subject: RE: [DFDL-WG] What are the consequences of a failed assert? Wouldn’t the use of a discriminator suppress the backtracking behavior in the face of a processing error? That is, if a discriminator is used to confirm that the Content-Type header branch is the correct one, wouldn’t a subsequent assert that causes a processing error force the parser to halt with an error? -Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Garriss Jr., James P. Sent: Friday, July 05, 2013 12:09 PM To: Steve Hanson; Tim Kimber Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Very helpful, guys, as always. Thank you. Question: Is it possible to force the failure of an assert to be a validation error instead of a processing error? Here’s why: All the email headers are modeled as an unordered list. The last item on the list is a catch-all, designed to scoop up all the unexpected headers and hide them from the infoset (via hiddenGroupRef). So if I have an Content-Type header that is obviously invalid: Content-Type: bogus/data The assert fails as expected. But since there is always another choice, the catch-all, the Content-Type header quietly disappears. What I really want is for there to be a validation error for “bogus/data,” so that processing stops! Is that possible? Is that reasonable? From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Steve Hanson Sent: Thursday, July 04, 2013 4:26 AM To: Tim Kimber Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Almost. A Recoverable Error does not cause backtracking. It was added to enable an assert to perform validation that checked the data stream rather than just the infoset. For example, checking the max physical length of a non-string value. So it has the same effect on the parse as a Validation Error. I've corrected your summary. The definition and description of the different errors is covered in the spec by section 2. There have been some errata in this section to clarify the behaviour. Some of the wording has been around since early drafts of the spec, so any suggestions for improvement are welcome. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 03/07/2013 21:51 Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Sent by: dfdl-wg-bounces(a)ogf.org There are four types of error in DFDL: - a Schema Definition Error : The schema itself is not valid. ( at least three kinds: xsd not valid, xsd not in the DFLD subset, DFDL annotations not following the rules ) - A Processing Error : The data cannot be parsed. Or if unparsing, the info set cannot be unparsed. - A Recoverable Error : ( see the Errata ). This is effectively a user-defined form of Validation Error, raised while executing a DFDL assert, - A Validation Error : The info set does not conform to the XSD A Schema Definition Error is immediately fatal. Most of these can be detected by the processor before parsing/unparsing begins. A Processing Error or a Recoverable Error will cause the parser to suppress the error and backtrack to the nearest point of uncertainty. So it will only stop the parse if there are no points of uncertainty currently active. A Recoverable Error does not cause backtracking - the parser continues to parse after reporting the error. A Validation Error is only reported if validation is enabled in the DFDL processor. It does not cause backtracking - the parser continues to parse after reporting the error. That's the gist of it. Further details from other WG members may follow shortly, depending on how accurate I have managed to be. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 03/07/2013 19:06 Subject: [DFDL-WG] What are the consequences of a failed assert? Sent by: dfdl-wg-bounces(a)ogf.org I have an element with an assert, <xsd:element name="Type" dfdl:inputValueCalc="{ fn:lower-case(../MixedCaseType) }"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="The type must match one of the values on the enumerated list."/> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="application"/> <xsd:enumeration value="multipart"/> <xsd:enumeration value="message"/> <xsd:enumeration value="text"/> </xsd:restriction> </xsd:simpleType> </xsd:element> and the assert is failing (as it should in this case!). Parse Error: Assertion failed. The type must match one of the values on the enumerated list. What are the consequences of a failed assert? I have an old version of the spec—is there a place to a get a current, complete copy?—but it says “An unsuccessful dfdl:assert causes a processing error.” 1. What does “processing error” mean in English? 2. Does it mean the input is invalid? 3. Does it mean the processor should stop here and go no further? 4. Does it mean the process should simply ignore the problem and move on to the next item in the schema? TIA -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Fw: Action 205: whitespace in DFDL annotations
by Steve Hanson 10 Jul '13

10 Jul '13

Suman Please can you review the proposal below, we would like to close on this on the next WG call. We are particularly interested in why you chose xs:string for DFDL string literal, but then xs:token for List of DFDL string literal, as that means there is different whitespace behaviour for the same DFDL string literal depending on which property it is used in, which does not sound right. <xsd:simpleType name="DFDLStringLiteral"> <xsd:restriction base="xsd:string"> </xsd:restriction> </xsd:simpleType> <xsd:simpleType name="ListOfDFDLStringLiteral"> <xsd:list itemType="xsd:token"/> </xsd:simpleType> I was expecting to see: <xsd:simpleType name="DFDLStringLiteral"> <xsd:restriction base="xsd:token"> </xsd:restriction> </xsd:simpleType> <xsd:simpleType name="ListOfDFDLStringLiteral"> <xsd:list itemType="dfdl:DFDLStringLiteral"/> </xsd:simpleType> Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 10/07/2013 09:49 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 09/07/2013 12:02 Subject: [DFDL-WG] Action 205: whitespace in DFDL annotations For discussion on today's WG call. Action 205 was raised to ensure that DFDL 'property types' are declared with XML Schema types that provide the correct whitespace handling behaviour. The XML Schema types of the various DFDL 'property types' are given in Part 1 of IBM's Schemas-for-DFDL. The question boils down to whether a 'property type' should be an xs:string or xs:token. The former preserves whitespace, the latter normalizes and trims. (Note that xs:NMTOKEN is intended for attributes only, so should not be used for DFDL properties as they can be expressed in attribute or element forms.) My recommendation is; - Enumeration changed from xs:string to xs:token (reason: to match XSDL enums and trim leading/trailing whitespace) - DFDL regular expression stays as xs:string (reason: regex may contain literal white space) - DFDL string literal changed from xs:string to xs:token (reason: currently inconsistent with List of DFDL string literal) - List of DFDL string literal stays as list of xs:token - DFDL expression changed from xs:token to xs:string (reason: XPath may contain non-ignorable whitespace) Further: - DFDL regular expression should not trim leading/trailing whitespace - DFDL expression should trim leading whitespace before { and trailing whitespace after } - The enum of DFDL property names should be based on xs:token The xs:unions for DFDL properties that can be two or more of the above may/will need the member ordering reviewed. Example: <xsd:simpleType name="BinaryFloatRepEnum_Or_DFDLExpression"> <xsd:union> <xsd:simpleType> <xsd:restriction base="dfdl:DFDLExpression" /> </xsd:simpleType> <xsd:simpleType> <xsd:restriction base="dfdl:BinaryFloatRepEnum"/> </xsd:simpleType> </xsd:union> </xsd:simpleType> Usually in a union, the most restrictive member is placed first. With the current types, the above has xs:token followed by xs:string, in accordance with this practice. But the recommendation changes the types of both members, so that the above becomes xs:string followed by xs:token. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Suman Kalia/Toronto/IBM@IBMCA, Cc: dfdl-wg(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 27/03/2013 16:38 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Suman Looking at the XML schema-for-schemas, and doing a test in the XSD editor in eclipse, XSD enumeration facets are modelled as xs:NMTOKEN and not xs:string, like DFDL enums. XSD is perfectly happy to strip/collapse white space. I think therefore that we should be doing the same for DFDL enum properties. I don't see any harm in this - an enum is a contiguous sequence of non-whitespace characters anyway, so any leading/trailing whitespace is harmless. Looks like XSD pattern facet is modelled as xs:string, preserving white space. We should do the same for DFDL regex properties. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 19/03/2013 17:41 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token The type called DFDLExpressionOrPatternOrNothing only makes sense for use in one place - the element value of an assert or discriminator. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia <kalia(a)ca.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 19/03/2013 17:29 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Sent by: dfdl-wg-bounces(a)ogf.org Mike - I am not sure but my gut feeling is that it would start with the most restrictive one first. i.e If empty string ( assuming it has length facet 1) - would match Nothing , then xsd:token which is restricted form of xs:string. I think you are going to get string with white spaces collpsed ( xsd:token) if it not empty string. You can run few tests to see the behavior.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 03/19/2013 12:43 PM Subject: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Sent by: dfdl-wg-bounces(a)ogf.org This came up on the call. The schemas I have for DFDL annotations have DFDLRegularExpression as an xs:string, and DFDLExpression as an xs:token. I have no clue what a union of these types behaves like. But we have a union called DFDLExpressionOrPatternOrNothing which is a 3-way union of DFDLExpression, DFDLRegularExpression, and EmptyString (which is also derived from xs:string but has length facet of 0 as well. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

clarification needed about lengthKind='pattern' when the pattern does not match
by Mike Beckerle 09 Jul '13

09 Jul '13

>From what I can find, neither the DFDL spec, nor the errata seem to make it clear that when using lengthKind='pattern' if there is no match to the lengthPattern regex, it is NOT a processing error, but rather just results in length 0. We need to clarify this. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

OGF DFDL WG Call Minutes 2013-07-09
by Steve Hanson 09 Jul '13

09 Jul '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/13093?download= Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 205: whitespace in DFDL annotations
by Steve Hanson 09 Jul '13

09 Jul '13

We might be able to retain xs:token for DFDL expression on the grounds that users can use the dfdl:decodeDFDLEntities() function if they need to use white space in XPath string literals. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 09/07/2013 14:52 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 09/07/2013 12:02 Subject: [DFDL-WG] Action 205: whitespace in DFDL annotations For discussion on today's WG call. Action 205 was raised to ensure that DFDL 'property types' are declared with XML Schema types that provide the correct whitespace handling behaviour. The XML Schema types of the various DFDL 'property types' are given in Part 1 of IBM's Schemas-for-DFDL. The question boils down to whether a 'property type' should be an xs:string or xs:token. The former preserves whitespace, the latter normalizes and trims. (Note that xs:NMTOKEN is intended for attributes only, so should not be used for DFDL properties as they can be expressed in attribute or element forms.) My recommendation is; - Enumeration changed from xs:string to xs:token (reason: to match XSDL more closely and trim leading/trailing whitespace) - DFDL regular expression stays as xs:string (reason: regex may contain literal white space) - DFDL string literal changed from xs:string to xs:token (reason: currently inconsistent with List of DFDL string literal) - List of DFDL string literal stays as list of xs:token - DFDL expression changed from xs:token to xs:string (reason: XPath may contain non-ignorable whitespace) Further: - DFDL regular expression should not trim leading/trailing whitespace - DFDL expression should trim leading whitespace before { and trailing whitespace after } - The enum of DFDL property names should be based on xs:token The xs:unions for DFDL properties that can be two or more of the above may need the member ordering reviewed. Example: <xsd:simpleType name="BinaryFloatRepEnum_Or_DFDLExpression"> <xsd:union> <xsd:simpleType> <xsd:restriction base="dfdl:DFDLExpression" /> </xsd:simpleType> <xsd:simpleType> <xsd:restriction base="dfdl:BinaryFloatRepEnum"/> </xsd:simpleType> </xsd:union> </xsd:simpleType> Usually in a union, the most restrictive member is placed first. With the current types, the above has xs:token followed by xs:string, in accordance with this practice. But the recommendation changes the types of both members, so that the above becomes xs:string followed by xs:token. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Suman Kalia/Toronto/IBM@IBMCA, Cc: dfdl-wg(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 27/03/2013 16:38 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Suman Looking at the XML schema-for-schemas, and doing a test in the XSD editor in eclipse, XSD enumeration facets are modelled as xs:NMTOKEN and not xs:string, like DFDL enums. XSD is perfectly happy to strip/collapse white space. I think therefore that we should be doing the same for DFDL enum properties. I don't see any harm in this - an enum is a contiguous sequence of non-whitespace characters anyway, so any leading/trailing whitespace is harmless. Looks like XSD pattern facet is modelled as xs:string, preserving white space. We should do the same for DFDL regex properties. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 19/03/2013 17:41 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token The type called DFDLExpressionOrPatternOrNothing only makes sense for use in one place - the element value of an assert or discriminator. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia <kalia(a)ca.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 19/03/2013 17:29 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Sent by: dfdl-wg-bounces(a)ogf.org Mike - I am not sure but my gut feeling is that it would start with the most restrictive one first. i.e If empty string ( assuming it has length facet 1) - would match Nothing , then xsd:token which is restricted form of xs:string. I think you are going to get string with white spaces collpsed ( xsd:token) if it not empty string. You can run few tests to see the behavior.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 03/19/2013 12:43 PM Subject: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Sent by: dfdl-wg-bounces(a)ogf.org This came up on the call. The schemas I have for DFDL annotations have DFDLRegularExpression as an xs:string, and DFDLExpression as an xs:token. I have no clue what a union of these types behaves like. But we have a union called DFDLExpressionOrPatternOrNothing which is a 3-way union of DFDLExpression, DFDLRegularExpression, and EmptyString (which is also derived from xs:string but has length facet of 0 as well. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0