July 2013 - dfdl-wg - lists.ogf.org

Action 205: whitespace in DFDL annotations
by Steve Hanson 09 Jul '13

09 Jul '13

For discussion on today's WG call. Action 205 was raised to ensure that DFDL 'property types' are declared with XML Schema types that provide the correct whitespace handling behaviour. The XML Schema types of the various DFDL 'property types' are given in Part 1 of IBM's Schemas-for-DFDL. The question boils down to whether a 'property type' should be an xs:string or xs:token. The former preserves whitespace, the latter normalizes and trims. (Note that xs:NMTOKEN is intended for attributes only, so should not be used for DFDL properties as they can be expressed in attribute or element forms.) My recommendation is; - Enumeration changed from xs:string to xs:token (reason: to match XSDL more closely and trim leading/trailing whitespace) - DFDL regular expression stays as xs:string (reason: regex may contain literal white space) - DFDL string literal changed from xs:string to xs:token (reason: currently inconsistent with List of DFDL string literal) - List of DFDL string literal stays as list of xs:token - DFDL expression changed from xs:token to xs:string (reason: XPath may contain non-ignorable whitespace) Further: - DFDL regular expression should not trim leading/trailing whitespace - DFDL expression should trim leading whitespace before { and trailing whitespace after } - The enum of DFDL property names should be based on xs:token The xs:unions for DFDL properties that can be two or more of the above may need the member ordering reviewed. Example: <xsd:simpleType name="BinaryFloatRepEnum_Or_DFDLExpression"> <xsd:union> <xsd:simpleType> <xsd:restriction base="dfdl:DFDLExpression" /> </xsd:simpleType> <xsd:simpleType> <xsd:restriction base="dfdl:BinaryFloatRepEnum"/> </xsd:simpleType> </xsd:union> </xsd:simpleType> Usually in a union, the most restrictive member is placed first. With the current types, the above has xs:token followed by xs:string, in accordance with this practice. But the recommendation changes the types of both members, so that the above becomes xs:string followed by xs:token. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Suman Kalia/Toronto/IBM@IBMCA, Cc: dfdl-wg(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 27/03/2013 16:38 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Suman Looking at the XML schema-for-schemas, and doing a test in the XSD editor in eclipse, XSD enumeration facets are modelled as xs:NMTOKEN and not xs:string, like DFDL enums. XSD is perfectly happy to strip/collapse white space. I think therefore that we should be doing the same for DFDL enum properties. I don't see any harm in this - an enum is a contiguous sequence of non-whitespace characters anyway, so any leading/trailing whitespace is harmless. Looks like XSD pattern facet is modelled as xs:string, preserving white space. We should do the same for DFDL regex properties. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 19/03/2013 17:41 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token The type called DFDLExpressionOrPatternOrNothing only makes sense for use in one place - the element value of an assert or discriminator. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia <kalia(a)ca.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 19/03/2013 17:29 Subject: Re: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Sent by: dfdl-wg-bounces(a)ogf.org Mike - I am not sure but my gut feeling is that it would start with the most restrictive one first. i.e If empty string ( assuming it has length facet 1) - would match Nothing , then xsd:token which is restricted form of xs:string. I think you are going to get string with white spaces collpsed ( xsd:token) if it not empty string. You can run few tests to see the behavior.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 03/19/2013 12:43 PM Subject: [DFDL-WG] whitespace in DFDL annotations: right now regex is xs:string, expression is xs:token Sent by: dfdl-wg-bounces(a)ogf.org This came up on the call. The schemas I have for DFDL annotations have DFDLRegularExpression as an xs:string, and DFDLExpression as an xs:token. I have no clue what a union of these types behaves like. But we have a union called DFDLExpressionOrPatternOrNothing which is a 3-way union of DFDLExpression, DFDLRegularExpression, and EmptyString (which is also derived from xs:string but has length facet of 0 as well. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] DFDL regular expressions and Unicode
by Cranford, Jonathan W. 08 Jul '13

08 Jul '13

Update: I just found errata 3.29, which answers this question, I think. >From the description in the errata, and looking at the documentation for java 7 regular expressions, it looks like DFDL regular expressions conform to level 1 of Unicode Regular expressions (UTS#18). I still think there would be value in stating such conformance in the DFDL spec, but I suppose that would take some legwork for someone to actually confirm the conformance of ICU and Java7 to level 1. Very respectfully, -- Jonathan Cranford >-----Original Message----- >From: Cranford, Jonathan W. >Sent: Friday, July 05, 2013 1:36 PM >To: dfdl-wg(a)ogf.org >Subject: DFDL regular expressions and Unicode > >I've been going through the spec recently, and I have a few questions about DFDL >regular expressions. > >Rather than put them into one long email, I'll break them up into separate emails. > >First question: What level of conformance to Unicode Technical Standard #18 >UNICODE > REGULAR EXPRESSIONS do DFDL regular expressions claim? > > For example, > * XML Schema regular expressions are "targeted at support of 'Level 1' >features" > (http://www.w3.org/TR/xmlschema-2/#dt-ccesN) > * Java 1.4 regular expressions "implement its second level of support" > (http://docs.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.html) > * Perl 5.18 seems to implement most of Level 1 > (http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression- >Support-Level) > > I think the conformance level should be specified in the DFDL spec so that it is >clear to schema > designers what a regular expression would really match against. Details > like case conversion and canonical equivalence make a difference when > matching against a Unicode string. > >Thanks in advance, > >-- >Jonathan W. Cranford <jcranford(a)mitre.org> >Senior Information Systems Engineer >The MITRE Corporation (http://www.mitre.org)

2 2

OGF DFDL WG Call Agenda 2013-07-09
by Steve Hanson 08 Jul '13

08 Jul '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13093?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] dfdl:calendarLanguage - should allow an expression?
by Steve Hanson 08 Jul '13

08 Jul '13

Agreed on 18th June call that additionally dfdl:calendarLanguage should be allowed to take an expression. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 29/05/2013 17:36 Subject: Re: [DFDL-WG] dfdl:calendarLanguage - should allow an expression? We already have expression capability for the following properties that are potentially locale/platform sensitive: encoding byteOrder binaryFloatRep outputNewLine textStandardDecimalSeparator textStandardGroupingSeparator textStandardExponentRep Properties binaryPackedSignCodes and textZonedSignStyle are platform sensitive, but in practice it's unlikely that a single schema would be created that needed to handle (say) packed numbers from both IBM z/OS or IBM iSeries, or zoned numbers from both IBM and Micro Focus COBOL compilers. I'm happy to leave these as they are for 1.0. Properties calendarTimeZone and calendarObserveDST could certainly be considered locale sensitive. I think properties like calendarFirstDayOfWeek and calendarDaysInFirstWeek and calendarCenturyStart less so. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 29/05/2013 16:35 Subject: Re: [DFDL-WG] dfdl:calendarLanguage - should allow an expression? This argument would apply to anything that has locale-specific behavior. are there any other properties that should be revisited this way as well? On Wed, May 29, 2013 at 4:58 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: DFDL has the design point that the combination of schema & data should give the same result regardless of locale and/or machine. To create a schema that does vary across locale or machine requires the user deliberately to use expressions for the relevant DFDL properties (encoding, decimal separator, etc) and parameterise the schema, using either built-in DFDL variables or user-defined variables. However the dfdl:calendarLanguage property is only defined as a String, I think it should also allow an expression as otherwise a schema is tied to a particular locale for it's calendar language support. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] What are the consequences of a failed assert? - resending with updates
by Steve Hanson 08 Jul '13

08 Jul '13

Errata 3.4 added the failureType attribute to dfdl:assert so that an assert could report the equivalent of a validation error and carry on ('recoverable error') instead of throwing a 'processing error'. What is being asked for here sounds like the ability to throw an error and stop the parser regardless of what points of uncertainty are in scope. That does sound like a genuinely useful thing to be able to do, and gives dfdl:assert the full range of failure actions. We'd have to create a new error type, say 'fatal error'. It would have the same effect as a 'schema definition error'. James, would this 'fatal error' give you the capability you are looking for? Or will Jonathan's suggestion be sufficient - it backs the parser out of Content-Type but what happens next depends on whether there is a higher point of uncertainty? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, "dfdl-wg-bounces(a)ogf.org" <dfdl-wg-bounces(a)ogf.org>, "Garriss Jr., James P." <jgarriss(a)mitre.org>, Steve Hanson/UK/IBM@IBMGB Date: 08/07/2013 09:29 Subject: RE: [DFDL-WG] What are the consequences of a failed assert? It depends on whether there was exactly one point of uncertainty active when the discriminator expression evaluated to 'true'. Points of uncertainty can be nested within one another. A POU can be resolved ( or made 'inactive' as I was trying to hint at ) by a discriminator that evaluates to 'true'. When a processing error occurs, it is caught by the nearest ( most deeply nested ) *active* POU. If there is no active POU then the parser halts and reports the error. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "Garriss Jr., James P." <jgarriss(a)mitre.org>, Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, "dfdl-wg-bounces(a)ogf.org" <dfdl-wg-bounces(a)ogf.org> Date: 05/07/2013 20:31 Subject: RE: [DFDL-WG] What are the consequences of a failed assert? Wouldn’t the use of a discriminator suppress the backtracking behavior in the face of a processing error? That is, if a discriminator is used to confirm that the Content-Type header branch is the correct one, wouldn’t a subsequent assert that causes a processing error force the parser to halt with an error? -Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Garriss Jr., James P. Sent: Friday, July 05, 2013 12:09 PM To: Steve Hanson; Tim Kimber Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Very helpful, guys, as always. Thank you. Question: Is it possible to force the failure of an assert to be a validation error instead of a processing error? Here’s why: All the email headers are modeled as an unordered list. The last item on the list is a catch-all, designed to scoop up all the unexpected headers and hide them from the infoset (via hiddenGroupRef). So if I have an Content-Type header that is obviously invalid: Content-Type: bogus/data The assert fails as expected. But since there is always another choice, the catch-all, the Content-Type header quietly disappears. What I really want is for there to be a validation error for “bogus/data,” so that processing stops! Is that possible? Is that reasonable? From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Steve Hanson Sent: Thursday, July 04, 2013 4:26 AM To: Tim Kimber Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Almost. A Recoverable Error does not cause backtracking. It was added to enable an assert to perform validation that checked the data stream rather than just the infoset. For example, checking the max physical length of a non-string value. So it has the same effect on the parse as a Validation Error. I've corrected your summary. The definition and description of the different errors is covered in the spec by section 2. There have been some errata in this section to clarify the behaviour. Some of the wording has been around since early drafts of the spec, so any suggestions for improvement are welcome. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 03/07/2013 21:51 Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Sent by: dfdl-wg-bounces(a)ogf.org There are four types of error in DFDL: - a Schema Definition Error : The schema itself is not valid. ( at least three kinds: xsd not valid, xsd not in the DFLD subset, DFDL annotations not following the rules ) - A Processing Error : The data cannot be parsed. Or if unparsing, the info set cannot be unparsed. - A Recoverable Error : ( see the Errata ). This is effectively a user-defined form of Validation Error, raised while executing a DFDL assert, - A Validation Error : The info set does not conform to the XSD A Schema Definition Error is immediately fatal. Most of these can be detected by the processor before parsing/unparsing begins. A Processing Error or a Recoverable Error will cause the parser to suppress the error and backtrack to the nearest point of uncertainty. So it will only stop the parse if there are no points of uncertainty currently active. A Recoverable Error does not cause backtracking - the parser continues to parse after reporting the error. A Validation Error is only reported if validation is enabled in the DFDL processor. It does not cause backtracking - the parser continues to parse after reporting the error. That's the gist of it. Further details from other WG members may follow shortly, depending on how accurate I have managed to be. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 03/07/2013 19:06 Subject: [DFDL-WG] What are the consequences of a failed assert? Sent by: dfdl-wg-bounces(a)ogf.org I have an element with an assert, <xsd:element name="Type" dfdl:inputValueCalc="{ fn:lower-case(../MixedCaseType) }"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="The type must match one of the values on the enumerated list."/> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="application"/> <xsd:enumeration value="multipart"/> <xsd:enumeration value="message"/> <xsd:enumeration value="text"/> </xsd:restriction> </xsd:simpleType> </xsd:element> and the assert is failing (as it should in this case!). Parse Error: Assertion failed. The type must match one of the values on the enumerated list. What are the consequences of a failed assert? I have an old version of the spec—is there a place to a get a current, complete copy?—but it says “An unsuccessful dfdl:assert causes a processing error.” 1. What does “processing error” mean in English? 2. Does it mean the input is invalid? 3. Does it mean the processor should stop here and go no further? 4. Does it mean the process should simply ignore the problem and move on to the next item in the schema? TIA -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] regex free-spacing mode
by Steve Hanson 08 Jul '13

08 Jul '13

Andy, did you test both ICU4J and ICU4C ? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 27/06/2013 17:18 Subject: Re: [DFDL-WG] regex free-spacing mode Hi Mike (et al), I've gone back to the ICU doc for this and run a few tests locally. It looks like both cases for non capturing groups can now be used in Java 7 and ICU 51.1. In other words, both of the following constructs are supported: (?imsx-imsx) (?imsx-imsx:X) So the quick answer is that what you are trying to do in your example below is supported. The long answer is that errata 3.29 can probably be updated by removing the restriction on (?imsx-imsx:X), as below 3.29 Sections 24 and 30. The DFDL specification is not prescriptive enough when specifying what is allowed for regular expressions used in the length property and testPattern property. Section 24 is replaced by the following. "A DFDL regular expression may be specified for the dfdl:lengthPattern format property and the dfdl:testPattern attribute of the dfdl:assert and dfdl:discriminator annotations. DFDL regular expressions do not interpret DFDL entities. A DFDL regular expression is defined by a set of valid pattern characters. For portability, a DFDL regular expression pattern is restricted to the inclusive subset of the ICU regular expression [ICURE] and the Java(R) 7 regular expression [JAVARE] with the Unicode flags UNICODE_CASE and UNICODE_CHARACTER_CLASS turned on. The following regular expression constructs are not common to both ICU and Java(R) 7 and it is a schema definition error if any are used in a DFDL regular expression: *Construct* *Meaning* *Notes* \N{UNICODE CHARACTER NAME} Match the named character ICU only \X Match a Grapheme Cluster ICU only \Uhhhhhhhh Match the character with the hex value hhhhhhhh. ICU only (?# ... ) Free-format comment ICU only (?w-w) UREGEX_UWORD - Controls the behaviour of \b in ICU only a pattern. (?d-d) UNIX_LINES - Enables Unix lines mode. Java 7 only (?u-u) UNICODE_CASE - Enables Unicode-aware case folding. Java 7 only (1) (?U-U) UNICODE_CHARACTER_CLASS - Enables the Unicode Java 7 only (1) version of Predefined character classes and POSIX character classes. (?imsx-imsx:X) X, as a non-capturing group with the given flags. Java 7 only Note that the flags i,s,m,x are valid, but appending :X to the flag is not. Notes: (1) Implementations using Java 7 must set flag UNICODE_CASE by default to match ICU. (2) Implementations using Java 7 must set flag UNICODE_CHARACTER_CLASS by default to match ICU. Additionally, the behaviour of the word character construct (\w) is not consistent in ICU and Java 7. In Java 7 \w is [\p{Alpha}\p{gc=Mn}\p{gc=Me}\p{gc=Mc}\p{Digit}\p{gc=Pc}],which is a larger set than ICU where \w is [\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Nd}]. The use of \w is not recommended in DFDL regular expressions in conjunction with Unicode encodings, and an implementation must issue a warning if such usage is detected. Character properties are detailed by the Unicode Re gular Expressions [UNICODERE]." Section 30 is updated to correct the references used in section 24: -Add:[ICURE] - http://userguide.icu-project.org/strings/regexp -Add:[UNICODERE] - http://www.unicode.org/reports/tr18/ -Remove:[PERLRE] - http://perldoc.perl.org/perlre.html#Extended-Patterns -Change:[JAVARE] - http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html Cheers, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Steve Hanson/UK/IBM 27/06/2013 10:26 To Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] regex free-spacing mode Mike, I believe that is the case but I have copied Andy Edwards who is the person in the IBM DFDL team who added our regex support. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 26/06/2013 18:56 Subject: Re: [DFDL-WG] regex free-spacing mode Sent by: dfdl-wg-bounces(a)ogf.org To clarify, errata v13 has this in the table for erratum 3.29 in the list of non-portables: (?imsx-imsx:X) X, as a non-capturing group with the given flags. Note that the flags i,s,m,x are valid, but appending :X to the flag is not. Java 7 only I interpret this as meaning that only the so-called modifier-span notation (the : suffix) is disallowed, but not just plain (?x), but I wanted to be sure that was the correct interpretation. On Wed, Jun 26, 2013 at 1:13 PM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: I wrote this complicated regex today and it works in Daffodil. Question is this. Is the (?x) which turns on regex free-spacing mode, officially supported in DFDL? You can see from below that it is VERY desirable that it works..... <xs:simpleType name="frontMatterType"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:simpleType lengthKind="pattern" terminator="%FF;"> <dfdl:property name="lengthPattern"><![CDATA[(?x) # regex free spacing mode # # match the front matter of the document # .{1,8192}? # up to 8K of front matter content # # front matter ends at the first message description page # (?= # lookahead (followed by but not including...) \f # a formfeed character (?> \s | \x08 ){1,100}? # whitespace or backspace (x08) MESSAGE\ DESCRIPTION\r # this literal text \s{1,100}? # up to 100 whitespaces -{19}\r # exactly 19 hyphens and a CR ) # end lookahead ]]></dfdl:property> </dfdl:simpleType> </xs:appinfo> </xs:annotation> <xs:restriction base="xs:string" /> </xs:simpleType> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

Re: [DFDL-WG] What are the consequences of a failed assert?
by Cranford, Jonathan W. 08 Jul '13

08 Jul '13

Wouldn’t the use of a discriminator suppress the backtracking behavior in the face of a processing error? That is, if a discriminator is used to confirm that the Content-Type header branch is the correct one, wouldn’t a subsequent assert that causes a processing error force the parser to halt with an error? -Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Garriss Jr., James P. Sent: Friday, July 05, 2013 12:09 PM To: Steve Hanson; Tim Kimber Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Very helpful, guys, as always. Thank you. Question: Is it possible to force the failure of an assert to be a validation error instead of a processing error? Here’s why: All the email headers are modeled as an unordered list. The last item on the list is a catch-all, designed to scoop up all the unexpected headers and hide them from the infoset (via hiddenGroupRef). So if I have an Content-Type header that is obviously invalid: Content-Type: bogus/data The assert fails as expected. But since there is always another choice, the catch-all, the Content-Type header quietly disappears. What I really want is for there to be a validation error for “bogus/data,” so that processing stops! Is that possible? Is that reasonable? From: dfdl-wg-bounces(a)ogf.org<mailto:dfdl-wg-bounces@ogf.org> [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Steve Hanson Sent: Thursday, July 04, 2013 4:26 AM To: Tim Kimber Cc: dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>; dfdl-wg-bounces(a)ogf.org<mailto:dfdl-wg-bounces@ogf.org> Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Almost. A Recoverable Error does not cause backtracking. It was added to enable an assert to perform validation that checked the data stream rather than just the infoset. For example, checking the max physical length of a non-string value. So it has the same effect on the parse as a Validation Error. I've corrected your summary. The definition and description of the different errors is covered in the spec by section 2. There have been some errata in this section to clarify the behaviour. Some of the wording has been around since early drafts of the spec, so any suggestions for improvement are welcome. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>, Date: 03/07/2013 21:51 Subject: Re: [DFDL-WG] What are the consequences of a failed assert? Sent by: dfdl-wg-bounces(a)ogf.org<mailto:dfdl-wg-bounces@ogf.org> ________________________________ There are four types of error in DFDL: - a Schema Definition Error : The schema itself is not valid. ( at least three kinds: xsd not valid, xsd not in the DFLD subset, DFDL annotations not following the rules ) - A Processing Error : The data cannot be parsed. Or if unparsing, the info set cannot be unparsed. - A Recoverable Error : ( see the Errata ). This is effectively a user-defined form of Validation Error, raised while executing a DFDL assert, - A Validation Error : The info set does not conform to the XSD A Schema Definition Error is immediately fatal. Most of these can be detected by the processor before parsing/unparsing begins. A Processing Error or a Recoverable Error will cause the parser to suppress the error and backtrack to the nearest point of uncertainty. So it will only stop the parse if there are no points of uncertainty currently active. A Recoverable Error does not cause backtracking - the parser continues to parse after reporting the error. A Validation Error is only reported if validation is enabled in the DFDL processor. It does not cause backtracking - the parser continues to parse after reporting the error. That's the gist of it. Further details from other WG members may follow shortly, depending on how accurate I have managed to be. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com<mailto:kimbert@uk.ibm.com> Tel. 01962-816742 Internal tel. 37246742 From: "Garriss Jr., James P." <jgarriss(a)mitre.org<mailto:jgarriss@mitre.org>> To: "dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>" <dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>>, Date: 03/07/2013 19:06 Subject: [DFDL-WG] What are the consequences of a failed assert? Sent by: dfdl-wg-bounces(a)ogf.org<mailto:dfdl-wg-bounces@ogf.org> ________________________________ I have an element with an assert, <xsd:element name="Type" dfdl:inputValueCalc="{ fn:lower-case(../MixedCaseType) }"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="The type must match one of the values on the enumerated list."/> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="application"/> <xsd:enumeration value="multipart"/> <xsd:enumeration value="message"/> <xsd:enumeration value="text"/> </xsd:restriction> </xsd:simpleType> </xsd:element> and the assert is failing (as it should in this case!). Parse Error: Assertion failed. The type must match one of the values on the enumerated list. What are the consequences of a failed assert? I have an old version of the spec—is there a place to a get a current, complete copy?—but it says “An unsuccessful dfdl:assert causes a processing error.” 1. What does “processing error” mean in English? 2. Does it mean the input is invalid? 3. Does it mean the processor should stop here and go no further? 4. Does it mean the process should simply ignore the problem and move on to the next item in the schema? TIA -- dfdl-wg mailing list dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org> https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU-- dfdl-wg mailing list dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org> https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

DFDL regular expressions: using Unicode features to match against non-Unicode encoding
by Cranford, Jonathan W. 05 Jul '13

05 Jul '13

Next question: What should happen if a regular expression uses a Unicode block or category to match against a string in a non-Unicode encoding? Should that be a Schema Definition Error, or should the regular expression just silently fail to match anything? I would prefer a Schema Definition Error, even though detecting such a condition would be difficult. For example: <?xml encoding="UTF-8"?> ... <xs:element name="foo" type="xs:string" dfdl:encoding="Shift_JIS"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert testKind="pattern" testPattern="\p{InGreek}"/> </xs:appinfo> </xs:annotation> </xs:element> ... \p{InGreek} matches a character in the Greek Unicode block, but such a category is incongruent with the Shift_JIS encoding. In short, any Unicode block or category would not make sense against a non-Unicode encoding; for example, \p{Lu} matches a character in the uppercase letter category, but the list of Unicode characters in that category cannot be easily compared to a Shift_JIS encoding. Thanks in advance, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

1 0

matching DFDL regular expressions against different encoding
by Cranford, Jonathan W. 05 Jul '13

05 Jul '13

Next question: What does it mean to match a regular expression written in one encoding system against a different encoding system? For example: <?xml encoding="UTF-8"?> ... <xs:element name="foo" type="xs:string" dfdl:encoding="Shift_JIS"> <xs:annotation> <xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:assert testKind="pattern" testPattern="\\"/> </xs:appinfo> </xs:annotation> </xs:element> ... The only interpretation that makes sense to me is that the regular expression is *always* expressed in the encoding of the XML schema (UTF-8 in this case), since that’s what any XML parser will do; that is, any XML parser will interpret all element and attribute content according to the encoding of the XML file. Otherwise, to have a regular expression expressed in the target encoding represented by dfdl:encoding (Shift_JIS in this example), you could not use an XML parser to parse the DFDL schema; you would have to use a custom parser that applies a *different* encoding to select attributes or elements in the DFDL schema, and that just seems contrary to the spirit and intent of DFDL. However, this could lead to some very interesting problems. A single string, testPattern, has to be compiled as a regular expression *and* converted to match another encoding. The biggest question, I think, is how character literals and character classes in the regular expression should be treated. Here, if you map the raw bytes, \ in UTF-8 is 0x5C, which maps to the yen character (\) in Shift_JIS. That’s one approach. Another approach is to map the logical character to its equivalent encoding in Shift_JIS. Backslash (\) is 0x825F in Shift_JIS. To do this across the board, the underlying library would need a pretty sophisticated mapping for each encoding. So should the above match a yen character (0x5C) in the foo element? Or should it match the backslash character (0x825F) in the foo element? More complicated example using multi-byte characters: <?xml encoding="UTF-8"?> ... <xs:element name="foo" type="xs:string" dfdl:encoding="Shift_JIS"> <xs:annotation> <xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:assert testKind="pattern" testPattern="たながさん"/> </xs:appinfo> </xs:annotation> </xs:element> ... The above characters in testPattern map to たなかさん in Unicode. How should this match work? a. The code points can’t be converted directly to Shift_JIS, as it doesn’t represent a valid Shift_JIS encoding. The bytes that correspond to a UTF-8 encoding of that same string also don’t represent a valid Shift_JIS encoding. In neither case can the bytes be simply treated as Shift_JIS, so this approach should cause a processing error. b. Mapping each character to its logical equivalent would work, but the underlying library would have to know how to map each character from one encoding to another, and sometimes a single character from one encoding can be mapped to more than character in another encoding. In this case, a single mapping does exist; the equivalent characters in Shift_JIS encoding are encoded as 82BD 82C8 82A9 82B3 82F1. Thoughts? -- Jonathan W. Cranford <jcranford(a)mitre.org> Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

1 0

DFDL regular expressions and Unicode
by Cranford, Jonathan W. 05 Jul '13

05 Jul '13

I've been going through the spec recently, and I have a few questions about DFDL regular expressions. Rather than put them into one long email, I'll break them up into separate emails. First question: What level of conformance to Unicode Technical Standard #18 UNICODE REGULAR EXPRESSIONS do DFDL regular expressions claim? For example, * XML Schema regular expressions are "targeted at support of 'Level 1' features" (http://www.w3.org/TR/xmlschema-2/#dt-ccesN) * Java 1.4 regular expressions "implement its second level of support" (http://docs.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.html) * Perl 5.18 seems to implement most of Level 1 (http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression-Support…) I think the conformance level should be specified in the DFDL spec so that it is clear to schema designers what a regular expression would really match against. Details like case conversion and canonical equivalence make a difference when matching against a Unicode string. Thanks in advance, -- Jonathan W. Cranford <jcranford(a)mitre.org> Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

1 0