- dfdl-wg - lists.ogf.org

for errata r14 - Fwd: DFDL regular expressions and Unicode - conformance
by Mike Beckerle 24 Jul '13

24 Jul '13

I am assuming this issue will get handled as part of a r14 erratum. ---------- Forwarded message ---------- From: Steve Hanson <smh(a)uk.ibm.com> Date: Tue, Jul 16, 2013 at 11:13 AM Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode - conformance To: Andrew Edwards <andy.edwards(a)uk.ibm.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Jonathan No need for us to contact ICU, as Andy indicates below ICU and Java both claim conformance. Here's the words from errata 3.29. Please can you rephrase to combine the conformance requirement and the restrictions, so that we end up with a form you are happy with, then we can update the errata? *A DFDL regular expression is defined by a set of valid pattern characters. For portability, a DFDL regular expression pattern is restricted to the inclusive subset of the ICU regular expression [ICURE] and the Java(R) 7 regular expression [JAVARE] with the Unicode flags UNICODE_CASE and UNICODE_CHARACTER_CLASS turned on.* Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, "Cranford, Jonathan W." <jcranford(a)mitre.org> Date: 11/07/2013 14:19 Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode ------------------------------ Hi Jonathan, Sorry for the delay; first week back in the office... As you've noted, errata 3.29 describes what DFDL regexes are supported. Specifically, it is a subset of Java 7's java.util.regex ( http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html) and ICU's regular expression support ( http://userguide.icu-project.org/strings/regexp) both of which conform with level 1 of Unicode technical standard #18 It looks like there are 2 stages to checking conformance: - Logical - do the available regex constructs provide conformance to the technical standard. This is probably just a couple of hours of reading the Unicode standard rules and cross-checking the constructs in each matching engine. - Actual - do Java 7 and ICU really match properly for each of the conformance statements. This can take an ever increasing amount of time testing various sets of data and regex patterns, and it risks the only reward being that we find bugs in Java 7 or ICU. Minimum would be 3 or 4 days of test generation. Does that answer the issue? Andy *Andy Edwards* - *IBM Integration Bus*<http://www-03.ibm.com/software/products/us/en/integration-bus>- *DFDL*<https://w3-connections.ibm.com/wikis/home?lang=en-gb#!/wiki/IBM%20Data%20Fo…> *Email:* *andy.edwards(a)uk.ibm.com* <andy.edwards(a)uk.ibm.com> *Snail Mail:* MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN *Tel int:* 247222 *Tel ext:* +44 (0)1962 817222 *Desk:* DE3 V17 *The Feynman problem solving Algorithm* 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU *Steve Hanson/UK/IBM* 08/07/2013 11:08 To "Cranford, Jonathan W." <jcranford(a)mitre.org>, cc "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] DFDL regular expressions and UnicodeLink<Notes://D06ML014/80256D7F004ED63A/38D46BF5E8F08834852564B500129B2C/8054F31FB22A8880A1C918FA98057ED6> Jonathan I've copied Andy who added regexs support into IBM DFDL recently. He might have an idea as to the effort involved in stating conformance. We will discuss your other two emails on next DFDL-WG call or so. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 06/07/2013 00:56 Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode Sent by: dfdl-wg-bounces(a)ogf.org ------------------------------ Update: I just found errata 3.29, which answers this question, I think. >From the description in the errata, and looking at the documentation for java 7 regular expressions, it looks like DFDL regular expressions conform to level 1 of Unicode Regular expressions (UTS#18). I still think there would be value in stating such conformance in the DFDL spec, but I suppose that would take some legwork for someone to actually confirm the conformance of ICU and Java7 to level 1. Very respectfully, -- Jonathan Cranford >-----Original Message----- >From: Cranford, Jonathan W. >Sent: Friday, July 05, 2013 1:36 PM >To: dfdl-wg(a)ogf.org >Subject: DFDL regular expressions and Unicode > >I've been going through the spec recently, and I have a few questions about DFDL >regular expressions. > >Rather than put them into one long email, I'll break them up into separate emails. > >First question: What level of conformance to Unicode Technical Standard #18 >UNICODE > REGULAR EXPRESSIONS do DFDL regular expressions claim? > > For example, > * XML Schema regular expressions are "targeted at support of 'Level 1' >features" > (http://www.w3.org/TR/xmlschema-2/#dt-ccesN) > * Java 1.4 regular expressions "implement its second level of support" > ( http://docs.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.html) > * Perl 5.18 seems to implement most of Level 1 > ( http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression- >Support-Level) > > I think the conformance level should be specified in the DFDL spec so that it is >clear to schema > designers what a regular expression would really match against. Details > like case conversion and canonical equivalence make a difference when > matching against a Unicode string. > >Thanks in advance, > >-- >Jonathan W. Cranford <jcranford(a)mitre.org> >Senior Information Systems Engineer >The MITRE Corporation (http://www.mitre.org) -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

Broadcast: OGF DFDL WG Extra Call (24 Jul 15:00 GDT in No room booked)
by Steve Hanson 24 Jul '13

24 Jul '13

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

questions/feedback on dfdl:assert
by Cranford, Jonathan W. 23 Jul '13

23 Jul '13

DFDL WG, Some questions and feedback on Section 7.3.1 Properties for dfdl:assert. I don't think any of these are major issues. > "Schema authors can insert xs:sequence constructs to control the timing of evaluation of statements more precisely." An example here would illustrate and clarify the concept. This same verbiage is used in other sections of the spec, but I would think that only a single example would be sufficient to demonstrate the technique. Under the "test" property: > "The expression must have been evaluated by the time this element and it descendents have been processed." I'm lost in the grammar here. Is that equivalent to "The expression must be evaluated before this element and its descendants are fully processed"? (Changes highlighted). Also, should the spec define what it means to "process" an element for this statement to have any weight? > "It is a schema definition error if dfdl:test is the empty string and the value is not specified and dfdl:testKind is 'expression' or not specified." Huh? If the value of what isn't specified? The value of the dfdl:assert annotation? Under the "testPattern" property: > "It is a schema definition error if dfdl:testPattern is the empty string and the value is not specified and dfdl:testKind is 'pattern'." Same comment here. Should this be "... and the value of the dfdl:assert annotation is not specified..."? > "In order for a testPattern to be used, the data subject to the pattern must be scannable using a DFDL regular expression otherwise the results are not predictable." What does it mean for data to be "scannable"? Does this mean that the representation has to be text, since it's the physical representation of the data that is compared to the regular expression? Very respectfully, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

3 2

OGF DFDL WG Call Minutes 2013-07-23
by Steve Hanson 23 Jul '13

23 Jul '13

Please find minutes from the above call attached. These are intermediate minutes as there is a follow-on call Wed 24th @ 3pm UK / 10am Eastern. Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

issue: scannable and 'results are not predictable'
by Mike Beckerle 23 Jul '13

23 Jul '13

I was editing the definition of scannable into the glossary and when I looked at usage of 'scannable' in testPattern I found this: In the box for testPattern it says if the data is not scannable "the results are not predictable". Is that sufficient? We can sometimes statically determine that the schema says the data should all be scannable (e.g., no change of encoding, no binary elements), and that would rule out one non-predictability. So, if data is non-scannable in the sense that the schema contains say, a binary element, we can issue an SDE if lengthKind is pattern or a testPattern assert is being used. We could also SDE if runtime-valued encoding properties are used and the encoding changes inside a scannable context. Well, I guess testKind pattern asserts/discriminators are an issue because they may look only at the first part of the data of a complex component, so they don't require everything to be scannable, only the part the regex actually examines. So in this case it's user-beware, and if non-scannable I suppose we could issue a warning. But the spec does not say this is an SDE or warning currently. It just says results are not predictable. There is also the fact that the data might be broken, i.e., the schema might say the data is scannable, but at parse time character decode errors occur. I believe our policy on this is that these cause processing errors. This really is orthogonal to scannable, which is a property of a schema component. Comments? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 4

OGF DFDL WG Call Agenda 2013-07-23
by Steve Hanson 22 Jul '13

22 Jul '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13097?download= Note earlier start time of 2:30 UK (9:30 Eastern) Same finish time. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL WG Call (23 Jul 14:30 GDT in DE3S09)
by Steve Hanson 22 Jul '13

22 Jul '13

Rescheduled: OGF DFDL WG Call 23/07/2013 - (Repeats) Location: DE3S09 Steve Hanson has rescheduled this meeting. You have not added this entry to your calendar. Repeats This entry repeats Required: mbeckerle.dfdl(a)gmail.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org, Stephanie Fetzer/Charlotte/IBM@IBMUS OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

Fw: OGF DFDL WG Call
by Steve Hanson 22 Jul '13

22 Jul '13

For this week only I'd like to start earlier at either 2pm or 2:30pm UK (9am or 9:30am Eastern) and finish at usual time, because we are not getting through the agenda items. Key attendees: please let me know if that is ok and which time you can make. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2013 13:36 ----- OGF DFDL WG Call Tue 08/01/2013 15:00 - 16:00 (Repeats) Location: DE3S09 Required: mbeckerle.dfdl(a)gmail.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org Repeats: OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Fw: Unordered Groups : new draft
by Steve Hanson 22 Jul '13

22 Jul '13

Hopefully we can close on this week's WG call. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2013 13:13 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Date: 18/07/2013 16:07 Subject: Re: Unordered Groups : new draft Tim That looks great. I've made a few minor editorial changes so it can be dropped straight into the errata document and then the spec with no further tweaks. Also added that occurrences of the same member need not be contiguous, and replaced 'real infoset' with 'infoset implied by the original unordered group'. You didn't incorporate Mike's review comment? "If a member element is found to have the empty representation then the parsing of that element must use the original value of minOccurs. In this example, element "b" has minOccurs="0" and if it is found with the empty representation then it must not be defaulted." should perhaps be: "If a member element is found to have the empty representation, then the decision about whether a default value is to be computed is based on the original value of minOccurs, and the count of how many other occurrences of that member element have been accumulated so far by the parsing. In this example, element "b" has minOccurs="0" and if it is found with the empty representation then it must not be defaulted." Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com>, Date: 18/07/2013 11:38 Subject: Unordered Groups : new draft I've addressed Steve last round of comments, and improved the wording of the introductory paragraph. [attachment "Section 14.3 - Unordered Sequence Groups.doc" deleted by Steve Hanson/UK/IBM] regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

proposed dfdl:hexBinary function for hex constants in schemas
by Mike Beckerle 22 Jul '13

22 Jul '13

Separated out from prior proposal where it was mixed in with another topic. dfdl:hexBinary($arg as xs:anyAtomicType?) as xs:hexBinary? This function provides an extension to the behavior of the xs:hexBinary constructor. The argument can be a string, in which case the behavior is as the xs:hexBinary constructor. The argument can also be a long, unsignedLong, or any subtype thereof, and in that case a string containing hexadecimal digits is produced. The hexadecimal digits produced that are letters are always uppercase. The number of hex digits in the resulting string is a function of the input type. If byte or unsigned byte, exactly 2 hex digits are produced, for short and unsignedShort, 4 hex digits, and so on. The hex digits correspond to a big-endian representation of a twos-complement binary representation of the argument value. So: dfdl:short(xs:concat('x', dfdl:hexBinary(xs:short(208)))) eq xs:short(208) is true, and a corresponding tautology holds for all the other DFDL functions that construct integers from hexadecimal, if you replace the 208 above with a value in range for the corresponding numeric type. If the argument is a numeric literal, then the smallest signed integer type (long, int, short, byte) is selected that can contain the value, and the number of hexadecimal digits produced corresponds to that type. So: dfdl:hexBinary(208) eq 'D0' dfdl:hexBinary(-2084) eq 'F7FF' -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 3