- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language
by Tim Kimber 24 Jul '13

24 Jul '13

Interesting...that never occurred to me. I have always considered the # to be a qualifier on the XML character reference entity. In XML, the carriage-return character can be specified in any of the following ways: &cr; - A named entity  - A numeric entity ( starts with '#' )  - A numeric entity in base 16 ( the numeric part starts with 'x' ) So the '#' character is signalling that the entity is a numeric entity. But there's no need for that to be signalled in this case - the parameter for these integer constructors will always be a number. On that basis, the prefix should be 'x'. I would prefer that to '#x' on the grounds of readability. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 20/06/2013 14:01 Subject: Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language I was thinking along the same lines as Tim. I'd prefer the hex string to start with '#x' for consistency with DFDL hex entity syntax. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 20/06/2013 09:44 Subject: Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language Sent by: dfdl-wg-bounces(a)ogf.org As an alternative, we could - drop the 'h' prefix - use the same names as the XPath integer constructors, including the use of the full name for unsigned variants ( e.g. 'dfdl:unsignedInt' ) - behave exactly like xs:byte / xs:short etc when the string does not begin with '0x' The only difference would be the namespace of the function. So the modeller would use the dfdl variant if they needed the '0x' notation, and they could choose to use the dfdl variant always when constructing integers from string literals. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 19/06/2013 23:28 Subject: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language Sent by: dfdl-wg-bounces(a)ogf.org Rationale: in writing DFDL schemas for binary data formats, there is a strong need to express binary data constants in hexadecimal. Proposed new functions: dfdl:hByte dfdl:hUByte dfdl:hShort dfdl:hUShort dfdl:hInt dfdl:hUInt dfdl:hLong dfdl:hULong The prefix 'h' is a reminder that the function converts from hexadecimal. The prefix 'U' on the name denotes that the function creates an unsigned type result. Normally one might like to write this out fully, but in this case brevity is helpful given the expected usage of these functions to construct literal constants in DFDL expressions. All the functions take a single string argument. The string must begin with "0x" and contain at least one hex digit after that. The string must contain only hexadecimal digits, that is, the characters 0-9a-zA-Z. It is a schema definition error otherwise. The hex digits represent a big endian twos complement representation of a binary number. Each function has a limit on the number of hex digits, with no more digits than 2, 4, 8, or 16 for the byte, short, int, and long versions. That is to say for dfdl:byte and dfdl:uByte, there can be at most 2 hex digits. For dfdl:short and dfdl:uShort, there can be at most 4 hex digits, and so on. It is a schema definition error if more digits are encountered than are suitable for the type being created. Examples: dfdl:hUInt("0xa1b2c3d4") is the value 2712847316. Note that in the first byte 'a1' the most significant bit is set, but since this is an unsigned type, this is not interpreted as a sign bit. dfdl:hInt("0xFFFFFFFF") is the int value -1. The sign bit indicates that the number is negative, and this twos complement value represents -1. dfdl:hUByte("0xFF") is the unsigned byte value 255 dfdl:hByte("0xff") is the byte value -1. dfdl:hByte("0x7F") is the byte value 127 dfdl:hByte("0x80") is the byte value -128 dfdl:hUByte("0x80") is the unsigned byte value 128 dfdl:hByte("0x0A3") is a schema definition error as the leading zero is not allowed because at most 2 digits are allowed for byte types. dfdl:hShort("0x0A3") is short value 208. Leading zero causes no issue here because up to 4 digits are allowed. Example of usage in expressions: <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:setVariable ref="ex:bOrd"> { if (xs:unsignedInt(.) eq dfdl:hUInt('0xa1b2c3d4')) then 'bigEndian' else if (xs:unsignedInt(.) eq dfdl:hUInt('0xd4c3b2a1')) then 'littleEndian' else fn:error(ex:magic_number, "Magic number was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).") } </dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 6

for errata r14 - Fwd: DFDL regular expressions and Unicode - conformance
by Mike Beckerle 24 Jul '13

24 Jul '13

I am assuming this issue will get handled as part of a r14 erratum. ---------- Forwarded message ---------- From: Steve Hanson <smh(a)uk.ibm.com> Date: Tue, Jul 16, 2013 at 11:13 AM Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode - conformance To: Andrew Edwards <andy.edwards(a)uk.ibm.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Jonathan No need for us to contact ICU, as Andy indicates below ICU and Java both claim conformance. Here's the words from errata 3.29. Please can you rephrase to combine the conformance requirement and the restrictions, so that we end up with a form you are happy with, then we can update the errata? *A DFDL regular expression is defined by a set of valid pattern characters. For portability, a DFDL regular expression pattern is restricted to the inclusive subset of the ICU regular expression [ICURE] and the Java(R) 7 regular expression [JAVARE] with the Unicode flags UNICODE_CASE and UNICODE_CHARACTER_CLASS turned on.* Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, "Cranford, Jonathan W." <jcranford(a)mitre.org> Date: 11/07/2013 14:19 Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode ------------------------------ Hi Jonathan, Sorry for the delay; first week back in the office... As you've noted, errata 3.29 describes what DFDL regexes are supported. Specifically, it is a subset of Java 7's java.util.regex ( http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html) and ICU's regular expression support ( http://userguide.icu-project.org/strings/regexp) both of which conform with level 1 of Unicode technical standard #18 It looks like there are 2 stages to checking conformance: - Logical - do the available regex constructs provide conformance to the technical standard. This is probably just a couple of hours of reading the Unicode standard rules and cross-checking the constructs in each matching engine. - Actual - do Java 7 and ICU really match properly for each of the conformance statements. This can take an ever increasing amount of time testing various sets of data and regex patterns, and it risks the only reward being that we find bugs in Java 7 or ICU. Minimum would be 3 or 4 days of test generation. Does that answer the issue? Andy *Andy Edwards* - *IBM Integration Bus*<http://www-03.ibm.com/software/products/us/en/integration-bus>- *DFDL*<https://w3-connections.ibm.com/wikis/home?lang=en-gb#!/wiki/IBM%20Data%20Fo…> *Email:* *andy.edwards(a)uk.ibm.com* <andy.edwards(a)uk.ibm.com> *Snail Mail:* MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN *Tel int:* 247222 *Tel ext:* +44 (0)1962 817222 *Desk:* DE3 V17 *The Feynman problem solving Algorithm* 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU *Steve Hanson/UK/IBM* 08/07/2013 11:08 To "Cranford, Jonathan W." <jcranford(a)mitre.org>, cc "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] DFDL regular expressions and UnicodeLink<Notes://D06ML014/80256D7F004ED63A/38D46BF5E8F08834852564B500129B2C/8054F31FB22A8880A1C918FA98057ED6> Jonathan I've copied Andy who added regexs support into IBM DFDL recently. He might have an idea as to the effort involved in stating conformance. We will discuss your other two emails on next DFDL-WG call or so. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 06/07/2013 00:56 Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode Sent by: dfdl-wg-bounces(a)ogf.org ------------------------------ Update: I just found errata 3.29, which answers this question, I think. >From the description in the errata, and looking at the documentation for java 7 regular expressions, it looks like DFDL regular expressions conform to level 1 of Unicode Regular expressions (UTS#18). I still think there would be value in stating such conformance in the DFDL spec, but I suppose that would take some legwork for someone to actually confirm the conformance of ICU and Java7 to level 1. Very respectfully, -- Jonathan Cranford >-----Original Message----- >From: Cranford, Jonathan W. >Sent: Friday, July 05, 2013 1:36 PM >To: dfdl-wg(a)ogf.org >Subject: DFDL regular expressions and Unicode > >I've been going through the spec recently, and I have a few questions about DFDL >regular expressions. > >Rather than put them into one long email, I'll break them up into separate emails. > >First question: What level of conformance to Unicode Technical Standard #18 >UNICODE > REGULAR EXPRESSIONS do DFDL regular expressions claim? > > For example, > * XML Schema regular expressions are "targeted at support of 'Level 1' >features" > (http://www.w3.org/TR/xmlschema-2/#dt-ccesN) > * Java 1.4 regular expressions "implement its second level of support" > ( http://docs.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.html) > * Perl 5.18 seems to implement most of Level 1 > ( http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression- >Support-Level) > > I think the conformance level should be specified in the DFDL spec so that it is >clear to schema > designers what a regular expression would really match against. Details > like case conversion and canonical equivalence make a difference when > matching against a Unicode string. > >Thanks in advance, > >-- >Jonathan W. Cranford <jcranford(a)mitre.org> >Senior Information Systems Engineer >The MITRE Corporation (http://www.mitre.org) -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

Broadcast: OGF DFDL WG Extra Call (24 Jul 15:00 GDT in No room booked)
by Steve Hanson 24 Jul '13

24 Jul '13

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

questions/feedback on dfdl:assert
by Cranford, Jonathan W. 23 Jul '13

23 Jul '13

DFDL WG, Some questions and feedback on Section 7.3.1 Properties for dfdl:assert. I don't think any of these are major issues. > "Schema authors can insert xs:sequence constructs to control the timing of evaluation of statements more precisely." An example here would illustrate and clarify the concept. This same verbiage is used in other sections of the spec, but I would think that only a single example would be sufficient to demonstrate the technique. Under the "test" property: > "The expression must have been evaluated by the time this element and it descendents have been processed." I'm lost in the grammar here. Is that equivalent to "The expression must be evaluated before this element and its descendants are fully processed"? (Changes highlighted). Also, should the spec define what it means to "process" an element for this statement to have any weight? > "It is a schema definition error if dfdl:test is the empty string and the value is not specified and dfdl:testKind is 'expression' or not specified." Huh? If the value of what isn't specified? The value of the dfdl:assert annotation? Under the "testPattern" property: > "It is a schema definition error if dfdl:testPattern is the empty string and the value is not specified and dfdl:testKind is 'pattern'." Same comment here. Should this be "... and the value of the dfdl:assert annotation is not specified..."? > "In order for a testPattern to be used, the data subject to the pattern must be scannable using a DFDL regular expression otherwise the results are not predictable." What does it mean for data to be "scannable"? Does this mean that the representation has to be text, since it's the physical representation of the data that is compared to the regular expression? Very respectfully, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

3 2

OGF DFDL WG Call Minutes 2013-07-23
by Steve Hanson 23 Jul '13

23 Jul '13

Please find minutes from the above call attached. These are intermediate minutes as there is a follow-on call Wed 24th @ 3pm UK / 10am Eastern. Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

issue: scannable and 'results are not predictable'
by Mike Beckerle 23 Jul '13

23 Jul '13

I was editing the definition of scannable into the glossary and when I looked at usage of 'scannable' in testPattern I found this: In the box for testPattern it says if the data is not scannable "the results are not predictable". Is that sufficient? We can sometimes statically determine that the schema says the data should all be scannable (e.g., no change of encoding, no binary elements), and that would rule out one non-predictability. So, if data is non-scannable in the sense that the schema contains say, a binary element, we can issue an SDE if lengthKind is pattern or a testPattern assert is being used. We could also SDE if runtime-valued encoding properties are used and the encoding changes inside a scannable context. Well, I guess testKind pattern asserts/discriminators are an issue because they may look only at the first part of the data of a complex component, so they don't require everything to be scannable, only the part the regex actually examines. So in this case it's user-beware, and if non-scannable I suppose we could issue a warning. But the spec does not say this is an SDE or warning currently. It just says results are not predictable. There is also the fact that the data might be broken, i.e., the schema might say the data is scannable, but at parse time character decode errors occur. I believe our policy on this is that these cause processing errors. This really is orthogonal to scannable, which is a property of a schema component. Comments? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 4

OGF DFDL WG Call Agenda 2013-07-23
by Steve Hanson 22 Jul '13

22 Jul '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13097?download= Note earlier start time of 2:30 UK (9:30 Eastern) Same finish time. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL WG Call (23 Jul 14:30 GDT in DE3S09)
by Steve Hanson 22 Jul '13

22 Jul '13

Rescheduled: OGF DFDL WG Call 23/07/2013 - (Repeats) Location: DE3S09 Steve Hanson has rescheduled this meeting. You have not added this entry to your calendar. Repeats This entry repeats Required: mbeckerle.dfdl(a)gmail.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org, Stephanie Fetzer/Charlotte/IBM@IBMUS OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

Fw: OGF DFDL WG Call
by Steve Hanson 22 Jul '13

22 Jul '13

For this week only I'd like to start earlier at either 2pm or 2:30pm UK (9am or 9:30am Eastern) and finish at usual time, because we are not getting through the agenda items. Key attendees: please let me know if that is ok and which time you can make. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2013 13:36 ----- OGF DFDL WG Call Tue 08/01/2013 15:00 - 16:00 (Repeats) Location: DE3S09 Required: mbeckerle.dfdl(a)gmail.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org Repeats: OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Fw: Unordered Groups : new draft
by Steve Hanson 22 Jul '13

22 Jul '13

Hopefully we can close on this week's WG call. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2013 13:13 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Date: 18/07/2013 16:07 Subject: Re: Unordered Groups : new draft Tim That looks great. I've made a few minor editorial changes so it can be dropped straight into the errata document and then the spec with no further tweaks. Also added that occurrences of the same member need not be contiguous, and replaced 'real infoset' with 'infoset implied by the original unordered group'. You didn't incorporate Mike's review comment? "If a member element is found to have the empty representation then the parsing of that element must use the original value of minOccurs. In this example, element "b" has minOccurs="0" and if it is found with the empty representation then it must not be defaulted." should perhaps be: "If a member element is found to have the empty representation, then the decision about whether a default value is to be computed is based on the original value of minOccurs, and the count of how many other occurrences of that member element have been accumulated so far by the parsing. In this example, element "b" has minOccurs="0" and if it is found with the empty representation then it must not be defaulted." Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com>, Date: 18/07/2013 11:38 Subject: Unordered Groups : new draft I've addressed Steve last round of comments, and improved the wording of the introductory paragraph. [attachment "Section 14.3 - Unordered Sequence Groups.doc" deleted by Steve Hanson/UK/IBM] regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0