- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Agenda 2013-02-05
by Steve Hanson 04 Feb '13

04 Feb '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13001?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

dfdl:checkConstraints
by Mike Beckerle 04 Feb '13

04 Feb '13

The error case of an argument that is an XPath to a complex type isn't discussed. Should this return true (hey, a complex element can't have facets, so there are none to be violated), false (errors are false), or cause a processing error or schema definition error? Most conservative choice here is a schema definition error. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

4 5

Action 193: Second draft for errata for RegEx
by Steve Hanson 04 Feb '13

04 Feb '13

The errata for action 193 has been updated below, please review for next WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 04/02/2013 12:50 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 28/01/2013 13:49 Subject: Re: [DFDL-WG] Action 193: First draft for errata for RegEx To me this is excellent work, much appreciated. I'd like to be much more directing about the non-portable constructs. We should decide among only these choices: 1) the non-portable constructs are disallowed. It is an SDE to use them. The check is required for all compliant DFDL implementations (that implement regular expressions at all.) 2) the non-portable constructs are allowed, but not recommended, and DFDL implementations are *required* to issue non-portability warnings if these constructs are used. Not checking this, hoping for the best, user-beware, is a bad idea. A scanner to find these syntaxes and disallow them is pretty easy to write. Regular expressions are, by their very nature, not very rich. You implement an escape scheme, anything else you scan for appearance of the offending constructs. Ironically, it's something that can be done with a regular expression itself. ...mike On Mon, Jan 28, 2013 at 8:22 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Here's a draft errata for action 193, for review on the next WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ============================================================ Section 24 to read as follows: A DFDL regular expression may be specified for the dfdl:lengthPattern format property and the dfdl:testPattern attribute of the dfdl:assert and dfdl:disciminator annotations. DFDL regular expressions do not interpret DFDL entities. A DFDL regular expression is defined by a set of valid pattern characters. For portability, a DFDL regular expression pattern is restricted to the inclusive subset of the ICU regular expression [ICURE] and the Java(R) 7 regular expression [JAVARE] with the Unicode flags UNICODE_CASE and UNICODE_CHARACTER_CLASS turned on. The following regular expression constructs are not common to both ICU and Java(R) 7 and it is a schema definition error if any are used in a DFDL regular expression: *Construct* *Meaning* *Notes* \N{UNICODE CHARACTER NAME} Match the named character ICU only \X Match a Grapheme Cluster ICU only \Uhhhhhhhh Match the character with the hex value hhhhhhhh. ICU only (?# ... ) Free-format comment ICU only (?w-w) UREGEX_UWORD - Controls the behaviour of \b in ICU only a pattern. (?d-d) UNIX_LINES - Enables Unix lines mode. Java 7 only (?u-u) UNICODE_CASE - Enables Unicode-aware case folding. Java 7 only (1) (?U-U) UNICODE_CHARACTER_CLASS - Enables the Unicode Java 7 only (2) version of Predefined character classes and POSIX character classes. (?imsx-imsx:X) X, as a non-capturing group with the given flags. Java 7 only Note that the flags i,s,m,x are valid, but appending :X to the flag is not. Notes: (1) Implementations using Java 7 must set flag UNICODE_CASE by default to match ICU: (2) Implementations using Java 7 must set flag UNICODE_CHARACTER_CLASS by default to match ICU: Additionally, the behaviour of the word character construct (\w) is not consistent in ICU and Java 7. In Java 7 \w is [\p{Alpha}\p{gc=Mn}\p{gc=Me}\p{gc=Mc}\p{Digit}\p{gc=Pc}], which is a larger set than ICU where \w is [\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Nd}]. The use of \w is not recommended in DFDL regular expressions in conjunction with Unicode encodings, and an implementation must issue a warning if such usage is detected. Character properties are detailed by the Unicode Regular Expressions [UNICODERE]. Section 30 to add: [ICURE] - http://userguide.icu-project.org/strings/regexp [JAVARE] - http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html [UNICODERE] - http://www.unicode.org/reports/tr18/ Section 30 to remove: [PERLRE] - http://perldoc.perl.org/perlre.html#Extended-Patterns [JAVARE] - http://download.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.ht… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

occursCountWIthDefault when parsing?
by Mike Beckerle 01 Feb '13

01 Feb '13

Same question different function. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

dfdl:contentLength and dfdl:valueLength when parsing?
by Mike Beckerle 01 Feb '13

01 Feb '13

Do these functions work when parsing? or only when unparsing? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

Use of xs:fixed and xs: prefix generally on XSD attributes
by Mike Beckerle 01 Feb '13

01 Feb '13

This may be a nit, but in prose when we want to distinguish an xsd attribute from a dfdl attribute or property, we prefix it with xs: as in xs:fixed, xs:nillable, etc. However, this is technically incorrect as those attributes are in no namespace. (XML Schema of XSD itself uses the default attributeFormDefault which is 'unqualified'.) A reader might think we are referring to some XSD construct they haven't heard of such as an <xs:nillable...> XSD element, but which doesn't exist. I am ok with keeping things as they are, or we can change to language like "XSD's nillable attribute" and "XSD's fixed attribute", and so forth. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

OGF DFDL WG Call Minutes 2013-01-29
by Steve Hanson 30 Jan '13

30 Jan '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12993?download= ** Updated** Errata document is now at v011.2 here: http://redmine.ogf.org/dmsf_files/8104?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] clarification needed: datetime and time - use of ZZZ
by Steve Hanson 30 Jan '13

30 Jan '13

I think I understand. ICU treats Z, ZZ and ZZZ as equivalent. Maybe there was a difference historically but there isn't now. Web page http://userguide.icu-project.org/formatparse/datetime shows all three, but they don't bother elsewhere. For clarity then we should probably change the xs:time pattern to be HH:mm:ssZ. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Date: 29/01/2013 20:02 Subject: Re: [DFDL-WG] clarification needed: datetime and time - use of ZZZ I got the idea from this page: http://icu-project.org/apiref/icu4j/com/ibm/icu/text/SimpleDateFormat.html which I found by google searching the terms "ICU ZZZZZ" The page mentions only z, zzzz, zzzzz, Z, ZZZZ, ZZZZZ. (that's one, four, and five 'z' combinations). I believe this is probably misleading however. A google search for "ICU ZZZ" turns up a different page which describes the full set of "Z" options. On Tue, Jan 29, 2013 at 12:16 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: Mike & Steve, what makes you think that ZZZ is deprecated in ICU? I'm not seeing a need to change. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Lawrence <slawrence(a)tresys.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, <dfdl-wg(a)ogf.org> Date: 29/01/2013 14:11 Subject: Re: [DFDL-WG] clarification needed: datetime and time I'll add that ZZZZZ in ICU4J accepts all types of timezone formats (RFC 822, GMT Offset, and ISO 8601). So it will also accept formats with both Z and ZZZZ timezone patterns. It outputs as ISO 8601, though. - Steve On 01/29/2013 09:04 AM, Steve Hanson wrote: > Hi Mike, > > The phrase " derived from the XML schema date/time type" just means that > the pattern varies depends on whether the xsd type is date, dateTime or > time, it does not mean that it is set up for parsing a date, dateTime or > time value that conforms to the lexical space for the xsd type. The > pattern values were chosen to be the typical patterns seen in the non-XML > world. > > We can debate whether ZZZ is obsolescent and needs to be updated. IBM DFDL > has shipped with ZZZ as far as I am aware. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, OGF DFDL Working Group > IBM SWG, Hursley, UK > smh(a)uk.ibm.com > tel:+44-1962-815848 > > > > From: Mike Beckerle<mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org, > Date: 29/01/2013 13:55 > Subject: [DFDL-WG] clarification needed: datetime and time > Sent by: dfdl-wg-bounces(a)ogf.org > > > > In the below. Should xs:datetime have trailing timezone as well? Right now > it has no tz component. > > Seems to me it should to match iso 8601 format which is what XSD uses. > > Also, I believe we now think ZZZ is not the right behavior, and seems to > be deprecated in ICU. The right thing is ZZZZZ. > > calendarPatternKind > Enum > Valid values ‘explicit’, ‘implicit’ > ‘explicit’ means the pattern is given by dfdl:calendarPattern, > ‘implicit’ means the pattern is derived from the XML schema date/time > type. > > > Logical Type > Default Pattern > xs:date > yyyy-MM-dd > xs:dateTime > yyyy-MM-dd'T'HH:mm:ss > xs:time > HH:mm:ssZZZ > Annotation: dfdl:element, dfdl:simpleType > > > > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Glossary items needed (for a v12 errata?)
by Mike Beckerle 29 Jan '13

29 Jan '13

We need to add some entries associated with character set and encoding terminology that we use quite a bit. I would note that our usage of the term 'codepoint' differs somewhat from the Unicode Glossary: http://unicode.org/glossary. First, we use codepoint as one word not "code point" (there was some inconsistency on this that I have now fixed), second, what we call codepoint is closer to what Unicode Glossary calls 'code unit'. I suspect we should just provide our definitions rather than switching terms, but I'm open to it if we want to convert all uses of codepoint to "code unit". Encoding - See *Character Set Encoding* Codepoint - When a *character set encoding* uses differing *variable width*representations for characters, the units making up these variable width representations are called codepoints. For example the UTF-8 encoding uses between 1 and 4 codepoints to represent characters, and for UTF-8, the codepoints are single bytes. The UTF-16 encoding is either fixed or variable width. When dfdl:utf16Width='variable' this encoding uses either one or two codepoints per character and each codepoint is a 16-bit value. When a character set is fixed width, then there is no distinction between a codepoint and a character code. Code page - An alternate identifier for a Character Set Encoding. Character Code - The numeric value assigned to a character in a character set that is independent of any specific encoding of that character set. For any fixed-size encoding (all characters have the same size representation) Character Set - An abstract set of characters independent of any specific encoding scheme: Examples are the Unicode character set, or the USASCII character set. Character Set Encoding - A specific representation of a character set as bytes or bits of data. A character set encoding is usually identified by a standard character set name or a recognized alias name, or by a *code page*identifier. These identifiers are standardized by the IANA. Examples are UTF-8, USASCII, GB2312, ebcdic-cp-it, ISO-8859-5, UTF-16BE, Shift_JIS. The DFDL standard allows for implementation-specific character set encodings to be supported, and standardizes one name that is DFDL-specific which is USASCII-7bit-packed. Character Width - The number of codepoints or bytes used to represent a character in a specific character set encoding is called the character width. Encodings are either fixed width (all characters encoded using the same width), or variable-width (different characters are encoded using different widths). For example the UTF-32 character set encoding has 4-byte character width, whereas USASCII has a 1-byte character width. Fixed-Width Character Encoding - A character set encoding where all characters are encoded using a single codepoint for their representation. Note that a codepoint may take up one or more bytes. Surrogate Pair - A Unicode character whose character code value is greater than 0xFFFF can be encoded into variable-width UTF-16BE or UTF-16LE which are variable-width encodings when the DFDL property utf16Width='variable'. In this case the representation uses two adjacent *codepoints *each of which is called a surrogate, and the pair of which is called a surrogate pair. Variable-Width Character Encoding - A character set encoding where characters are encoded using one or more codepoints for their representation depending on which specific character is being encoded. An example is UTF-8 which uses from 1 to 4 bytes to encode a character. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 3

clarification needed: datetime and time
by Mike Beckerle 29 Jan '13

29 Jan '13

In the below. Should xs:datetime have trailing timezone as well? Right now it has no tz component. Seems to me it should to match iso 8601 format which is what XSD uses. Also, I believe we now think ZZZ is not the right behavior, and seems to be deprecated in ICU. The right thing is ZZZZZ. calendarPatternKind Enum Valid values ‘explicit’, ‘implicit’ ‘explicit’ means the pattern is given by dfdl:calendarPattern, ‘implicit’ means the pattern is derived from the XML schema date/time type. *Logical Type * *Default Pattern* xs:date yyyy-MM-dd xs:dateTime yyyy-MM-dd'T'HH:mm:ss xs:time HH:mm:ssZZZ Annotation: dfdl:element, dfdl:simpleType -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 4