January 2013 - dfdl-wg - lists.ogf.org

Implementation limits
by Steve Hanson 11 Jul '13

11 Jul '13

Just checking - did we close on this? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 21/01/2013 18:38 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 19/11/2012 21:08 Subject: Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Do we have a section where this goes? There are lots of implied implementation limits. Maximum string length? Maximum hexbinary length? Maximum number of packed digits in various numbers. Maximum length of a delimiter? Maximum length of a fixed attribute? Maximum length of a pattern regex expression? Maximum length of a regular expression? There's also limits on buffering of data for unparsing, etc. There's also limits on lookahead/speculation. I.e., how far an implementation is willing to speculate forward. etc. For some types these are clear: e.g., binary xs:int is 32 bits max. Do we need a numbered section on implementation specific limits. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 4

Use of xs:fixed and xs: prefix generally on XSD attributes
by Mike Beckerle 01 Feb '13

01 Feb '13

This may be a nit, but in prose when we want to distinguish an xsd attribute from a dfdl attribute or property, we prefix it with xs: as in xs:fixed, xs:nillable, etc. However, this is technically incorrect as those attributes are in no namespace. (XML Schema of XSD itself uses the default attributeFormDefault which is 'unqualified'.) A reader might think we are referring to some XSD construct they haven't heard of such as an <xs:nillable...> XSD element, but which doesn't exist. I am ok with keeping things as they are, or we can change to language like "XSD's nillable attribute" and "XSD's fixed attribute", and so forth. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

OGF DFDL WG Call Minutes 2013-01-29
by Steve Hanson 30 Jan '13

30 Jan '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12993?download= ** Updated** Errata document is now at v011.2 here: http://redmine.ogf.org/dmsf_files/8104?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] clarification needed: datetime and time - use of ZZZ
by Steve Hanson 30 Jan '13

30 Jan '13

I think I understand. ICU treats Z, ZZ and ZZZ as equivalent. Maybe there was a difference historically but there isn't now. Web page http://userguide.icu-project.org/formatparse/datetime shows all three, but they don't bother elsewhere. For clarity then we should probably change the xs:time pattern to be HH:mm:ssZ. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Date: 29/01/2013 20:02 Subject: Re: [DFDL-WG] clarification needed: datetime and time - use of ZZZ I got the idea from this page: http://icu-project.org/apiref/icu4j/com/ibm/icu/text/SimpleDateFormat.html which I found by google searching the terms "ICU ZZZZZ" The page mentions only z, zzzz, zzzzz, Z, ZZZZ, ZZZZZ. (that's one, four, and five 'z' combinations). I believe this is probably misleading however. A google search for "ICU ZZZ" turns up a different page which describes the full set of "Z" options. On Tue, Jan 29, 2013 at 12:16 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: Mike & Steve, what makes you think that ZZZ is deprecated in ICU? I'm not seeing a need to change. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Lawrence <slawrence(a)tresys.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, <dfdl-wg(a)ogf.org> Date: 29/01/2013 14:11 Subject: Re: [DFDL-WG] clarification needed: datetime and time I'll add that ZZZZZ in ICU4J accepts all types of timezone formats (RFC 822, GMT Offset, and ISO 8601). So it will also accept formats with both Z and ZZZZ timezone patterns. It outputs as ISO 8601, though. - Steve On 01/29/2013 09:04 AM, Steve Hanson wrote: > Hi Mike, > > The phrase " derived from the XML schema date/time type" just means that > the pattern varies depends on whether the xsd type is date, dateTime or > time, it does not mean that it is set up for parsing a date, dateTime or > time value that conforms to the lexical space for the xsd type. The > pattern values were chosen to be the typical patterns seen in the non-XML > world. > > We can debate whether ZZZ is obsolescent and needs to be updated. IBM DFDL > has shipped with ZZZ as far as I am aware. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, OGF DFDL Working Group > IBM SWG, Hursley, UK > smh(a)uk.ibm.com > tel:+44-1962-815848 > > > > From: Mike Beckerle<mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org, > Date: 29/01/2013 13:55 > Subject: [DFDL-WG] clarification needed: datetime and time > Sent by: dfdl-wg-bounces(a)ogf.org > > > > In the below. Should xs:datetime have trailing timezone as well? Right now > it has no tz component. > > Seems to me it should to match iso 8601 format which is what XSD uses. > > Also, I believe we now think ZZZ is not the right behavior, and seems to > be deprecated in ICU. The right thing is ZZZZZ. > > calendarPatternKind > Enum > Valid values ‘explicit’, ‘implicit’ > ‘explicit’ means the pattern is given by dfdl:calendarPattern, > ‘implicit’ means the pattern is derived from the XML schema date/time > type. > > > Logical Type > Default Pattern > xs:date > yyyy-MM-dd > xs:dateTime > yyyy-MM-dd'T'HH:mm:ss > xs:time > HH:mm:ssZZZ > Annotation: dfdl:element, dfdl:simpleType > > > > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Glossary items needed (for a v12 errata?)
by Mike Beckerle 29 Jan '13

29 Jan '13

We need to add some entries associated with character set and encoding terminology that we use quite a bit. I would note that our usage of the term 'codepoint' differs somewhat from the Unicode Glossary: http://unicode.org/glossary. First, we use codepoint as one word not "code point" (there was some inconsistency on this that I have now fixed), second, what we call codepoint is closer to what Unicode Glossary calls 'code unit'. I suspect we should just provide our definitions rather than switching terms, but I'm open to it if we want to convert all uses of codepoint to "code unit". Encoding - See *Character Set Encoding* Codepoint - When a *character set encoding* uses differing *variable width*representations for characters, the units making up these variable width representations are called codepoints. For example the UTF-8 encoding uses between 1 and 4 codepoints to represent characters, and for UTF-8, the codepoints are single bytes. The UTF-16 encoding is either fixed or variable width. When dfdl:utf16Width='variable' this encoding uses either one or two codepoints per character and each codepoint is a 16-bit value. When a character set is fixed width, then there is no distinction between a codepoint and a character code. Code page - An alternate identifier for a Character Set Encoding. Character Code - The numeric value assigned to a character in a character set that is independent of any specific encoding of that character set. For any fixed-size encoding (all characters have the same size representation) Character Set - An abstract set of characters independent of any specific encoding scheme: Examples are the Unicode character set, or the USASCII character set. Character Set Encoding - A specific representation of a character set as bytes or bits of data. A character set encoding is usually identified by a standard character set name or a recognized alias name, or by a *code page*identifier. These identifiers are standardized by the IANA. Examples are UTF-8, USASCII, GB2312, ebcdic-cp-it, ISO-8859-5, UTF-16BE, Shift_JIS. The DFDL standard allows for implementation-specific character set encodings to be supported, and standardizes one name that is DFDL-specific which is USASCII-7bit-packed. Character Width - The number of codepoints or bytes used to represent a character in a specific character set encoding is called the character width. Encodings are either fixed width (all characters encoded using the same width), or variable-width (different characters are encoded using different widths). For example the UTF-32 character set encoding has 4-byte character width, whereas USASCII has a 1-byte character width. Fixed-Width Character Encoding - A character set encoding where all characters are encoded using a single codepoint for their representation. Note that a codepoint may take up one or more bytes. Surrogate Pair - A Unicode character whose character code value is greater than 0xFFFF can be encoded into variable-width UTF-16BE or UTF-16LE which are variable-width encodings when the DFDL property utf16Width='variable'. In this case the representation uses two adjacent *codepoints *each of which is called a surrogate, and the pair of which is called a surrogate pair. Variable-Width Character Encoding - A character set encoding where characters are encoded using one or more codepoints for their representation depending on which specific character is being encoded. An example is UTF-8 which uses from 1 to 4 bytes to encode a character. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 3

clarification needed: datetime and time
by Mike Beckerle 29 Jan '13

29 Jan '13

In the below. Should xs:datetime have trailing timezone as well? Right now it has no tz component. Seems to me it should to match iso 8601 format which is what XSD uses. Also, I believe we now think ZZZ is not the right behavior, and seems to be deprecated in ICU. The right thing is ZZZZZ. calendarPatternKind Enum Valid values ‘explicit’, ‘implicit’ ‘explicit’ means the pattern is given by dfdl:calendarPattern, ‘implicit’ means the pattern is derived from the XML schema date/time type. *Logical Type * *Default Pattern* xs:date yyyy-MM-dd xs:dateTime yyyy-MM-dd'T'HH:mm:ss xs:time HH:mm:ssZZZ Annotation: dfdl:element, dfdl:simpleType -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 4

OGF DFDL WG Call Agenda 2013-01-29
by Steve Hanson 29 Jan '13

29 Jan '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12993?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

representation properties vs. format properties
by Mike Beckerle 29 Jan '13

29 Jan '13

We use both the terms representation property and format property extensively. Do we want to declare these to mean the same thing as each other, or globally replace one with the other. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

Action 193: First draft for errata for RegEx
by Steve Hanson 28 Jan '13

28 Jan '13

Here's a draft errata for action 193, for review on the next WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ============================================================ Section 24 to read as follows: A DFDL regular expression may be specified for the dfdl:lengthPattern format property and the dfdl:testPattern attribute of the dfdl:assert and dfdl:disciminator annotations. DFDL regular expressions do not interpret DFDL entities. A DFDL regular expression is defined by a set of valid pattern characters. For portability, it is recommended that the regular expression pattern is restricted to the inclusive subset of the ICU regular expression [ICURE] and the Java(R) 7 regular expression [JAVARE] with the Unicode character classes flag (UNICODE_CHARACTER_CLASS) turned on. The following regular expression constructs are not common to both ICU and Java(R) 7 and are not recommended in a DFDL regular expression: *Construct* *Meaning* *Notes* \N{UNICODE CHARACTER NAME} Match the named character ICU only \X Match a Grapheme Cluster ICU only \Uhhhhhhhh Match the character with the hex value hhhhhhhh. ICU only (?# ... ) Free-format comment ICU only (?w-w) UREGEX_UWORD - Controls the behaviour of \b in ICU only a pattern. (?d-d) UNIX_LINES - Enables Unix lines mode. Java 7 only (?u-u) UNICODE_CASE - Enables Unicode-aware case folding. Java 7 only - always on for DFDL (?U-U) UNICODE_CHARACTER_CLASS - Enables the Unicode Java 7 only - version of Predefined character classes and POSIX always on for character classes. DFDL (?imsx-imsx:X) X, as a non-capturing group with the given flags. Java 7 only Note that the flags i,s,m,x are valid, but appending :X to the flag is not. Additionally, the behaviour of the word character construct (\w) is not consistent in ICU and Java(R) 7, and is not recommended. In Java (R) 7 \w is [\p{Alpha}\p{gc=Mn}\p{gc=Me}\p{gc=Mc}\p{Digit}\p{gc=Pc}], which is a larger set than ICU where \w is [\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Nd}]. Character properties are detailed by the Unicode Regular Expressions [UNICODERE]. Section 30 to add: [ICURE] - http://userguide.icu-project.org/strings/regexp [JAVARE] - http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html [UNICODERE] - http://www.unicode.org/reports/tr18/ Section 30 to remove: [PERLRE] - http://perldoc.perl.org/perlre.html#Extended-Patterns [JAVARE] - http://download.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.ht… Andy Andy Edwards - WebSphere Message Broker - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE2 U20 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Minutes 2013-01-22
by Steve Hanson 25 Jan '13

25 Jan '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12991?download= Errata document is now at v011 (draft) here: http://redmine.ogf.org/dmsf_files/8104?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0