- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2013-01-15
by Steve Hanson 17 Jan '13

17 Jan '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12987?download= Errata document is now at v011 (draft) here: http://redmine.ogf.org/dmsf_files/8104?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: date & time and latest ICU possible issues/conflicts
by Steve Hanson 17 Jan '13

17 Jan '13

The ICU ticket has been answered, with reference to the following document: http://cldr.unicode.org/development/development-process/design-proposals/ti… 'x'/'X' symbols In a not-too-distant ICU there will be new symbols 'x' and 'X' to handle Z as a time zone for ISO8601 date/times, effectively replacing 'ZZZZZ'. Both 'x' and 'X' will tolerate Z or +00:00 (or variants) on parsing, and on formatting X will result in Z and x will result in +00:00 (or variants). DFDL's use of 'U' is wider than this, as we allow 'U' to appear with any number of 'Z's, meaning that Z is accepted with non-ISO8601 date/times. DFDL also adds the use of 'I' symbol on its own to mean any ISO8601 compliant date/time, and again we allow 'U' to appear with 'I'. However the motivating use case for adding 'U' was IBM MRM which supports this today. But it does so primarily for XML use cases, in particular ISO8601. I am not personally aware of an actual non-XML use case. I suggest we drop the DFDL-specific use of the 'U' symbol in conjunction with 'Z' and 'I' symbols from the DFDL specification via errata, and allow the use of 'ZZZZZ' instead, which at least will accept Z when parsing. When 'x'/'X' support appears in ICU, we can take a future errata to support it or leave until DFDL 2.0. IBM DFDL already supports 'U' but I am ok with deprecating it as I don't believe it will be being used for real. 'V' symbol In a not-too-distant ICU there will be new symbols 'VV' and 'VVV' to handle time zones expressed as Time Zone Ids and localized locations, respectively. We can add that via errata in the future, or leave until DFDL 2.0. However, at the same time the meaning of V is changed slightly. DFDL supports 'V'. I have asked ICU for a clarification. 'O' symbol In a not-too-distant ICU there will be new symbol 'O' to handle localized GMT format variants. We can add that via errata in the future, or leave until DFDL 2.0. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 17/01/2013 08:58 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 16/01/2013 18:20 Subject: Re: [DFDL-WG] date & time and latest ICU possible issues/conflicts ICU ticket raised as the help does not give an example. https://icu.sanjose.ibm.com/gcoctrac/ticket/469#ticket Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 16/01/2013 15:11 Subject: [DFDL-WG] date & time and latest ICU possible issues/conflicts Sent by: dfdl-wg-bounces(a)ogf.org Steve Lawrence is on the Daffodil Open Source DFDL team (on CC), and he has dug into date/time types. He raised some concerns to me that I really haven't been tracking at all, so I wanted to put in front of the rest of the group. The date time format syntax for the latest version of icu4j contains a 'U' character, which means "cyclic year name". However, the daffodil spec says the 'U' character, following a Z makes it so a timezone of UTC is represented as Z instead of +00:00. This seems to be a conflict, and would prevent us from ever upgrading to the newest version of ICU (which might be a good idea). I will point out that the latest version of ICU supports ZZZZZ (5 Z's), which is the ISO8601 timezone format. This doesn't add all the functionality that the DFDL 'U' gives. My question is, is this enough? Are there cases where the ZU, ZZU, etc. are necessary? I'm just concerned that the U is going to quite a bit more complexity, and want to make sure the updates to latest ICU don't address the DFDL-WG concerns. And if we still need the 'U', maybe it should change to a different letter to prevent conflicts with the latest ICU4J? I would point out that the ICU pattern language cannot deal with dual-purpose letters very well, i.e., ambiguities are introduced if the same letter both introduces a format, and if following another format string, modifies its behavior. E.g., does ZU mean Z modified by U, or Z first, and then U. So it seems pretty unfortunate if the ICU libraries added a conflicting use of letter U. I believe the point of DFDL's use of the U modifier for letters I and Z was to be absolutely clear on the GMT timezone 'Z' issue, i.e., to indicate that 'Z' is to be used, and -00:00 is not to be output, nor accepted when parsing. The ICU specification ZZZZZ says ISO format, but that allows either 'Z' or -00:00 to be used for GMT timezone, and it's not clear what it means on output. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

date & time and latest ICU possible issues/conflicts
by Mike Beckerle 16 Jan '13

16 Jan '13

Steve Lawrence is on the Daffodil Open Source DFDL team (on CC), and he has dug into date/time types. He raised some concerns to me that I really haven't been tracking at all, so I wanted to put in front of the rest of the group. The date time format syntax for the latest version of icu4j contains a 'U' character, which means "cyclic year name". However, the daffodil spec says the 'U' character, following a Z makes it so a timezone of UTC is represented as Z instead of +00:00. This seems to be a conflict, and would prevent us from ever upgrading to the newest version of ICU (which might be a good idea). I will point out that the latest version of ICU supports ZZZZZ (5 Z's), which is the ISO8601 timezone format. This doesn't add all the functionality that the DFDL 'U' gives. My question is, is this enough? Are there cases where the ZU, ZZU, etc. are necessary? I'm just concerned that the U is going to quite a bit more complexity, and want to make sure the updates to latest ICU don't address the DFDL-WG concerns. And if we still need the 'U', maybe it should change to a different letter to prevent conflicts with the latest ICU4J? I would point out that the ICU pattern language cannot deal with dual-purpose letters very well, i.e., ambiguities are introduced if the same letter both introduces a format, and if following another format string, modifies its behavior. E.g., does ZU mean Z modified by U, or Z first, and then U. So it seems pretty unfortunate if the ICU libraries added a conflicting use of letter U. I believe the point of DFDL's use of the U modifier for letters I and Z was to be absolutely clear on the GMT timezone 'Z' issue, i.e., to indicate that 'Z' is to be used, and -00:00 is not to be output, nor accepted when parsing. The ICU specification ZZZZZ says ISO format, but that allows either 'Z' or -00:00 to be used for GMT timezone, and it's not clear what it means on output. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

3.13 on encoding errors - rewording - was Re: Second draft of DFDL Errata v011
by Mike Beckerle 16 Jan '13

16 Jan '13

*Re: Errata v11 3.13 I concluded that this needs a rewrite entirely, and that the policy we adopt has to allow for a reasonable implementation of scanning that doesn't require re-invention of the entire world of character I/O. I believe no changes are needed to section 4.1.2, so this errata is not about that section anymore, only about section 11. Suggested revised errata 3.13 text follows:* The encodingErrorPolicy property specified in Errata v10 is removed. A new sub-section is added to section 11. *(this is probably 11.2, if 11.1 is about Unicode byte order marks)* 11.2 Character Encoding and Decoding Errors When parsing, these are the errors that can occur when decoding characters into Unicode/ISO 10646. 1. The data is broken - invalid bit/byte sequences are found which do not match the definition of a character for the encoding. 2. Not enough data is found to make up the entire encoding of a character. That is, a fragment of a valid encoding is found. When unparsing, these are the errors that can occur when encoding characters from Unicode/ISO 10646 into the specified encoding. 1. No mapping provided by the encoding specification. 2. Not enough room to output the entire encoding of the character (e.g., need 3 bytes for a character encoding that uses 3-bytes for that character, but only 1 byte remains in the available length. The subsections below describe how these errors are handled. 11.2.1 Parsing: Decoding Errors When Scanning Scanning is searching of text for data matching: - a delimiter (this can be with or without an associated escape scheme) - initiators, terminators, and separators - a literal nil value (nilKind of 'literalValue' or 'literalCharacter') - a regular expression - for lengthKind='pattern' - for dfdl:assert or dfdl:discriminator with testKind='pattern' In all cases, if scanning encounters a decode error, then it stops with the the successfully decoded character immediately preceding the decode-error. This means that neither framing nor content that is scanned can ever encompass data that caused decode errors. It is worth specific note that the "." regular expression character which matches any character, does not allow a regular expression to subsume a decode error. It is also worth noting that the Unicode Replacement Character (U+FFFD) can appear in data as an ordinary character, but the scanning process will never create these from decode-errors. 11.2.2 Parsing: Decode Errors in Elements of Specified Length For elements with specified lengths (see glossary), the content region is processed and decode errors in conversion of this content region to text will result in the replacement of the non-decodable part of the data by one or more Unicode replacement characters (U+FFFD). If lengthUnits='characters', then such a Unicode replacement character counts as contributing a single character to the length. The trimming of padding characters happens after this replacement. 11.2.2.2 Parsing: The Not-Enough-Data Decode Error There is one special case for the 'not enough data' decode error. For lengthUnits='bytes' when the encoding is a fixed-width character set (see section 12.3.7.1.1 Character Width). If the number of bytes is not a multiple of the character set width, then there will be some number of bytes left over at the end of the data which are insufficient to hold an entire character code. In this case no attempt is made to decode a character from these left-over bytes. They are skipped when parsing (and filled with the dfdl:fillByte on unparsing). 11.2.3 Parsing: Unicode Decoding Non-Errors The following specific situations involving UTF-16, UTF-16LE, and UTF-16BE and do not cause a decoding or encoding error. • UTF-16 and unpaired surrogate code-point • UTF-16 and out-of-order surrogate code-point pair • UTF-16 encoding utf16Width='fixed' and a surrogate code point is encountered • UTF-16 byte-order-marks found not at the beginning of the data In all these cases the code-point becomes a character code in the DFDL Information Item for the string. 11.2.4 Unparsing: Substitution Character For unparsing, each encoding has a replacement/substitution character specified by the ICU. This character is substituted for the unmapped character or the character that has too large an encoding. The definitions of these substitution characters can be conveniently found for many encodings in the ICU Converter Explorer ( http://demo.icu-project.org/icu-bin/convexp) An encoding error is a processing error if the encoding does not provide a substitution/replacement character definition. (This would be rare, but could occur if a DFDL implementation allows many encodings beyond the minimum set.) 11.2.4.1 Unparsing: The Not-Enough-Room Encoding Error There is one special case for the 'not enough room' encoding error. For lengthUnits='bytes' when the encoding is a fixed-width character set (see section 12.3.7.1.1 Character Width). If the number of bytes is not a multiple of the character set width, then there will be some number of bytes of space left over at the end of the data which are insufficient to hold an entire character code. In this case no attempt is made to encode a character into these left-over bytes. They are filled with the dfdl:fillByte. (On parsing they are skipped.) 11.2.5 Preserving Data Containing Decoding Errors There can be situations where data wants to be preserved exactly even if it contains errors. It is suggested that if a DFDL schema author wants to preserve information containing data where the data may have decoding of errors, that they model such data as xs:hexBinary, or as xs:string but using an encoding such as iso-8859-1 which preserves all bytes.

1 2

Action 191 - Clarify what constitutes Schema Definition vs. Processing Error
by Mike Beckerle 15 Jan '13

15 Jan '13

Attached is a document about Action 191 which is a list of specfic errors. -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

1 2

NaN (Not a Number) in DFDL Infoset
by Mike Beckerle 15 Jan '13

15 Jan '13

On our call today we were discussing whether a NaN is allowed in the Infoset or not. I found that XML Schema allows NaN as a value, hence, the PSVI allows NaN as a value. I also found that XPath 2.0 specifically says that any operation involving NaN produces another NaN. So, I believe we should allow NaNs in the DFDL Infoset without causing any error. An implementation note, and probably the source of my confusion on this issue is this: I have observed that java's text number parser, if setup and used to create a float or double, and given some gibberish data that I used, produces a java NaN. In this case, a DFDL implementation must not just accept this NaN into the DFDL infoset, but must scrutinize the data to see if it matches the required representation for a NaN based on the DFDL property dfdl:textStandardNaNRep (as well as dfdl:textNumberJustification since the NaN rep may be surrounded by pad characters). If not, then it is a parse error, as the data is not convertible to the type. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

OGF DFDL WG Call Agenda 2013-01-15
by Steve Hanson 15 Jan '13

15 Jan '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12987?download= Please note that all 2012 calls have been moved to Archive - 2012 folder ( http://redmine.ogf.org/dmsf/dfdl-wg?folder_id=5486) Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Errata 2.11 Question
by Mike Beckerle 11 Jan '13

11 Jan '13

This issue came up in Daffodil test. Errata 2.11 Says: Section 6.3. Clarify that literal white space is only ever used as list token separator, and that entities must be used if literal white space is needed as part of the property value. Question: does this apply to all string literals, or only those that are lists; and hence cannot use whitespace. Can one write escapeBlockStart="[ START ]" or do I have to write escapeBlockStart="[%SP;START%SP;]" I understand the requirement is needed for the lists, but we could choose to enforce it for all string literals if we want. ...mikeb -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

Re: [DFDL-WG] Fw: Proposed Errata: New function dfdl:stringLiteralToString
by Suman Kalia 11 Jan '13

11 Jan '13

Example: dfdl:outputValueCalc='{ fn:concat(../x, dfdl:stringLiteralToString("%NUL;")) }' The proposed function stringLiteralToString and stringLiteralFromString are essentially equivalent to the escape and unescape function in other programming language.. I would suggest renaming the function to escape and unescape as they truly capture the semantics of proposed functionality. Here are references to other programming languages that use escape and unescape function http://php.net/manual/en/sqlite3.escapestring.php http://www.w3schools.com/jsref/jsref_escape.asp Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Date: 01/08/2013 10:31 AM Subject: Fw: [DFDL-WG] Proposed Errata: New function dfdl:stringLiteralToString Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 08/01/2013 15:31 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 12/12/2012 18:14 Subject: Re: [DFDL-WG] Proposed Errata: New function dfdl:stringLiteralToString My vote is From then To style, as in dfdl:stringLiteralToString - new function dfdl:stringToStringLiteral - renamed from dfdl:stringLiteralFromString On Wed, Dec 12, 2012 at 1:02 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: Should name be dfdl:stringFromStringLiteral to mirror existing dfdl:stringLiteralFromString (or should the latter be dfdl:stringToStringLiteral - follows the Java toString() pattern) ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 12/12/2012 17:20 Subject: Re: [DFDL-WG] Proposed Errata: New function dfdl:stringLiteralToString Mike, should we disallow DFDL Character Class entities also ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 12/12/2012 15:40 Subject: [DFDL-WG] Proposed Errata: New function dfdl:stringLiteralToString Sent by: dfdl-wg-bounces(a)ogf.org Proposed Errata: The following function is provided to assist in creating data that contains characters expressed using DFDL String Literal syntax. dfdl:stringLiteralToString($arg) Returns a string constructed from the $arg string argument. If $arg contains syntax matching DFDL Character Entites syntax, then the corresponding characters are used in the result. Any characters in $arg not matching the DFDL Character Entities syntax remain unchanged in the result. Note that DFDL Byte Value Entities are not allowed. It is a schema definition error if $arg contains syntax matching DFDL Byte Value Entities syntax. Use this function when you need to create a value which contains characters for which DFDL Character Entities are needed. An example is to create data containing the NUL (character code 0) codepoint. This character code is not allowed in XML documents, including DFDL Schemas; hence, it must be specified using a DFDL Character Entity. Within a DFDL Expression, use this function to obtain a string containing this character. Example: dfdl:outputValueCalc='{ fn:concat(../x, dfdl:stringLiteralToString("%NUL;")) }' -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

4 8

bug in spec. No dfdl:property(...) function.
by Mike Beckerle 08 Jan '13

08 Jan '13

Latest internal version of spec has section 7.8.1 which is an example using an expression language function that we've dropped from the spec: dfdl:property(...) We need a new example. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0