January 2012 - dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2012-01-17
by Steve Hanson 17 Jan '12

17 Jan '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Latest errata document may be found via https://forge.ogf.org/sf/projects/dfdl-wg Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Proposed Errata Language: Issue - DFDL Expressions may not return empty string for ...
by Mike Beckerle 17 Jan '12

17 Jan '12

Here's some proposed errata language: *The following properties descriptions are amended to include this stipulation: When a DFDL Expression is used, it may not produce empty string. The affected properties are:** * - *textBooleanTrueRep* - *textBooleanFalseRep* - *initiator* - * terminator* - *separator* I did verify that the other properties that allow DFDL Expression to compute a string do not need further clarification. On Tue, Jan 10, 2012 at 11:52 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Hi Mike > > As will be minuted, we agreed on the WG call today that we disallow > expressions that return empty string for properties where empty string > turns off the property. Please can you take a look through the spec and see > if any properties other than initiator, terminator, separator are impacted, > then I can complete the errata. > > (I would expect that inputValueCalc and outputValueCalc are not affected > by this errata, as empty string is a legal value for the element in > question if it is of type xs:string). > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org > Date: 10/01/2012 08:06 > Subject: [DFDL-WG] spec clarification needed: is > dfdl:terminator='{ ...returns empty string ... }' allowed? > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > > Let's use the example of terminator as a delimiter. > > If I provide an expression so that I can compute terminator at runtime, is > it allowed to return empty string? I.e., equivalent to writing > dfdl:terminator="" which is effectively "turning off" use of terminator? > > It seems very problematic to me if we allow this. Nor do I think this > generality is needed. > > We should clarify that for initiator/terminator/separator, if a runtime > expression is used, then it must return at least one non-zero-length value. > So using a runtime expression for a delimiter is effectively saying "yes > there will be a delimiter", you are just not binding its specific value. > > I believe this runtime expression capability for delimiters was intended > to allow the choice of the specific delimiter to be made based on data > containing the value. This is common practice in data formats. > > However, turning on/off whether delimiters are present or not, is not > something I anticipated, and it has far bigger implications for the format. > I mean you really can't decide much about the data format statically if > even the existence of delimiters as part of the format or not can be > postponed to runtime. > > Comments? > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 5

Action 133: Create OGF document for default properties DFDL schema
by Steve Hanson 17 Jan '12

17 Jan '12

Attached for review on today's call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-01-17
by Steve Hanson 17 Jan '12

17 Jan '12

Please find agenda attached for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… This is the first call at the new time of 16:00 GMT / 11:00 Eastern. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Proposed Errata Language: Issue 156 - ICU fallback mappings - character encoding/decoding errors
by Mike Beckerle 17 Jan '12

17 Jan '12

Below is proposed errata language for Issue 156 (ICU fallback mappings, and character encoding/decoding errors) * Errata - Character encoding/decoding errors* * * A format property is added to section 11 (properties common to both content and framing) at the bottom of the table. The property is encodingErrorPolicy it is an Enum. Valid values are 'skip', 'error', 'replace'* For Parsing/Decoding Errors* There are two errors that can occur when decoding characters into Unicode/ISO 10646. 1. the data is broken - invalid byte sequences that don't match the definition of the encoding are encountered. 2. not enough bytes are found to make up the entire encoding of a character. That is, a fragment of a valid encoding is found. The behavior in these cases is controlled by dfdl:inputEncodingErrorPolicy. If 'replace', then the Unicode *replacement character*<http://en.wikipedia.org/wiki/Replacement_character>'�' (U+FFFD) is substituted for the offending bytes with one replacement character for any incorrect fragment of an encoding. If 'skip' then the invalid byte sequences are dropped/ignored. No corresponding characters are created in the DFDL infoset. If 'error' then a processing error occurs. It is suggested that if a DFDL user wants to preserve information containing data where the encodings have these kinds of errors, that they model such data as xs:hexBinary, or as a xs:string, but using an encoding such as iso-8859-1 which preserves all bytes. * * The following specific situations involving UTF-16, UTF-16LE, and UTF-16BE do not cause an encoding error: - *utf-16 and unpaired surrogate code-point* - *utf-16 and out-of-order surrogate code-point pair* - *utf-16 encoding utf16Width='fixed' and a surrogate code point is encountered* - *utf-16 byte-order-marks found not at the beginning of the data* In all these cases the code-point becomes part of the DFDL Information Item for the string. *For Unparsing/Encoding Errors* The following are kinds of errors when unparsing characters: 1. no mapping provided by the encoding specification. 2. not enough room to output the entire encoding of the character (e.g., need 2 bytes for a DBCS, but only 1 byte remains in the available length. The behavior in these cases is controlled by dfdl:encodingErrorPolicy. If the policy is 'error' then a processing error occurs. If the policy is 'skip' then the character is skipped. No character is encoded to be output for case 1, and no partial character is attempted in case 2. If the policy is 'replace' then the behavior is determined by the encoding specification. Each encoding has a replacement/substitution character specified by the ICU. (These can be found conveniently in the *ICU Converter Explorer.*<http://demo.icu-project.org/icu-bin/convexp> This character is substituted for the unmapped character or the character that has too large an encoding.) It is a processing error if it is not possible to output the replacement character because there is not enough room for its representation. It is a processing error if a character encoding does not provide a substitution/replacement character definition and one is needed because of dfdl:encodingErrorPolicy='replace'. (This would be rare, but could occur if a DFDL implementation allows many encodings beyond the minimum set.)

2 1

New Issue / Discussion topic - Clarification about Literal Nil Values Greedy Parsing
by Mike Beckerle 16 Jan '12

16 Jan '12

This arose out of discussion of greedy parse behavior for delimiters, but is clearly distinct from that. I wrote this language as proposed clarification around the issue of greedy parsing for literal nil values. *When nilKind='literalValue' the list of values provided by the nilValue property is processed in a greedy manner, meaning it takes all the strings from the whitespace separated list that is the value of the nilValue property, and sorts them longest first, with length-ties remaining in the same order as written in the schema. (This is sometimes called a stable sort - ties aren't re-ordered.) The processor then attempts to match them against the data stream one by one. Once a match is found, the literal nil representation is said to be found, and no other possibilities for the literal nil representation of that element will be subsequently attempted. (That is, the DFDL processor will not attempt to try any of the other literal nil representations in hopes of obtaining a successful parse, should the selected one lead to a processing error.) Having matched the literal nil representation, the element may still fail to parse as a nil value if the matched literal nilValue is not followed by the appropriate termination as specified by the terminator property and the nilValueDelimiterPolicy property. * I'm not sure this is clear enough, so I offer it as a topic for discussion.

1 0

Proposed Errata Language: Issue - clarify that our infoset values are not exactly ISO 10646, but are a superset
by Mike Beckerle 16 Jan '12

16 Jan '12

Proposed Errata Language: *The definition of [dataValue] in section 4.1.2 is changed to the following (note the first sentence is unchanged) [dataValue] The value in the value space (as defined by XML Schema Part 2: Datatypes <http://www.w3.org/TR/xmlschema-2/> [XSDLV1] ) of the [datatype] member or special value nil. In a complex element information item this member has no value. * * For information items of datatype xs:string, the value is an ordered collection of unsigned 16-bit integer codepoints each having any value from 0x0000 to 0xFFFF. Where defined, these are interpreted as the ISO646 character codes. Codepoints disallowed by ISO 10646, such as 0xD800 to 0xDFFF are explicitly allowed by the DFDL infoset. The codepoints of the string are stored in 'implicit' (also known as logical), left-to-right bidirectional ordering and orientation. DFDL's infoset represents Unicode characters with character codes beyond 0xFFFF by way of surrogate pairs (2 adjacent codepoints) in a manner consistent with the UTF-16 encoding of ISO 10646. * -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Re: [DFDL-WG] Proposed spec errata for calendarPattern I and T symbols
by Steve Hanson 16 Jan '12

16 Jan '12

Tim, If a user wants a calendar type in his infoset for a 4 digit year, the recommendation would be to use a calendar pattern of 'YYYY'. The missing components get defaulted to values from the Unix epoch. That's much easier than using inputValueCalc (which would only be necessary if he wanted values other than the Unix epoch). Bearing that in mind, allowing the 'Year' and 'Year and month' granularities is not unnatural. On balance I think it is easier just to say that DFDL 'I' symbol supports all the granularities, and therefore all the W3C ISO8601 profile. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 16/01/2012 10:00 Subject: Re: [DFDL-WG] Proposed spec errata for calendarPattern I and T symbols I would be in favour of the behaviour listed under 'Alternatives'. If DFDL doesn't support gYear, gYearMonth etc then DFDL users should model the various parts of the type as integers/strings. They can always use XPath constructors with InputValueCalc if they really need to handle these types as calendar types. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 16/01/2012 05:05 Subject: [DFDL-WG] Proposed spec errata for calendarPattern I and T symbols Sent by: dfdl-wg-bounces(a)ogf.org For discussion on next WG call. Section 13.11.1. of the DFDL 1.0 spec defines the formatting symbols that can be used for calendar patterns. In addition to the supported ICU symbols, it also permits some extra symbols (I and T) that are used as a shorthand for ISO8601 format date/times. The following is quoted from the spec. Currently Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) T ISO8601 Time (Text) 12:06:56.568+01:00 (up to HH:mm:ss.SSSZZZ) TU ISO8601 Time (Text) 12:06:56.568Z (similar to T, but a time zone of +00:00 is replaced with 'Z') The 'I' and 'T' pattern characters should be used on their own to format dates and times which match the following subset of the ISO8601 standard. · The restricted profile as proposed by the W3C at http://www.w3.org/TR/NOTE-datetime · Truncated representations of calendar dates, as specified in section 5.2.1.3 of ISO8601:2000 o Basic format (subsections c, e, and f) o Extended format (subsections a, b, and d) An analysis shows that this definition of I and T is both incomplete and overly complex. The restricted W3C profile only defines the format for date/time and date-only, there is no time-only format The restricted W3C profile supports 6 'granularities' and states that referencing standards must say which they support The restricted W3C profile states that referencing standards must be explicit as to the number of fractional seconds supported The support for truncated representations of calendar dates was added to support xs:gYear, xs:gYearMonth, etc, which are not supported in 1.0 It does not say what the canonical output 'granularity' is. Proposal Here is what is proposed to correct this. Note the T symbol is dropped. It was introduced to allow xs:dateTime to expect just a time, but that is not necessary (it was copied from IBM MRM). I've stated any alternatives that we can consider at the end. Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) The 'I' symbol must not be used with any other symbol other than the 'escape for text' symbol. It represents calendar formats that match those defined in the restricted profile of the ISO8601 standard proposed by the W3C at http://www.w3.org/TR/NOTE-datetime.The formats are referred to as 'granularities'. xs:dateTime. When parsing, the data must match one of the granularities. When unparsing, the fullest granularity is used. xs:date. When parsing, the data must match one of the date-only granularities. When unparsing, the fullest date-only granularity is used. 'IU' is permitted for xs:date but the 'U' is effectively ignored as there is no time zone in the date-only granularities. xs:time. When parsing, the data must match only the time components of one of the granularities that contains time components. When unparsing, the time components of the fullest granularity are used. The literal 'T' character is not expected in the data when parsing and is not output when unparsing. The number of fractional second digits supported is implementation dependent but must be at least one. For a granularity that omits components, when parsing the values for the omitted components are supplied from the Unix epoch 1970-01-01T00:00:00.000 . Alternatives: As above but don't support the first two granularities ('Year', 'Year and month') on the grounds that they are really matching xs:gYear and xs:gYearMonth. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Proposed spec errata for calendarPattern I and T symbols
by Steve Hanson 16 Jan '12

16 Jan '12

For discussion on next WG call. Section 13.11.1. of the DFDL 1.0 spec defines the formatting symbols that can be used for calendar patterns. In addition to the supported ICU symbols, it also permits some extra symbols (I and T) that are used as a shorthand for ISO8601 format date/times. The following is quoted from the spec. Currently Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) T ISO8601 Time (Text) 12:06:56.568+01:00 (up to HH:mm:ss.SSSZZZ) TU ISO8601 Time (Text) 12:06:56.568Z (similar to T, but a time zone of +00:00 is replaced with 'Z') The 'I' and 'T' pattern characters should be used on their own to format dates and times which match the following subset of the ISO8601 standard. · The restricted profile as proposed by the W3C at http://www.w3.org/TR/NOTE-datetime · Truncated representations of calendar dates, as specified in section 5.2.1.3 of ISO8601:2000 o Basic format (subsections c, e, and f) o Extended format (subsections a, b, and d) An analysis shows that this definition of I and T is both incomplete and overly complex. The restricted W3C profile only defines the format for date/time and date-only, there is no time-only format The restricted W3C profile supports 6 'granularities' and states that referencing standards must say which they support The restricted W3C profile states that referencing standards must be explicit as to the number of fractional seconds supported The support for truncated representations of calendar dates was added to support xs:gYear, xs:gYearMonth, etc, which are not supported in 1.0 It does not say what the canonical output 'granularity' is. Proposal Here is what is proposed to correct this. Note the T symbol is dropped. It was introduced to allow xs:dateTime to expect just a time, but that is not necessary (it was copied from IBM MRM). I've stated any alternatives that we can consider at the end. Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) The 'I' symbol must not be used with any other symbol other than the 'escape for text' symbol. It represents calendar formats that match those defined in the restricted profile of the ISO8601 standard proposed by the W3C at http://www.w3.org/TR/NOTE-datetime.The formats are referred to as 'granularities'. xs:dateTime. When parsing, the data must match one of the granularities. When unparsing, the fullest granularity is used. xs:date. When parsing, the data must match one of the date-only granularities. When unparsing, the fullest date-only granularity is used. 'IU' is permitted for xs:date but the 'U' is effectively ignored as there is no time zone in the date-only granularities. xs:time. When parsing, the data must match only the time components of one of the granularities that contains time components. When unparsing, the time components of the fullest granularity are used. The literal 'T' character is not expected in the data when parsing and is not output when unparsing. The number of fractional second digits supported is implementation dependent but must be at least one. For a granularity that omits components, when parsing the values for the omitted components are supplied from the Unix epoch 1970-01-01T00:00:00.000 . Alternatives: As above but don't support the first two granularities ('Year', 'Year and month') on the grounds that they are really matching xs:gYear and xs:gYearMonth. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

DFDL WG action 153 : dfdl:calendarTimezone
by Steve Hanson 13 Jan '12

13 Jan '12

Minutes from DFDL WG call from last year: 13.11. The DFDL property combination of dfdl:calendarTimezone and dfdl:calendarObserveDST is insufficient as it does not cater for different start dates for DST. To achieve this we would have to relax dfdl:calendarTimeZone rules and allow it to take other formats, such as: a) Olson format (eg, "Europe/Paris") b) Unicode short time zone IDs as per (Unicode Locale Data Markup Language ) spec (eg, "frpar") The list of TZs for b) is here: http://lawiki.org/extensions/core/common/bcp47/timezone.xml Note there a special value for 'unknown' - "unk" or "Etc/Unknown" - prefer to empty string? Agreed in principle, raised action 153 to decide on the extended format to use. Noted that calendarObserveDST would still be honoured.for UTC values. Action 153 proposal Allow dfdl:calendarTimezone to be either UTC offset format or Olson format, as per option (a) above. Olson is the most widely-used naming convention for time zones. Option (b) above refers to 5 character UN LOCODE codes (where possible) which identify locations and map to Olson time zones. However they are not well known and therefore less usable. For representing 'not specified' time zone stick with the current errata (2.50) and use empty string. It seems more obvious than "Etc/Unknown" which is not defined in the Olson database, so would have to be handled as a special case. When unparsing and dfdl:calendarPattern contains a formatting symbol for time zone (zzz, Z, VVVV etc) and the infoset value does not contain a time zone, it is a processing error. This matches the behaviour on parsing when the data does not contain a time zone but the pattern does. Spec should be clear on this. When dfdl:calendarObserveDST is set to 'yes', it only makes sense if dfdl: calendarTimeZone is an Olson time zone. If it is a UTC offset then it should be a schema definition error if dfdl:calendarObserveDST is 'yes'. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0