- dfdl-wg - lists.ogf.org

New Issue / Discussion topic - Clarification about Literal Nil Values Greedy Parsing
by Mike Beckerle 16 Jan '12

16 Jan '12

This arose out of discussion of greedy parse behavior for delimiters, but is clearly distinct from that. I wrote this language as proposed clarification around the issue of greedy parsing for literal nil values. *When nilKind='literalValue' the list of values provided by the nilValue property is processed in a greedy manner, meaning it takes all the strings from the whitespace separated list that is the value of the nilValue property, and sorts them longest first, with length-ties remaining in the same order as written in the schema. (This is sometimes called a stable sort - ties aren't re-ordered.) The processor then attempts to match them against the data stream one by one. Once a match is found, the literal nil representation is said to be found, and no other possibilities for the literal nil representation of that element will be subsequently attempted. (That is, the DFDL processor will not attempt to try any of the other literal nil representations in hopes of obtaining a successful parse, should the selected one lead to a processing error.) Having matched the literal nil representation, the element may still fail to parse as a nil value if the matched literal nilValue is not followed by the appropriate termination as specified by the terminator property and the nilValueDelimiterPolicy property. * I'm not sure this is clear enough, so I offer it as a topic for discussion.

1 0

Proposed Errata Language: Issue - clarify that our infoset values are not exactly ISO 10646, but are a superset
by Mike Beckerle 16 Jan '12

16 Jan '12

Proposed Errata Language: *The definition of [dataValue] in section 4.1.2 is changed to the following (note the first sentence is unchanged) [dataValue] The value in the value space (as defined by XML Schema Part 2: Datatypes <http://www.w3.org/TR/xmlschema-2/> [XSDLV1] ) of the [datatype] member or special value nil. In a complex element information item this member has no value. * * For information items of datatype xs:string, the value is an ordered collection of unsigned 16-bit integer codepoints each having any value from 0x0000 to 0xFFFF. Where defined, these are interpreted as the ISO646 character codes. Codepoints disallowed by ISO 10646, such as 0xD800 to 0xDFFF are explicitly allowed by the DFDL infoset. The codepoints of the string are stored in 'implicit' (also known as logical), left-to-right bidirectional ordering and orientation. DFDL's infoset represents Unicode characters with character codes beyond 0xFFFF by way of surrogate pairs (2 adjacent codepoints) in a manner consistent with the UTF-16 encoding of ISO 10646. * -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Re: [DFDL-WG] Proposed spec errata for calendarPattern I and T symbols
by Steve Hanson 16 Jan '12

16 Jan '12

Tim, If a user wants a calendar type in his infoset for a 4 digit year, the recommendation would be to use a calendar pattern of 'YYYY'. The missing components get defaulted to values from the Unix epoch. That's much easier than using inputValueCalc (which would only be necessary if he wanted values other than the Unix epoch). Bearing that in mind, allowing the 'Year' and 'Year and month' granularities is not unnatural. On balance I think it is easier just to say that DFDL 'I' symbol supports all the granularities, and therefore all the W3C ISO8601 profile. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 16/01/2012 10:00 Subject: Re: [DFDL-WG] Proposed spec errata for calendarPattern I and T symbols I would be in favour of the behaviour listed under 'Alternatives'. If DFDL doesn't support gYear, gYearMonth etc then DFDL users should model the various parts of the type as integers/strings. They can always use XPath constructors with InputValueCalc if they really need to handle these types as calendar types. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 16/01/2012 05:05 Subject: [DFDL-WG] Proposed spec errata for calendarPattern I and T symbols Sent by: dfdl-wg-bounces(a)ogf.org For discussion on next WG call. Section 13.11.1. of the DFDL 1.0 spec defines the formatting symbols that can be used for calendar patterns. In addition to the supported ICU symbols, it also permits some extra symbols (I and T) that are used as a shorthand for ISO8601 format date/times. The following is quoted from the spec. Currently Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) T ISO8601 Time (Text) 12:06:56.568+01:00 (up to HH:mm:ss.SSSZZZ) TU ISO8601 Time (Text) 12:06:56.568Z (similar to T, but a time zone of +00:00 is replaced with 'Z') The 'I' and 'T' pattern characters should be used on their own to format dates and times which match the following subset of the ISO8601 standard. · The restricted profile as proposed by the W3C at http://www.w3.org/TR/NOTE-datetime · Truncated representations of calendar dates, as specified in section 5.2.1.3 of ISO8601:2000 o Basic format (subsections c, e, and f) o Extended format (subsections a, b, and d) An analysis shows that this definition of I and T is both incomplete and overly complex. The restricted W3C profile only defines the format for date/time and date-only, there is no time-only format The restricted W3C profile supports 6 'granularities' and states that referencing standards must say which they support The restricted W3C profile states that referencing standards must be explicit as to the number of fractional seconds supported The support for truncated representations of calendar dates was added to support xs:gYear, xs:gYearMonth, etc, which are not supported in 1.0 It does not say what the canonical output 'granularity' is. Proposal Here is what is proposed to correct this. Note the T symbol is dropped. It was introduced to allow xs:dateTime to expect just a time, but that is not necessary (it was copied from IBM MRM). I've stated any alternatives that we can consider at the end. Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) The 'I' symbol must not be used with any other symbol other than the 'escape for text' symbol. It represents calendar formats that match those defined in the restricted profile of the ISO8601 standard proposed by the W3C at http://www.w3.org/TR/NOTE-datetime.The formats are referred to as 'granularities'. xs:dateTime. When parsing, the data must match one of the granularities. When unparsing, the fullest granularity is used. xs:date. When parsing, the data must match one of the date-only granularities. When unparsing, the fullest date-only granularity is used. 'IU' is permitted for xs:date but the 'U' is effectively ignored as there is no time zone in the date-only granularities. xs:time. When parsing, the data must match only the time components of one of the granularities that contains time components. When unparsing, the time components of the fullest granularity are used. The literal 'T' character is not expected in the data when parsing and is not output when unparsing. The number of fractional second digits supported is implementation dependent but must be at least one. For a granularity that omits components, when parsing the values for the omitted components are supplied from the Unix epoch 1970-01-01T00:00:00.000 . Alternatives: As above but don't support the first two granularities ('Year', 'Year and month') on the grounds that they are really matching xs:gYear and xs:gYearMonth. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Proposed spec errata for calendarPattern I and T symbols
by Steve Hanson 16 Jan '12

16 Jan '12

For discussion on next WG call. Section 13.11.1. of the DFDL 1.0 spec defines the formatting symbols that can be used for calendar patterns. In addition to the supported ICU symbols, it also permits some extra symbols (I and T) that are used as a shorthand for ISO8601 format date/times. The following is quoted from the spec. Currently Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) T ISO8601 Time (Text) 12:06:56.568+01:00 (up to HH:mm:ss.SSSZZZ) TU ISO8601 Time (Text) 12:06:56.568Z (similar to T, but a time zone of +00:00 is replaced with 'Z') The 'I' and 'T' pattern characters should be used on their own to format dates and times which match the following subset of the ISO8601 standard. · The restricted profile as proposed by the W3C at http://www.w3.org/TR/NOTE-datetime · Truncated representations of calendar dates, as specified in section 5.2.1.3 of ISO8601:2000 o Basic format (subsections c, e, and f) o Extended format (subsections a, b, and d) An analysis shows that this definition of I and T is both incomplete and overly complex. The restricted W3C profile only defines the format for date/time and date-only, there is no time-only format The restricted W3C profile supports 6 'granularities' and states that referencing standards must say which they support The restricted W3C profile states that referencing standards must be explicit as to the number of fractional seconds supported The support for truncated representations of calendar dates was added to support xs:gYear, xs:gYearMonth, etc, which are not supported in 1.0 It does not say what the canonical output 'granularity' is. Proposal Here is what is proposed to correct this. Note the T symbol is dropped. It was introduced to allow xs:dateTime to expect just a time, but that is not necessary (it was copied from IBM MRM). I've stated any alternatives that we can consider at the end. Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) The 'I' symbol must not be used with any other symbol other than the 'escape for text' symbol. It represents calendar formats that match those defined in the restricted profile of the ISO8601 standard proposed by the W3C at http://www.w3.org/TR/NOTE-datetime.The formats are referred to as 'granularities'. xs:dateTime. When parsing, the data must match one of the granularities. When unparsing, the fullest granularity is used. xs:date. When parsing, the data must match one of the date-only granularities. When unparsing, the fullest date-only granularity is used. 'IU' is permitted for xs:date but the 'U' is effectively ignored as there is no time zone in the date-only granularities. xs:time. When parsing, the data must match only the time components of one of the granularities that contains time components. When unparsing, the time components of the fullest granularity are used. The literal 'T' character is not expected in the data when parsing and is not output when unparsing. The number of fractional second digits supported is implementation dependent but must be at least one. For a granularity that omits components, when parsing the values for the omitted components are supplied from the Unix epoch 1970-01-01T00:00:00.000 . Alternatives: As above but don't support the first two granularities ('Year', 'Year and month') on the grounds that they are really matching xs:gYear and xs:gYearMonth. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

DFDL WG action 153 : dfdl:calendarTimezone
by Steve Hanson 13 Jan '12

13 Jan '12

Minutes from DFDL WG call from last year: 13.11. The DFDL property combination of dfdl:calendarTimezone and dfdl:calendarObserveDST is insufficient as it does not cater for different start dates for DST. To achieve this we would have to relax dfdl:calendarTimeZone rules and allow it to take other formats, such as: a) Olson format (eg, "Europe/Paris") b) Unicode short time zone IDs as per (Unicode Locale Data Markup Language ) spec (eg, "frpar") The list of TZs for b) is here: http://lawiki.org/extensions/core/common/bcp47/timezone.xml Note there a special value for 'unknown' - "unk" or "Etc/Unknown" - prefer to empty string? Agreed in principle, raised action 153 to decide on the extended format to use. Noted that calendarObserveDST would still be honoured.for UTC values. Action 153 proposal Allow dfdl:calendarTimezone to be either UTC offset format or Olson format, as per option (a) above. Olson is the most widely-used naming convention for time zones. Option (b) above refers to 5 character UN LOCODE codes (where possible) which identify locations and map to Olson time zones. However they are not well known and therefore less usable. For representing 'not specified' time zone stick with the current errata (2.50) and use empty string. It seems more obvious than "Etc/Unknown" which is not defined in the Olson database, so would have to be handled as a special case. When unparsing and dfdl:calendarPattern contains a formatting symbol for time zone (zzz, Z, VVVV etc) and the infoset value does not contain a time zone, it is a processing error. This matches the behaviour on parsing when the data does not contain a time zone but the pattern does. Spec should be clear on this. When dfdl:calendarObserveDST is set to 'yes', it only makes sense if dfdl: calendarTimeZone is an Olson time zone. If it is a UTC offset then it should be a schema definition error if dfdl:calendarObserveDST is 'yes'. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Updated DFDL 1.0 Errata document
by Steve Hanson 11 Jan '12

11 Jan '12

The errata document is now at v7 and reflects all agreed errata up until 10th January 2012. https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Rescheduled: OGF DFDL Working Group weekly call (17 Jan 16:00 GMT in IBM Hursley DE3S09)
by Steve Hanson 11 Jan '12

11 Jan '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Moving one hour later as minuted

1 0

OGF DFDL WG Call Minutes 2012-01-10
by Steve Hanson 10 Jan '12

10 Jan '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Fw: Inconsistent naming for text bidi properties
by Steve Hanson 10 Jan '12

10 Jan '12

Agreed on WG call to take errata so that second occurrence of 'text' in a bidi property name is removed. That leaves: - textBidiOrdering - textBidiOrientation - textBidiSymmetric - textBidiShaped - textBidiNumeralShape Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 09/01/2012 14:44 Subject: Fw: Inconsistent naming for text bidi properties To be discussed on this week's DFDL WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 09/01/2012 14:43 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 15/12/2011 15:37 Subject: Inconsistent naming for text bidi properties I'd be interested in folks thoughts about this one. If textBidi is 'yes' then the following properties come in to play: - textBidiTextOrdering - textBidiOrientation - textBidiSymmetric - textBidiTextShaped - textBidiNumeralShape Two of the properties have 'text' appearing twice in the name. This seems superfluous to me. Any reason why these two properties have an extra 'text' ? I only noticed this because the IBM DFDL model has coded 'textBidiTextOrientation' which does not match the spec. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

spec clarification needed: is dfdl:terminator='{ ...returns empty string ... }' allowed?
by Mike Beckerle 10 Jan '12

10 Jan '12

Let's use the example of terminator as a delimiter. If I provide an expression so that I can compute terminator at runtime, is it allowed to return empty string? I.e., equivalent to writing dfdl:terminator="" which is effectively "turning off" use of terminator? It seems very problematic to me if we allow this. Nor do I think this generality is needed. We should clarify that for initiator/terminator/separator, if a runtime expression is used, then it must return at least one non-zero-length value. So using a runtime expression for a delimiter is effectively saying "yes there will be a delimiter", you are just not binding its specific value. I believe this runtime expression capability for delimiters was intended to allow the choice of the specific delimiter to be made based on data containing the value. This is common practice in data formats. However, turning on/off whether delimiters are present or not, is not something I anticipated, and it has far bigger implications for the format. I mean you really can't decide much about the data format statically if even the existence of delimiters as part of the format or not can be postponed to runtime. Comments? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1