- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Agenda 2012-01-24
by Steve Hanson 23 Jan '12

23 Jan '12

Please find agenda attached for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: * DFDL Errata* Clarification: Limitations on use of endOfParent
by Steve Hanson 23 Jan '12

23 Jan '12

----- Forwarded by Steve Hanson/UK/IBM on 23/01/2012 11:16 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Tim Kimber/UK/IBM Date: 18/01/2012 16:26 Subject: * DFDL Errata* Clarification: Limitations on use of endOfParent As agreed on WG extra call on 18th Jan. Will be raised as a separate issue on next DFDL WG call. Constraints on element lengthKind 'endOfParent'... - element maxOccurs = 1 - no terminator on element - if element is in a sequence - separatorPolicy of sequence must not be 'postFix' - sequenceKind of sequence must be 'ordered' - no floating elements in the sequence - must be the 'last' in the sequence statically ** - if element is in a choice it is always 'last' statically ** - parent element lengthKind must not be 'implicit' or 'delimited' - if element is complex then all possible 'last' elements ** must also be 'endOfParent' - not sensitive to any in-scope markup - not allowed on root element ** Need a concise description of walking the content of a complex element and building the list of 'last elements'. Involves factoring out local sequences and coping with choices. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Tim Kimber/UK/IBM@IBMGB Date: 08/11/2011 18:22 Subject: Re: Coping with Character code U+0000 - and how to end-of-parent in an array Hi Mike For the record, I knew we said something about U+0000 - it's in section 5... String – In DFDL a string can contain any character codes. None are reserved. (Including the character with character code U+0000, which is not permitted in XML documents.) After discussion with Sandy Gao, this is what we wrote in the DFDL to XDM mapping document: Note: SimpleElement [datavalue] values may contain characters that are illegal in XML, for example, DFDL strings can contain the character code 0 (zero) within them, but XML does not allow this character code in any XML content even if it is represented as a character entity. Nevertheless, a DFDL described string is mapped to an XDM string data value. and later for the actual mapping to XDM: SimpleElement: If the value of [datavalue] is special value “nil”, then the empty string, otherwise the value of [datavalue] converted to its canonical lexical representation. On to your examples (and assuming separatorPosition is 'infix')... You are right about the first (endOfParent) example being odd. This example would work fine if the lengthKind was 'delimited'. Remember that the 'explicit' length of the parent element creates a box which scopes the delimited behaviour. The second (delimited) works fine. endOfParent and delimited behave almost identically most of the time. When the element is an array, this looks to be one of the differences. I am thinking that endOfParent should not be allowed when maxOccurs > 1. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 08/11/2011 17:02 Subject: Coping with Character code U+0000 - and how to end-of-parent in an array We have this incompatiability with XML infoset around U+0000 aka NUL. However, one can model data containing character code U+0000 in the content as an array of strings with NUL termination. That is, we split the string on the NUL characters so as to avoid putting them in the infoset. So I tried to do this and ran into issues. E.g., if data contains a string of length 80, but inside it the character code 0 can appear, then this could be modeled as: <element name="stringsWithNul" dfdl:lengthKind="fixed" dfdl:length="80"> <sequence dfdl:separator="%x0000;"> <element name="substring" type="string" maxOccurs="80" dfdl:lengthKind="endOfParent" />  </sequence> </element> Problem: is that use of endOfParent length kind right? It's the last thing in the group, but the same element decl also describes the prior elements. If there are no NULs in the string, then endOfParent is exactly what you want. There will be only one substring, it will have all 80 characters. But if there are NULs in the middle, then you want the earlier array elements to be delimited by the sequence's separators, and only the last element to be delimited by endOfParent. This semantics where the parent is providing the constraints on length, but sometimes its separator, just for the last thing it's endOfParent, is not something we can express I believe. I was actually even unclear on this one: If the data 'string' has a terminator of ! then perhaps: <element name="stringsWithNul" dfdl:lengthKind="delimited" dfdl:terminator="!"> <sequence dfdl:separator="%x0000;"> <element name="substring" type="string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>  </sequence> </element> Is the array element delimited? Is that the right length kind for this situation? Thanks for comments ...mikeb ...mikeb Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Fw: Action 162: Re: * DFDL Errata * Proposed spec errata for calendarPattern I and T symbols
by Steve Hanson 19 Jan '12

19 Jan '12

Mike I don't think it's strictly necessary to specify that we will try the longest granularity first, because we can tell from the length of the input data, together with the presence/absence of a '.' (fractional second separator), which of the granularities we should attempt to parse the input data against. Remember that by the time the parser tries to process the date/time, it will already have extracted the text using lengthKind and done any trimming. Granularity Length YYYY 4 YYYY-MM 7 YYYY-MM-DD 10 YYYY-MM-DDThh:mmTZD 17 or 22 YYYY-MM-DDThh:mm:ssTZD 20 or 25 YYYY-MM-DDThh:mm:ss.sTZD 22 or more but input data will contain '.' (TZD can be Z or +hh:mm or -hh:mm) Similarly for xs:time we would have, Granularity Length hh:mmTZD 6 or 11 hh:mm:ssTZD 9 or 14 hh:mm:ss.sTZD 11 or more but input data will contain '.' Any other length of input data, or invalid data for the granularity matched, would be a processing error. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/01/2012 16:29 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 17/01/2012 19:07 Subject: Issue 162: Re: [DFDL-WG] Proposed spec errata for calendarPattern I and T symbols I reviewed this. The proposal looks fine as-is. I don't think the lack of gYear and gYearMonth types should affect the functionality offered. We left those out not because people don't want formats like that, but because we can handle those formats even without the additional types. Do we need to clarify that the longest granularity is tried first, and this is greedy parsing? If the data matches one of the granularities there will be no backtracking to try others. Proposal Here is what is proposed to correct this. Note the T symbol is dropped. It was introduced to allow xs:dateTime to expect just a time, but that is not necessary (it was copied from IBM MRM). I've stated any alternatives that we can consider at the end. Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) The 'I' symbol must not be used with any other symbol other than the 'escape for text' symbol. It represents calendar formats that match those defined in the restricted profile of the ISO8601 standard proposed by the W3C at http://www.w3.org/TR/NOTE-datetime.The formats are referred to as 'granularities'. xs:dateTime. When parsing, the data must match one of the granularities. When unparsing, the fullest granularity is used. xs:date. When parsing, the data must match one of the date-only granularities. When unparsing, the fullest date-only granularity is used. 'IU' is permitted for xs:date but the 'U' is effectively ignored as there is no time zone in the date-only granularities. xs:time. When parsing, the data must match only the time components of one of the granularities that contains time components. When unparsing, the time components of the fullest granularity are used. The literal 'T' character is not expected in the data when parsing and is not output when unparsing. The number of fractional second digits supported is implementation dependent but must be at least one. For a granularity that omits components, when parsing the values for the omitted components are supplied from the Unix epoch 1970-01-01T00:00:00.000 . Alternatives: As above but don't support the first two granularities ('Year', 'Year and month') on the grounds that they are really matching xs:gYear and xs:gYearMonth. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: DFDL WG action 153 : dfdl:calendarTimezone
by Steve Hanson 17 Jan '12

17 Jan '12

The DFDL-WG call today agreed with the proposal below, with one change - if calendarTimeZone is a UTC offset then dfdl:calendarObserveDST is ignored, it is not an error. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 17/01/2012 19:07 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 13/01/2012 11:37 Subject: DFDL WG action 153 : dfdl:calendarTimezone Minutes from DFDL WG call from last year: 13.11. The DFDL property combination of dfdl:calendarTimezone and dfdl:calendarObserveDST is insufficient as it does not cater for different start dates for DST. To achieve this we would have to relax dfdl:calendarTimeZone rules and allow it to take other formats, such as: a) Olson format (eg, "Europe/Paris") b) Unicode short time zone IDs as per (Unicode Locale Data Markup Language ) spec (eg, "frpar") The list of TZs for b) is here: http://lawiki.org/extensions/core/common/bcp47/timezone.xml Note there a special value for 'unknown' - "unk" or "Etc/Unknown" - prefer to empty string? Agreed in principle, raised action 153 to decide on the extended format to use. Noted that calendarObserveDST would still be honoured.for UTC values. Action 153 proposal Allow dfdl:calendarTimezone to be either UTC offset format or Olson format, as per option (a) above. Olson is the most widely-used naming convention for time zones. Option (b) above refers to 5 character UN LOCODE codes (where possible) which identify locations and map to Olson time zones. However they are not well known and therefore less usable. For representing 'not specified' time zone stick with the current errata (2.50) and use empty string. It seems more obvious than "Etc/Unknown" which is not defined in the Olson database, so would have to be handled as a special case. When unparsing and dfdl:calendarPattern contains a formatting symbol for time zone (zzz, Z, VVVV etc) and the infoset value does not contain a time zone, it is a processing error. This matches the behaviour on parsing when the data does not contain a time zone but the pattern does. Spec should be clear on this. When dfdl:calendarObserveDST is set to 'yes', it only makes sense if dfdl: calendarTimeZone is an Olson time zone. If it is a UTC offset then it should be a schema definition error if dfdl:calendarObserveDST is 'yes'. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Issue 162: Re: Proposed spec errata for calendarPattern I and T symbols
by Mike Beckerle 17 Jan '12

17 Jan '12

I reviewed this. The proposal looks fine as-is. I don't think the lack of gYear and gYearMonth types should affect the functionality offered. We left those out not because people don't want formats like that, but because we can handle those formats even without the additional types. Do we need to clarify that the longest granularity is tried first, and this is greedy parsing? If the data matches one of the granularities there will be no backtracking to try others. > *Proposal* > > Here is what is proposed to correct this. Note the T symbol is dropped. It > was introduced to allow xs:dateTime to expect just a time, but that is not > necessary (it was copied from IBM MRM). I've stated any alternatives that > we can consider at the end. > > > *Symbol Meaning Presentation > Example* > *I **ISO8601 Date/Time** **(Text) > 2006-10-07T12:06:56.568+01:00* > *IU **ISO8601 Date/Time** **(Text) > 2006-10-07T12:06:56.568Z* > * with output "Z" if the * > * time zone is +00:00)* > > *The 'I' symbol must not be used with any other symbol other than the > 'escape for text' symbol. It represents calendar formats that match those > defined in the restricted profile of the ISO8601 standard proposed by the > W3C at **http://www.w3.org/TR/NOTE-datetime.*<http://www.w3.org/TR/NOTE-datetime> > *The formats are referred to as 'granularities'.* > > - *xs:dateTime. When parsing, the data must match one of the > granularities. When unparsing, the fullest granularity is used. * > - *xs:date. When parsing, the data must match one of the date-only > granularities. When unparsing, the fullest date-only granularity is used. > 'IU' is permitted for xs:date but the 'U' is effectively ignored as there > is no time zone in the date-only granularities.* > - *xs:time. When parsing, the data must match only the time components > of one of the granularities that contains time components. When unparsing, > the time components of the fullest granularity are used. The literal 'T' > character is not expected in the data when parsing and is not output when > unparsing.* > - *The number of fractional second digits supported is implementation > dependent but must be at least one. * > - *For a granularity that omits components, when parsing the values > for the omitted components are supplied from the Unix epoch **1970-01-01T00:00:00.000 > **.* > > > *Alternatives:* > > As above but don't support the first two granularities ('Year', 'Year and > month') on the grounds that they are really matching xs:gYear and > xs:gYearMonth. > > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

OGF DFDL WG Call Minutes 2012-01-17
by Steve Hanson 17 Jan '12

17 Jan '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Latest errata document may be found via https://forge.ogf.org/sf/projects/dfdl-wg Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Proposed Errata Language: Issue - DFDL Expressions may not return empty string for ...
by Mike Beckerle 17 Jan '12

17 Jan '12

Here's some proposed errata language: *The following properties descriptions are amended to include this stipulation: When a DFDL Expression is used, it may not produce empty string. The affected properties are:** * - *textBooleanTrueRep* - *textBooleanFalseRep* - *initiator* - * terminator* - *separator* I did verify that the other properties that allow DFDL Expression to compute a string do not need further clarification. On Tue, Jan 10, 2012 at 11:52 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Hi Mike > > As will be minuted, we agreed on the WG call today that we disallow > expressions that return empty string for properties where empty string > turns off the property. Please can you take a look through the spec and see > if any properties other than initiator, terminator, separator are impacted, > then I can complete the errata. > > (I would expect that inputValueCalc and outputValueCalc are not affected > by this errata, as empty string is a legal value for the element in > question if it is of type xs:string). > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org > Date: 10/01/2012 08:06 > Subject: [DFDL-WG] spec clarification needed: is > dfdl:terminator='{ ...returns empty string ... }' allowed? > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > > Let's use the example of terminator as a delimiter. > > If I provide an expression so that I can compute terminator at runtime, is > it allowed to return empty string? I.e., equivalent to writing > dfdl:terminator="" which is effectively "turning off" use of terminator? > > It seems very problematic to me if we allow this. Nor do I think this > generality is needed. > > We should clarify that for initiator/terminator/separator, if a runtime > expression is used, then it must return at least one non-zero-length value. > So using a runtime expression for a delimiter is effectively saying "yes > there will be a delimiter", you are just not binding its specific value. > > I believe this runtime expression capability for delimiters was intended > to allow the choice of the specific delimiter to be made based on data > containing the value. This is common practice in data formats. > > However, turning on/off whether delimiters are present or not, is not > something I anticipated, and it has far bigger implications for the format. > I mean you really can't decide much about the data format statically if > even the existence of delimiters as part of the format or not can be > postponed to runtime. > > Comments? > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 5

Action 133: Create OGF document for default properties DFDL schema
by Steve Hanson 17 Jan '12

17 Jan '12

Attached for review on today's call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-01-17
by Steve Hanson 17 Jan '12

17 Jan '12

Please find agenda attached for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… This is the first call at the new time of 16:00 GMT / 11:00 Eastern. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Proposed Errata Language: Issue 156 - ICU fallback mappings - character encoding/decoding errors
by Mike Beckerle 17 Jan '12

17 Jan '12

Below is proposed errata language for Issue 156 (ICU fallback mappings, and character encoding/decoding errors) * Errata - Character encoding/decoding errors* * * A format property is added to section 11 (properties common to both content and framing) at the bottom of the table. The property is encodingErrorPolicy it is an Enum. Valid values are 'skip', 'error', 'replace'* For Parsing/Decoding Errors* There are two errors that can occur when decoding characters into Unicode/ISO 10646. 1. the data is broken - invalid byte sequences that don't match the definition of the encoding are encountered. 2. not enough bytes are found to make up the entire encoding of a character. That is, a fragment of a valid encoding is found. The behavior in these cases is controlled by dfdl:inputEncodingErrorPolicy. If 'replace', then the Unicode *replacement character*<http://en.wikipedia.org/wiki/Replacement_character>'�' (U+FFFD) is substituted for the offending bytes with one replacement character for any incorrect fragment of an encoding. If 'skip' then the invalid byte sequences are dropped/ignored. No corresponding characters are created in the DFDL infoset. If 'error' then a processing error occurs. It is suggested that if a DFDL user wants to preserve information containing data where the encodings have these kinds of errors, that they model such data as xs:hexBinary, or as a xs:string, but using an encoding such as iso-8859-1 which preserves all bytes. * * The following specific situations involving UTF-16, UTF-16LE, and UTF-16BE do not cause an encoding error: - *utf-16 and unpaired surrogate code-point* - *utf-16 and out-of-order surrogate code-point pair* - *utf-16 encoding utf16Width='fixed' and a surrogate code point is encountered* - *utf-16 byte-order-marks found not at the beginning of the data* In all these cases the code-point becomes part of the DFDL Information Item for the string. *For Unparsing/Encoding Errors* The following are kinds of errors when unparsing characters: 1. no mapping provided by the encoding specification. 2. not enough room to output the entire encoding of the character (e.g., need 2 bytes for a DBCS, but only 1 byte remains in the available length. The behavior in these cases is controlled by dfdl:encodingErrorPolicy. If the policy is 'error' then a processing error occurs. If the policy is 'skip' then the character is skipped. No character is encoded to be output for case 1, and no partial character is attempted in case 2. If the policy is 'replace' then the behavior is determined by the encoding specification. Each encoding has a replacement/substitution character specified by the ICU. (These can be found conveniently in the *ICU Converter Explorer.*<http://demo.icu-project.org/icu-bin/convexp> This character is substituted for the unmapped character or the character that has too large an encoding.) It is a processing error if it is not possible to output the replacement character because there is not enough room for its representation. It is a processing error if a character encoding does not provide a substitution/replacement character definition and one is needed because of dfdl:encodingErrorPolicy='replace'. (This would be rare, but could occur if a DFDL implementation allows many encodings beyond the minimum set.)

2 1