- dfdl-wg - lists.ogf.org

Invitation: OGF DFDL Workgroup Weekly Call for 2012-02-07T10:00:00-05:00 @ Tue Feb 7 10am - 11am (dfdl-wg@ogf.org)
by Mike Beckerle 06 Feb '12

06 Feb '12

You have been invited to the following event. Title: OGF DFDL Workgroup Weekly Call for 2012-02-07T10:00:00-05:00 This week only we have a different dial-in arrangement: Please call USA: +1 (781) 330-0114, and I will add you to the conference call. (or use google talk/voice to call mbeckerle(a)gmail.com) Agenda will be sent separately. When: Tue Feb 7 10am – 11am Eastern Time Where: Please call this USA number: +1 781 330 0114 (or google voice chat mbeckerle(a)gmail.com) Calendar: dfdl-wg(a)ogf.org Who: * mbeckerle.dfdl(a)gmail.com - organizer * dfdl-wg(a)ogf.org Event details: https://www.google.com/calendar/event?action=VIEW&eid=NGpxbDA3cHVkdTdlcHVsb… Invitation from Google Calendar: https://www.google.com/calendar/ You are receiving this courtesy email at the account dfdl-wg(a)ogf.org because you are an attendee of this event. To stop receiving future notifications for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar.

1 0

OGF DFDL WG Call Minutes 2012-01-31
by Steve Hanson 02 Feb '12

02 Feb '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Latest errata document may be found via https://forge.ogf.org/sf/projects/dfdl-wg Next call will be hosted by Mike Beckerle who will distribute call-in details. Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Fw: * DFDL Errata* Clarification: Limitations on use of endOfParent
by Steve Hanson 31 Jan '12

31 Jan '12

Here are the updates to the restrictions that we discussed on the call. Action 165 raised. Constraints on element lengthKind 'endOfParent'... - element maxOccurs = 1 - no terminator on element - no trailingSkip on element - if element is in a sequence - sequence must be the content of a complex type - element must be the last object in the sequence - separatorPosition of sequence must not be 'postFix' - sequenceKind of sequence must be 'ordered' - no terminator on sequence - no trailingSkip on sequence - no floating elements in the sequence - if element is in a choice - if choiceLengthKind is 'implicit' - choice must be the content of a complex type - no terminator on choice - no trailingSkip on choice - parent element lengthKind must not be 'implicit' or 'delimited' - not sensitive to any in-scope markup Notes: - complex element can have 'endOfParent', & its last child element can be any lengthKind including 'endOfParent' - element must be the last thing in its box - a box is defined as a portion of the data stream that has an established length prior to the parsing of its children - element with lengthKind 'explicit', 'prefixed', 'pattern' & no sequence/choice right framing - choice with choiceLengthKind 'explicit' Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 31/01/2012 16:54 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org Date: 23/01/2012 17:53 Subject: Re: [DFDL-WG] Fw: * DFDL Errata* Clarification: Limitations on use of endOfParent I have typically used 'implicit' for this because the children of the root element should match the data leaving no bytes unconsumed. Alternatively using 'delimited' should be equivalent, you are never actually scanning. However when the last child is endOfParent according to the rules below I can't use 'implicit' or 'delimited'....so in that scenario I am stuck. Need a rethink. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 23/01/2012 17:00 Subject: Re: [DFDL-WG] Fw: * DFDL Errata* Clarification: Limitations on use of endOfParent Only question I have is on "endOfParent" not being allowed on root element. If you have a DFDL implementation processing message buffers, and the root element's content ends at the end of the buffer/end-of-data, how do we express that? I expected that to be end-of-parent, the notion being that there's an implicit parent for all content, which has an end which is the true end-of-data. ...mikeb On Mon, Jan 23, 2012 at 6:19 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: ----- Forwarded by Steve Hanson/UK/IBM on 23/01/2012 11:16 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Tim Kimber/UK/IBM Date: 18/01/2012 16:26 Subject: * DFDL Errata* Clarification: Limitations on use of endOfParent As agreed on WG extra call on 18th Jan. Will be raised as a separate issue on next DFDL WG call. Constraints on element lengthKind 'endOfParent'... - element maxOccurs = 1 - no terminator on element - if element is in a sequence - separatorPolicy of sequence must not be 'postFix' - sequenceKind of sequence must be 'ordered' - no floating elements in the sequence - must be the 'last' in the sequence statically ** - if element is in a choice it is always 'last' statically ** - parent element lengthKind must not be 'implicit' or 'delimited' - if element is complex then all possible 'last' elements ** must also be 'endOfParent' - not sensitive to any in-scope markup - not allowed on root element ** Need a concise description of walking the content of a complex element and building the list of 'last elements'. Involves factoring out local sequences and coping with choices. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Tim Kimber/UK/IBM@IBMGB Date: 08/11/2011 18:22 Subject: Re: Coping with Character code U+0000 - and how to end-of-parent in an array Hi Mike For the record, I knew we said something about U+0000 - it's in section 5... String – In DFDL a string can contain any character codes. None are reserved. (Including the character with character code U+0000, which is not permitted in XML documents.) After discussion with Sandy Gao, this is what we wrote in the DFDL to XDM mapping document: Note: SimpleElement [datavalue] values may contain characters that are illegal in XML, for example, DFDL strings can contain the character code 0 (zero) within them, but XML does not allow this character code in any XML content even if it is represented as a character entity. Nevertheless, a DFDL described string is mapped to an XDM string data value. and later for the actual mapping to XDM: SimpleElement: If the value of [datavalue] is special value “nil”, then the empty string, otherwise the value of [datavalue] converted to its canonical lexical representation. On to your examples (and assuming separatorPosition is 'infix')... You are right about the first (endOfParent) example being odd. This example would work fine if the lengthKind was 'delimited'. Remember that the 'explicit' length of the parent element creates a box which scopes the delimited behaviour. The second (delimited) works fine. endOfParent and delimited behave almost identically most of the time. When the element is an array, this looks to be one of the differences. I am thinking that endOfParent should not be allowed when maxOccurs > 1. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 08/11/2011 17:02 Subject: Coping with Character code U+0000 - and how to end-of-parent in an array We have this incompatiability with XML infoset around U+0000 aka NUL. However, one can model data containing character code U+0000 in the content as an array of strings with NUL termination. That is, we split the string on the NUL characters so as to avoid putting them in the infoset. So I tried to do this and ran into issues. E.g., if data contains a string of length 80, but inside it the character code 0 can appear, then this could be modeled as: <element name="stringsWithNul" dfdl:lengthKind="fixed" dfdl:length="80"> <sequence dfdl:separator="%x0000;"> <element name="substring" type="string" maxOccurs="80" dfdl:lengthKind="endOfParent" />  </sequence> </element> Problem: is that use of endOfParent length kind right? It's the last thing in the group, but the same element decl also describes the prior elements. If there are no NULs in the string, then endOfParent is exactly what you want. There will be only one substring, it will have all 80 characters. But if there are NULs in the middle, then you want the earlier array elements to be delimited by the sequence's separators, and only the last element to be delimited by endOfParent. This semantics where the parent is providing the constraints on length, but sometimes its separator, just for the last thing it's endOfParent, is not something we can express I believe. I was actually even unclear on this one: If the data 'string' has a terminator of ! then perhaps: <element name="stringsWithNul" dfdl:lengthKind="delimited" dfdl:terminator="!"> <sequence dfdl:separator="%x0000;"> <element name="substring" type="string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>  </sequence> </element> Is the array element delimited? Is that the right length kind for this situation? Thanks for comments ...mikeb ...mikeb Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-01-31
by Steve Hanson 30 Jan '12

30 Jan '12

Please find agenda attached for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Note that calls are back to the original time 3pm GMT / 10am Eastern. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (31 Jan 15:00 GMT in IBM Hursley DE2K09)
by Steve Hanson 27 Jan '12

27 Jan '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Sorry to change things again, but need to revert the DFDL WG call back to it's 3pm GMT slot

1 0

OGF DFDL WG Call Minutes 2012-01-24
by Steve Hanson 24 Jan '12

24 Jan '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Latest errata document may be found via https://forge.ogf.org/sf/projects/dfdl-wg Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Issue 163: Expressions returning ES - type of error when expression returns wrong type.
by Mike Beckerle 24 Jan '12

24 Jan '12

I reviewed section 23. The last paragraph of 23.3 is inconsistent with the rest of the section. Suggested revised wording: The result of evaluating the expression must be a single atomic value of the type expected by the context, and it is a schema definition error otherwise. Some XPath expressions naturally return a sequence of values, and in this case it is also schema definition error if an expression returns a sequence containing more than one item. If the expression returns an empty sequence it will be treated as returning *nil*. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

OGF DFDL WG Call Agenda 2012-01-24
by Steve Hanson 23 Jan '12

23 Jan '12

Please find agenda attached for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: * DFDL Errata* Clarification: Limitations on use of endOfParent
by Steve Hanson 23 Jan '12

23 Jan '12

----- Forwarded by Steve Hanson/UK/IBM on 23/01/2012 11:16 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Tim Kimber/UK/IBM Date: 18/01/2012 16:26 Subject: * DFDL Errata* Clarification: Limitations on use of endOfParent As agreed on WG extra call on 18th Jan. Will be raised as a separate issue on next DFDL WG call. Constraints on element lengthKind 'endOfParent'... - element maxOccurs = 1 - no terminator on element - if element is in a sequence - separatorPolicy of sequence must not be 'postFix' - sequenceKind of sequence must be 'ordered' - no floating elements in the sequence - must be the 'last' in the sequence statically ** - if element is in a choice it is always 'last' statically ** - parent element lengthKind must not be 'implicit' or 'delimited' - if element is complex then all possible 'last' elements ** must also be 'endOfParent' - not sensitive to any in-scope markup - not allowed on root element ** Need a concise description of walking the content of a complex element and building the list of 'last elements'. Involves factoring out local sequences and coping with choices. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Tim Kimber/UK/IBM@IBMGB Date: 08/11/2011 18:22 Subject: Re: Coping with Character code U+0000 - and how to end-of-parent in an array Hi Mike For the record, I knew we said something about U+0000 - it's in section 5... String – In DFDL a string can contain any character codes. None are reserved. (Including the character with character code U+0000, which is not permitted in XML documents.) After discussion with Sandy Gao, this is what we wrote in the DFDL to XDM mapping document: Note: SimpleElement [datavalue] values may contain characters that are illegal in XML, for example, DFDL strings can contain the character code 0 (zero) within them, but XML does not allow this character code in any XML content even if it is represented as a character entity. Nevertheless, a DFDL described string is mapped to an XDM string data value. and later for the actual mapping to XDM: SimpleElement: If the value of [datavalue] is special value “nil”, then the empty string, otherwise the value of [datavalue] converted to its canonical lexical representation. On to your examples (and assuming separatorPosition is 'infix')... You are right about the first (endOfParent) example being odd. This example would work fine if the lengthKind was 'delimited'. Remember that the 'explicit' length of the parent element creates a box which scopes the delimited behaviour. The second (delimited) works fine. endOfParent and delimited behave almost identically most of the time. When the element is an array, this looks to be one of the differences. I am thinking that endOfParent should not be allowed when maxOccurs > 1. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 08/11/2011 17:02 Subject: Coping with Character code U+0000 - and how to end-of-parent in an array We have this incompatiability with XML infoset around U+0000 aka NUL. However, one can model data containing character code U+0000 in the content as an array of strings with NUL termination. That is, we split the string on the NUL characters so as to avoid putting them in the infoset. So I tried to do this and ran into issues. E.g., if data contains a string of length 80, but inside it the character code 0 can appear, then this could be modeled as: <element name="stringsWithNul" dfdl:lengthKind="fixed" dfdl:length="80"> <sequence dfdl:separator="%x0000;"> <element name="substring" type="string" maxOccurs="80" dfdl:lengthKind="endOfParent" />  </sequence> </element> Problem: is that use of endOfParent length kind right? It's the last thing in the group, but the same element decl also describes the prior elements. If there are no NULs in the string, then endOfParent is exactly what you want. There will be only one substring, it will have all 80 characters. But if there are NULs in the middle, then you want the earlier array elements to be delimited by the sequence's separators, and only the last element to be delimited by endOfParent. This semantics where the parent is providing the constraints on length, but sometimes its separator, just for the last thing it's endOfParent, is not something we can express I believe. I was actually even unclear on this one: If the data 'string' has a terminator of ! then perhaps: <element name="stringsWithNul" dfdl:lengthKind="delimited" dfdl:terminator="!"> <sequence dfdl:separator="%x0000;"> <element name="substring" type="string" maxOccurs="unbounded" dfdl:lengthKind="delimited"/>  </sequence> </element> Is the array element delimited? Is that the right length kind for this situation? Thanks for comments ...mikeb ...mikeb Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Fw: Action 162: Re: * DFDL Errata * Proposed spec errata for calendarPattern I and T symbols
by Steve Hanson 19 Jan '12

19 Jan '12

Mike I don't think it's strictly necessary to specify that we will try the longest granularity first, because we can tell from the length of the input data, together with the presence/absence of a '.' (fractional second separator), which of the granularities we should attempt to parse the input data against. Remember that by the time the parser tries to process the date/time, it will already have extracted the text using lengthKind and done any trimming. Granularity Length YYYY 4 YYYY-MM 7 YYYY-MM-DD 10 YYYY-MM-DDThh:mmTZD 17 or 22 YYYY-MM-DDThh:mm:ssTZD 20 or 25 YYYY-MM-DDThh:mm:ss.sTZD 22 or more but input data will contain '.' (TZD can be Z or +hh:mm or -hh:mm) Similarly for xs:time we would have, Granularity Length hh:mmTZD 6 or 11 hh:mm:ssTZD 9 or 14 hh:mm:ss.sTZD 11 or more but input data will contain '.' Any other length of input data, or invalid data for the granularity matched, would be a processing error. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/01/2012 16:29 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 17/01/2012 19:07 Subject: Issue 162: Re: [DFDL-WG] Proposed spec errata for calendarPattern I and T symbols I reviewed this. The proposal looks fine as-is. I don't think the lack of gYear and gYearMonth types should affect the functionality offered. We left those out not because people don't want formats like that, but because we can handle those formats even without the additional types. Do we need to clarify that the longest granularity is tried first, and this is greedy parsing? If the data matches one of the granularities there will be no backtracking to try others. Proposal Here is what is proposed to correct this. Note the T symbol is dropped. It was introduced to allow xs:dateTime to expect just a time, but that is not necessary (it was copied from IBM MRM). I've stated any alternatives that we can consider at the end. Symbol Meaning Presentation Example I ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568+01:00 IU ISO8601 Date/Time (Text) 2006-10-07T12:06:56.568Z with output "Z" if the time zone is +00:00) The 'I' symbol must not be used with any other symbol other than the 'escape for text' symbol. It represents calendar formats that match those defined in the restricted profile of the ISO8601 standard proposed by the W3C at http://www.w3.org/TR/NOTE-datetime.The formats are referred to as 'granularities'. xs:dateTime. When parsing, the data must match one of the granularities. When unparsing, the fullest granularity is used. xs:date. When parsing, the data must match one of the date-only granularities. When unparsing, the fullest date-only granularity is used. 'IU' is permitted for xs:date but the 'U' is effectively ignored as there is no time zone in the date-only granularities. xs:time. When parsing, the data must match only the time components of one of the granularities that contains time components. When unparsing, the time components of the fullest granularity are used. The literal 'T' character is not expected in the data when parsing and is not output when unparsing. The number of fractional second digits supported is implementation dependent but must be at least one. For a granularity that omits components, when parsing the values for the omitted components are supplied from the Unix epoch 1970-01-01T00:00:00.000 . Alternatives: As above but don't support the first two granularities ('Year', 'Year and month') on the grounds that they are really matching xs:gYear and xs:gYearMonth. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0