- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2011-09-20
by Steve Hanson 20 Sep '11

20 Sep '11

Please find minutes from the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 148: pattern based lengths - suggested revised language
by Steve Hanson 20 Sep '11

20 Sep '11

For discussion on today's call... Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 20/09/2011 11:01 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, mbeckerle.dfdl(a)gmail.com, Date: 20/09/2011 10:17 Subject: Re: Action 148: pattern based lengths - suggested revised language I'd like to discuss on the WG call today. I think the conservative approach I outline below is consistent with what we do for complex elements and specified lengths, and I'd prefer to stick with that for 1.0. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Date: 25/08/2011 16:23 Subject: Re: Action 148: pattern based lengths - suggested revised language I think we can afford to be a little less conservative, actually. Let's suppose that we allow patterns regardless of dfdl:representation and regardless of the encoding. That will provide users with maximum flexibility, at the ( not very large ) risk that they will occasionally do something silly. We can put a note into the specification to the effect that patterns should usually be used only with character data, but can ( with care ) be used to match bytes if that is the only way to achieve the desired result. I may be missing something, but I don't see what harm we can cause ourselves or our users by doing this. My concern is that we could take away a lot of the power that patterns provide ( particularly for discriminators / asserts ) and then end up regretting it when some strange format pops up. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Date: 25/08/2011 15:00 Subject: Action 148: pattern based lengths - suggested revised language Hi Tim Please could you have a think about my conservative proposal below? Firstly, can we get away with restricting patterns to text, or will we need to use patterns to grab large amounts of data they may include binary content? Secondly, are we able to apply the same validation criteria to use of testKind pattern on an assert or discriminator as we are to use of lengthKind pattern? . Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 25/08/2011 14:44 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 27/07/2011 15:00 Subject: Re: pattern based lengths - suggested revised language I support what you call the conservative approach. I.e. require text when patterns are used. Mike ------------------------------------------------------------------------------------------- On Jul 27, 2011 5:53 AM, "Steve Hanson" <smh(a)uk.ibm.com> wrote: Hi Mike I don't think we can reduce the wording that much. The second paragraph is needed because it covers the binary case, where encoding is not actually used. I think we either need to be conservative and disallow the combination of binary & pattern, or leave the second paragraph as-is and effectively say that if you use binary with pattern then that is the behaviour. If we are to be conservative then: - For a simple element or simple type, disallow lengthKind="pattern" with binary rep. - For a complex element with lengthKind = "pattern", all children must have lengthUnits = "characters" (so text only) and the encoding of the children must be the same as the encoding of the parent. (We already have a similar rule for complex elements with specified length and lengthUnits = "characters"). We also allow asserts and discriminators to carry patterns which are applied straight at the current position in the data stream. It would be difficult to police the conservative rules here. But we need to say what encoding is used and we currently do not. I would say it must be the encoding of the element or group that carries the assert/discriminator. I said on the call that we had extended DFDL regular expressions so that raw hex bytes could be specified. However I don't see any evidence of this in the DFDL spec. This facility was something we added to IBM MRM for a retail format called TLOG which consists of delimited packed decimal data with hex indicator bytes, so we needed a way to match the hex indicator bytes as part of the regexp. However, I think this was only necessary because MRM has neither speculation nor discriminators, and in a DFDL version of TLOG I would use a discriminator. So I think my statement was in error, and I don't believe raw hex in DFDL regexps is needed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 > > > From: > "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> > To: > Steve Hanson/UK/IBM@IBMGB > Date: > 26/07/2011 17:30 > Subject: > pattern based lengths - suggested revised language > > > > I suggest this language to tighten up this whole section (replace both > paragraphs). Given the concerns of Tim, that we make sure DFDL > implementations don’t have to reimplement regexp matching, I think this is > sufficient. > 1.1.1.1 Based Lengths - Scanability > Any element (complex, simple text, simple binary) may have a > dfdl:lengthKind 'pattern'. When an element contains binary data, and > lengthKind=’pattern’ is used, then it is a schema definition error if the > character set encoding is not iso-8859-1. > > > (Possible generalization 1: allow other character sets, e.g., iso-8859-15 > as well. This is ok because 8859-15 still maps all 256 codepoints. But > this is a slippery slope. ) > > (Possible generalization 2: allow any character set, Ascii, ebcdic, > utf-16be, etc. Note that using any character encoding other than one which > maps a valid character to any 8-bit byte creates ambiguity: e.g, the > regexp “.” is one where we normally think it means “any character”. But > do we really mean “any byte” ? If the character set encoding doesn’t have > a given byte as a codepoint, then this question really matters.) > > > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > > > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: DFDL WG action 143: Proposed resolution
by Steve Hanson 20 Sep '11

20 Sep '11

For discussion on today's call... Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 20/09/2011 11:00 ----- From: Steve Hanson/UK/IBM To: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Cc: Richard Schofield/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 25/08/2011 13:48 Subject: RE: DFDL WG action 143: Proposed resolution Although this action was closed on 5th July WG call, I did find this one extra issue about WSP and textStandardZeroRep and we did not reach a conclusion. I will add it to WG call agenda. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Tim Kimber/UK/IBM@IBMGB, Steve Hanson/UK/IBM@IBMGB Cc: Richard Schofield/UK/IBM@IBMGB Date: 05/07/2011 21:49 Subject: RE: DFDL WG action 143: Proposed resolution Well….. The use case I have is from ancient mainframe style data. Numbers that are all zero are suppressed and all blanks are output instead of anything with zeros in it, or even a single 0 character. This is for data that was both data and “report” simultaneously, i.e., people looked at it. …mikeb From: Tim Kimber [mailto:KIMBERT@uk.ibm.com] Sent: Tuesday, July 05, 2011 3:00 PM To: Steve Hanson Cc: mbeckerle.dfdl(a)gmail.com; Richard Schofield Subject: Re: DFDL WG action 143: Proposed resolution We might want to bear this scenario in mind: Field 'integerA' has explicit length of 5 and textNumberZeroRep='%WSP*;'. The next field is a string which happens to start with some white space. The parser must be careful to ensure that the matching of the %WSP; against the data stream does not walk on into the following field's data. On balance, I would prefer an alternative solution based on either defaulting or nil handling: a) We could define 'empty' in terms of the trimmed value when padKind != "none". That would allow this to be handled by setting the pad character to space and xs:default to zero. b) Without any change to the specification, the user could - set nilLiteralCharacter="%SP;" and xs:nillable="true" or - set nilLiteralValue of %ES; , set textStringPadCharacter="'%SP;", set xs:nillable="true" Either of these would result in a nil in the infoset. This may not be the exact infoset that the user wants, but DFDL ( esp. DFDL 1.0 ) does not guarantee that DFDL emits the exact infoset that you want. In other words, I don't think the spaces are really a representation of the value zero. They are padding or filling that the modeller wants to treat as zero. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: mbeckerle.dfdl(a)gmail.com Cc: Tim Kimber/UK/IBM@IBMGB Date: 05/07/2011 18:49 Subject: DFDL WG action 143: Proposed resolution Mike I was about to mail this out but noticed that textStandardZeroRep was included in the list, and was being prohibited from using WSPx entitities. This would not conveniently allow the scenario you mentioned on the call today where all spaces was allowed to represent zero. I can remove textStandardZeroRep from the list, and leave it as per spec if you prefer. Tim - do you have a problem with that? Regards Steve -------------------------------------------------------------- Following is the resolution from action 143 An errata document accompaniment to the DFDL 1.0 specification is under production and this will be added to it. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/07/2011 18:33 ----- Clarification to the allowable content that is allowed in some of the DFDL properties that are defined as type 'DFDL String Literal'. The properties below do not need the full power of DFDL String Literal. Properties escapeCharacter and escapeEscapeCharacter Property value must resolve to a single character DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties escapeBlockStart and escapeBlockEnd DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties textStringPadCharacter, textNumberPadCharacter, textBooleanPadCharacter and textCalendarPadCharacter Property value must resolve to a single character or a single byte value DFDL character entities are allowed The raw byte entity ( %#r ) is allowed subject to the restrictions already documented for these properties DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties textStandardDecimalSeparator and textStandardGroupingSeparator Property value must resolve to a single character DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties textStandardInfinityRep, textStandardNanRep, textStandardZeroRep DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties textBooleanTrueRep and textBooleanFalseRep DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Property nilValue DFDL character entities are allowed The raw byte entity ( %#r ) is allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are allowed Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2011-09-20
by Steve Hanson 19 Sep '11

19 Sep '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… The latest errata to the spec is here: http://forge.gridforum.org/sf/go/doc16280?nav=1 (updated 2011-09-08) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2011-09-13 - call cancelled
by Steve Hanson 13 Sep '11

13 Sep '11

Apologies but I have to be elsewhere and can't make the call. I'd appreciate if folk could review the updated errata document and let me know if there is anything in there that does not reflect recent WG call agreements, or if anything is missing. And check out the spec on the web. If you have any ideas for improvement let me know. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/09/2011 10:48 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 08/09/2011 15:58 Subject: OGF DFDL WG Call Agenda 2011-09-13 Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… The latest errata to the spec is here: http://forge.gridforum.org/sf/go/doc16280?nav=1 (updated 2011-09-08) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Cancelled: OGF DFDL Working Group weekly call
by Steve Hanson 13 Sep '11

13 Sep '11

Unfortunately I can't make the call today.

1 0

OGF DFDL WG Call Agenda 2011-09-13
by Steve Hanson 08 Sep '11

08 Sep '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… The latest errata to the spec is here: http://forge.gridforum.org/sf/go/doc16280?nav=1 (updated 2011-09-08) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Minutes 2011-09-06
by Steve Hanson 07 Sep '11

07 Sep '11

Call did not take place as only I was available. Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2011-09-06
by Steve Hanson 05 Sep '11

05 Sep '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… As per action 144 an errata to the spec has been created here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Action 151 - BOM handling
by Steve Hanson 05 Sep '11

05 Sep '11

Please find a proposal from the WG to simplify modeling Unicode documents that start with a BOM. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----------------------------------------------------------------------------------------------------------------------- Infoset New property added to the document information item: [unicodeByteOrderMark] String. When the encoding of the root element of the document is exactly UTF-8, UTF-16, or UTF-32 (or CCSID equivalent), the value indicates whether the document starts with a byte order mark (BOM). It there is a BOM then for UTF-8 encoding the value is 'UTF-8'; for UTF-16 encoding the value is 'UTF-16LE' or 'UTF-16BE'; for UTF-32 the value is 'UTF-32LE' or 'UTF-32BE'. If there is no BOM then the value is empty. When the encoding of the root element of the document is any other encoding, the value is empty. Grammar The grammar production for the overall document changes to: Document = UnicodeByteOrderMark Element Parsing When the dfdl:encoding property of the root element is specified, and is exactly one of UTF-8, UTF-16, or UTF-32 (or CCSID equivalents), then a DFDL parser will look for the appropriate byte order mark (BOM) as the very first bytes in the data stream. UTF-8. If a BOM is found then this is used to set the document information item 'unicodeByteOrderMark' property. If no BOM is found the parser takes no action. There is no need to model the BOM explicitly. UTF-16. If a BOM is found then this is used to set the document information item 'unicodeByteOrderMark' property value, and all data with dfdl:encoding UTF-16 throughout the rest of the stream are assumed to have the implied byte order . If no BOM is found then all data with dfdl:encoding UTF-16 throughout the rest of the stream are assumed to have big-endian byte order. There is no need to model the BOM explicitly. UTF-32. If a BOM is found then this is used to set the document information item 'unicodeByteOrderMark' property value, and all data with dfdl:encoding UTF-32 throughout the rest of the stream are assumed to have the implied byte order . If no BOM is found then all data with dfdl:encoding UTF-32 throughout the rest of the stream are assumed to have big-endian byte order. There is no need to model the BOM explicitly. When the dfdl:encoding property of the root element is specified, and is exactly one of UTF-16LE, UTF-16BE, UTF-32LE or UTF-32BE (or CCSID equivalents), then a DFDL parser will not look for appropriate BOM. The byte order to use is implicit in the encoding. If a BOM does appear at the start of the data stream, then it must be explicitly modelled as such, otherwise if parsed as part of an xs:string it will be interpreted as a zero-width non-breaking space (ZWNBS) character. The dfdl:byteOrder property is never used to establish the byte order for Unicode encodings. The parser never looks for a BOM at any other point in the data stream. If a BOM does appear at any other point, then it must be explicitly modelled, otherwise if parsed as part of an xs:string it will be interpreted as a ZWNBS character. Unparsing When the dfdl:encoding property of the root element is specified, and is exactly one of UTF-8, UTF-16 or UTF-32 (or CCSID equivalents), then a DFDL unparser will look in the infoset document information item for a byte order mark (BOM). UTF-8. If the document information item 'unicodeByteOrderMark' property value is 'UTF-8', the UTF-8 BOM is output as the very first bytes in the data stream. If the property is empty then no BOM is output. If the property has any other value, it is a processing error.There is no need to model the BOM explicitly. UTF-16. If the document information item 'unicodeByteOrderMark' property is 'UTF-16LE' or 'UTF-16BE', the corresponding UTF-16 BOM is output as the very first bytes in the data stream, and all data with dfdl:encoding UTF-16 throughout the rest of the document will be output with the implied byte order . If the property is empty then no BOM is output, and all data with dfdl:encoding UTF-16 throughout the rest of the document are assumed to have big-endian byte order. If the property has any other value, it is a processing error. There is no need to model the BOM explicitly. UTF-32. If the document information item 'unicodeByteOrderMark' property is 'UTF-32LE' or 'UTF-32BE', the corresponding UTF-32 BOM is output as the very first bytes in the data stream, and all data with dfdl:encoding UTF-32 throughout the rest of the document will be output with the implied byte order . If the property is empty then no BOM is output, and all data with dfdl:encoding UTF-32 throughout the rest of the document are assumed to have big-endian byte order. If the property has any other value, it is a processing error. There is no need to model the BOM explicitly. When the dfdl:encoding property of the root element is specified, and is exactly one of UTF-16LE, UTF-16BE, UTF-32LE or UTF-32BE (or CCSID equivalents), then a DFDL unparser will not look in the infoset document information item for a BOM and will not output a BOM. The byte order to use is implicit in the encoding. If a BOM does need to be output at the start of the data stream, then it must be explicitly modelled as such. The dfdl:byteOrder property is never used to establish the byte order for Unicode encodings. The unparser never outputs a BOM at any other point in the data stream. If a BOM needs to appear, then it must be explicitly modelled as such. ----------------------------------------------------------------------------- Explicit modelling of a BOM. Below is a natural way to model a Unicode data stream with an optional BOM. The parser starts with root element 'doc' and finds (scoped) dfdl:encoding is 'UTF-16BE' from the variable's default so it does not look for a BOM automatically. Then it parses element 'bom' and sets the variable according to the value. If there is no BOM in the data then the assert fails and the parser backtracks (note use of minOccurs="0"). Subsequent parsing of element 'data' uses the variable value. The model would also work if the variable's default had been 'UTF16'. The parser looks for the BOM explicitly in the stream, finds it, and sets the byte order. Then it parses element 'bom' but the assert fails and the parser backtracks. Subsequent parsing of element 'data' uses the variable value which will be the default of 'UTF-16', and byte order as per the BOM. (Some DFDL annotation syntax removed for clarity) <xs:schema ...> <dfdl:format encoding="{$myEncoding}" ... /> <dfdl:defineVariable name="myEncoding" type="xs:string" default="UTF-16BE"/> <xs:element name="doc"> <xs:complexType> <xs:sequence> <xs:element name="bom" type="xs:hexBinary" minOccurs="0" dfdl:lengthKind="explicit" dfdl:length="2"> <dfdl:assert test="{. eq x'FFFE' or . eq x'FEFF'}" /> <dfdl:setVariable ref="myEncoding" value="{if . eq x'FFFE then 'UTF16-LE'}" /> </xs:element> <xs:element name="data"> <xs:complexType> <xs:sequence> ... </xs:sequence> </xs:complexType> </xs:element> </xs:sequence> </xs:complexType> </xs:element> <xs:schema> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0