- dfdl-wg - lists.ogf.org

Fw: Action 148: pattern based lengths - suggested revised language
by Steve Hanson 26 Sep '11

26 Sep '11

On the call last week, it was agreed that a conservative approach to pattern based lengths was sufficient for DFDL 1.0. Revised text for 12.3.5.1: 12.3.5.1 Pattern-Based Lengths - Scanability dfdl:lengthKind 'pattern' may be specified for · elements of simple type with representation 'text' · elements of complex type For an element of complex type: all simple child elements must have representation 'text' and have the same encoding as the parent complex element, and all complex child elements must themselves follow 1 and 2 (recursively). Similar wording to apply to dfdl:assert testKind="pattern". Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2011-09-27
by Steve Hanson 26 Sep '11

26 Sep '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… The latest errata to the spec is here: http://forge.gridforum.org/sf/go/doc16280?nav=1 (updated 2011-09-08) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: time zones
by Suman Kalia 20 Sep '11

20 Sep '11

Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… ----- Forwarded by Suman Kalia/Toronto/IBM on 09/20/2011 11:58 AM ----- From: Suman Kalia/Toronto/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Steve Hanson/UK/IBM@IBMGB Date: 09/20/2011 11:51 AM Subject: Re: time zones Here are 2 ways to model it as discussed on the call today.. Mike - Based on what you have discovered for time zone in the range of +- nn:45, Pattern might be better in the long run.. Using Pattern facet <xsd:element name="calendarTimeZone" type= "tns:CalendarTimeZoneType" />   <xsd:simpleType name="CalendarTimeZoneType"> <xsd:restriction base="xsd:string"> <xsd:pattern value= "(UTC)([+-]([01]\d|\d)((([:][0-5]\d){1,2})?))?)" /> </xsd:restriction> </xsd:simpleType> Pros : Allows new times zones to be added without modification to spec as it just specifies the format.. Schema validation checks the format is valid , explicit checking for values is left to Implementation code. Explicit enumeration : - Strict checking at schema level. Validation error reported by schema. Any time a new time zone if added , spec and schema would require modification.  <xsd:simpleType name="CalendarTimeZoneEnum"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="UTC-12:00" /> <xsd:enumeration value="UTC-11:00" /> <xsd:enumeration value="UTC-10:00" /> <xsd:enumeration value="UTC-09:30" /> <xsd:enumeration value="UTC-09:00" /> <xsd:enumeration value="UTC-08:00" /> <xsd:enumeration value="UTC-07:00" /> <xsd:enumeration value="UTC-06:00" /> <xsd:enumeration value="UTC-05:00" /> <xsd:enumeration value="UTC-04:30" /> <xsd:enumeration value="UTC-04:00" /> <xsd:enumeration value="UTC-03:30" /> <xsd:enumeration value="UTC-03:00" /> <xsd:enumeration value="UTC-02:00" /> <xsd:enumeration value="UTC-01:00" /> <xsd:enumeration value="UTC" /> <xsd:enumeration value="UTC+01:00" /> <xsd:enumeration value="UTC+02:00" /> <xsd:enumeration value="UTC+03:00" /> <xsd:enumeration value="UTC+03:30" /> <xsd:enumeration value="UTC+04:00" /> <xsd:enumeration value="UTC+04:30" /> <xsd:enumeration value="UTC+05:00" /> <xsd:enumeration value="UTC+05:30" /> <xsd:enumeration value="UTC+05:45" /> <xsd:enumeration value="UTC+06:00" /> <xsd:enumeration value="UTC+06:30" /> <xsd:enumeration value="UTC+07:00" /> <xsd:enumeration value="UTC+08:00" /> <xsd:enumeration value="UTC+09:00" /> <xsd:enumeration value="UTC+09:30" /> <xsd:enumeration value="UTC+10:00" /> <xsd:enumeration value="UTC+10:30" /> <xsd:enumeration value="UTC+11:00" /> <xsd:enumeration value="UTC+11:30" /> <xsd:enumeration value="UTC+12:00" /> <xsd:enumeration value="UTC+12:45" /> <xsd:enumeration value="UTC+13:00" /> <xsd:enumeration value="UTC+14:00" /> </xsd:restriction> </xsd:simpleType> Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Suman Kalia/Toronto/IBM@IBMCA Date: 09/20/2011 11:04 AM Subject: time zones Hey Suman, I just googled, and this page says there are UTC-5:45 and UTC-12:45 time zones. That was news to me. I thought they were all :00 or :30. http://aa.usno.navy.mil/graphics/TimeZoneMap0911.pdf ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2125 | 100 Fifth Ave., 4th Floor, Waltham MA 02451 | mbeckerle.dfdl(a)gmail.com

1 0

OGF DFDL WG Call Minutes 2011-09-20
by Steve Hanson 20 Sep '11

20 Sep '11

Please find minutes from the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 148: pattern based lengths - suggested revised language
by Steve Hanson 20 Sep '11

20 Sep '11

For discussion on today's call... Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 20/09/2011 11:01 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, mbeckerle.dfdl(a)gmail.com, Date: 20/09/2011 10:17 Subject: Re: Action 148: pattern based lengths - suggested revised language I'd like to discuss on the WG call today. I think the conservative approach I outline below is consistent with what we do for complex elements and specified lengths, and I'd prefer to stick with that for 1.0. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Date: 25/08/2011 16:23 Subject: Re: Action 148: pattern based lengths - suggested revised language I think we can afford to be a little less conservative, actually. Let's suppose that we allow patterns regardless of dfdl:representation and regardless of the encoding. That will provide users with maximum flexibility, at the ( not very large ) risk that they will occasionally do something silly. We can put a note into the specification to the effect that patterns should usually be used only with character data, but can ( with care ) be used to match bytes if that is the only way to achieve the desired result. I may be missing something, but I don't see what harm we can cause ourselves or our users by doing this. My concern is that we could take away a lot of the power that patterns provide ( particularly for discriminators / asserts ) and then end up regretting it when some strange format pops up. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Date: 25/08/2011 15:00 Subject: Action 148: pattern based lengths - suggested revised language Hi Tim Please could you have a think about my conservative proposal below? Firstly, can we get away with restricting patterns to text, or will we need to use patterns to grab large amounts of data they may include binary content? Secondly, are we able to apply the same validation criteria to use of testKind pattern on an assert or discriminator as we are to use of lengthKind pattern? . Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 25/08/2011 14:44 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 27/07/2011 15:00 Subject: Re: pattern based lengths - suggested revised language I support what you call the conservative approach. I.e. require text when patterns are used. Mike ------------------------------------------------------------------------------------------- On Jul 27, 2011 5:53 AM, "Steve Hanson" <smh(a)uk.ibm.com> wrote: Hi Mike I don't think we can reduce the wording that much. The second paragraph is needed because it covers the binary case, where encoding is not actually used. I think we either need to be conservative and disallow the combination of binary & pattern, or leave the second paragraph as-is and effectively say that if you use binary with pattern then that is the behaviour. If we are to be conservative then: - For a simple element or simple type, disallow lengthKind="pattern" with binary rep. - For a complex element with lengthKind = "pattern", all children must have lengthUnits = "characters" (so text only) and the encoding of the children must be the same as the encoding of the parent. (We already have a similar rule for complex elements with specified length and lengthUnits = "characters"). We also allow asserts and discriminators to carry patterns which are applied straight at the current position in the data stream. It would be difficult to police the conservative rules here. But we need to say what encoding is used and we currently do not. I would say it must be the encoding of the element or group that carries the assert/discriminator. I said on the call that we had extended DFDL regular expressions so that raw hex bytes could be specified. However I don't see any evidence of this in the DFDL spec. This facility was something we added to IBM MRM for a retail format called TLOG which consists of delimited packed decimal data with hex indicator bytes, so we needed a way to match the hex indicator bytes as part of the regexp. However, I think this was only necessary because MRM has neither speculation nor discriminators, and in a DFDL version of TLOG I would use a discriminator. So I think my statement was in error, and I don't believe raw hex in DFDL regexps is needed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 > > > From: > "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> > To: > Steve Hanson/UK/IBM@IBMGB > Date: > 26/07/2011 17:30 > Subject: > pattern based lengths - suggested revised language > > > > I suggest this language to tighten up this whole section (replace both > paragraphs). Given the concerns of Tim, that we make sure DFDL > implementations don’t have to reimplement regexp matching, I think this is > sufficient. > 1.1.1.1 Based Lengths - Scanability > Any element (complex, simple text, simple binary) may have a > dfdl:lengthKind 'pattern'. When an element contains binary data, and > lengthKind=’pattern’ is used, then it is a schema definition error if the > character set encoding is not iso-8859-1. > > > (Possible generalization 1: allow other character sets, e.g., iso-8859-15 > as well. This is ok because 8859-15 still maps all 256 codepoints. But > this is a slippery slope. ) > > (Possible generalization 2: allow any character set, Ascii, ebcdic, > utf-16be, etc. Note that using any character encoding other than one which > maps a valid character to any 8-bit byte creates ambiguity: e.g, the > regexp “.” is one where we normally think it means “any character”. But > do we really mean “any byte” ? If the character set encoding doesn’t have > a given byte as a codepoint, then this question really matters.) > > > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > > > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: DFDL WG action 143: Proposed resolution
by Steve Hanson 20 Sep '11

20 Sep '11

For discussion on today's call... Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 20/09/2011 11:00 ----- From: Steve Hanson/UK/IBM To: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Cc: Richard Schofield/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 25/08/2011 13:48 Subject: RE: DFDL WG action 143: Proposed resolution Although this action was closed on 5th July WG call, I did find this one extra issue about WSP and textStandardZeroRep and we did not reach a conclusion. I will add it to WG call agenda. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Tim Kimber/UK/IBM@IBMGB, Steve Hanson/UK/IBM@IBMGB Cc: Richard Schofield/UK/IBM@IBMGB Date: 05/07/2011 21:49 Subject: RE: DFDL WG action 143: Proposed resolution Well….. The use case I have is from ancient mainframe style data. Numbers that are all zero are suppressed and all blanks are output instead of anything with zeros in it, or even a single 0 character. This is for data that was both data and “report” simultaneously, i.e., people looked at it. …mikeb From: Tim Kimber [mailto:KIMBERT@uk.ibm.com] Sent: Tuesday, July 05, 2011 3:00 PM To: Steve Hanson Cc: mbeckerle.dfdl(a)gmail.com; Richard Schofield Subject: Re: DFDL WG action 143: Proposed resolution We might want to bear this scenario in mind: Field 'integerA' has explicit length of 5 and textNumberZeroRep='%WSP*;'. The next field is a string which happens to start with some white space. The parser must be careful to ensure that the matching of the %WSP; against the data stream does not walk on into the following field's data. On balance, I would prefer an alternative solution based on either defaulting or nil handling: a) We could define 'empty' in terms of the trimmed value when padKind != "none". That would allow this to be handled by setting the pad character to space and xs:default to zero. b) Without any change to the specification, the user could - set nilLiteralCharacter="%SP;" and xs:nillable="true" or - set nilLiteralValue of %ES; , set textStringPadCharacter="'%SP;", set xs:nillable="true" Either of these would result in a nil in the infoset. This may not be the exact infoset that the user wants, but DFDL ( esp. DFDL 1.0 ) does not guarantee that DFDL emits the exact infoset that you want. In other words, I don't think the spaces are really a representation of the value zero. They are padding or filling that the modeller wants to treat as zero. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: mbeckerle.dfdl(a)gmail.com Cc: Tim Kimber/UK/IBM@IBMGB Date: 05/07/2011 18:49 Subject: DFDL WG action 143: Proposed resolution Mike I was about to mail this out but noticed that textStandardZeroRep was included in the list, and was being prohibited from using WSPx entitities. This would not conveniently allow the scenario you mentioned on the call today where all spaces was allowed to represent zero. I can remove textStandardZeroRep from the list, and leave it as per spec if you prefer. Tim - do you have a problem with that? Regards Steve -------------------------------------------------------------- Following is the resolution from action 143 An errata document accompaniment to the DFDL 1.0 specification is under production and this will be added to it. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/07/2011 18:33 ----- Clarification to the allowable content that is allowed in some of the DFDL properties that are defined as type 'DFDL String Literal'. The properties below do not need the full power of DFDL String Literal. Properties escapeCharacter and escapeEscapeCharacter Property value must resolve to a single character DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties escapeBlockStart and escapeBlockEnd DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties textStringPadCharacter, textNumberPadCharacter, textBooleanPadCharacter and textCalendarPadCharacter Property value must resolve to a single character or a single byte value DFDL character entities are allowed The raw byte entity ( %#r ) is allowed subject to the restrictions already documented for these properties DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties textStandardDecimalSeparator and textStandardGroupingSeparator Property value must resolve to a single character DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties textStandardInfinityRep, textStandardNanRep, textStandardZeroRep DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Properties textBooleanTrueRep and textBooleanFalseRep DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Property nilValue DFDL character entities are allowed The raw byte entity ( %#r ) is allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are allowed Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2011-09-20
by Steve Hanson 19 Sep '11

19 Sep '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… The latest errata to the spec is here: http://forge.gridforum.org/sf/go/doc16280?nav=1 (updated 2011-09-08) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2011-09-13 - call cancelled
by Steve Hanson 13 Sep '11

13 Sep '11

Apologies but I have to be elsewhere and can't make the call. I'd appreciate if folk could review the updated errata document and let me know if there is anything in there that does not reflect recent WG call agreements, or if anything is missing. And check out the spec on the web. If you have any ideas for improvement let me know. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/09/2011 10:48 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 08/09/2011 15:58 Subject: OGF DFDL WG Call Agenda 2011-09-13 Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… The latest errata to the spec is here: http://forge.gridforum.org/sf/go/doc16280?nav=1 (updated 2011-09-08) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Cancelled: OGF DFDL Working Group weekly call
by Steve Hanson 13 Sep '11

13 Sep '11

Unfortunately I can't make the call today.

1 0

OGF DFDL WG Call Agenda 2011-09-13
by Steve Hanson 08 Sep '11

08 Sep '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… The latest errata to the spec is here: http://forge.gridforum.org/sf/go/doc16280?nav=1 (updated 2011-09-08) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0