January 2010 - dfdl-wg - lists.ogf.org

Fw: DFDL: TLOG proposal
by Steve Hanson 29 Jan '10

29 Jan '10

Updated below as agreed on DFDL WG call Jan 29th. Regards Steve Hanson Programming Model Architect, WebSphere Message Broker, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 29/01/2010 17:40 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 26/01/2010 12:26 Subject: DFDL: TLOG proposal TLOG The individual fields are a mixture of ASCII strings, proprietary packed decimals, and the occasional pure binary data. All fields are delimited by a separator. Fields of all types can be fixed length or variable length with a maximum. Pure binary data is preceded by a field giving the actual length. All lengths in bytes. Packed decimals. Like a packed decimal in the IBM sense. These can carry negative numbers but use a leading xD sign nibble. No sign nibble if positive or unsigned. Odd number of digits (including sign if present) are padded with xF nibble. This is best illustrated using examples. 1234 => x12x34 123 => xF1x23 -1234 => xFDx12x34 -123 => xD1x23 Proposal 1) The 'variable length with a maximum' will be handled using a post-timing assertion. Note this only applies on parsing. ** 2) dfdl:lengthKind 'delimited' is permitted for numbers when dfdl:representation is 'binary' and dfdl:binaryNumberRep is 'packed' or 'bcd' because it is possible to know in advance the range of bytes being used, and therefore to choose suitable delimiters. 3) Core DFDL 1.0 will not be enhanced to handle the TLOG packed decimal type. A future version of DFDL will provide an extensibility mechanism that allows user-defined types to be handled. In the 1.0 timeframe IBM may implement its own proprietary extension to handle this type. ** While this can result in output from a DFDL unparser that can not be re-parsed, that is a problem general to the use of assertions, and a future version of DFDL may choose to change this by enhancements to the assertion annotation. Regards Steve Hanson Programming Model Architect, WebSphere Message Broker, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Action 071 - modified
by Steve Hanson 29 Jan '10

29 Jan '10

Updated below. Agreed on DFDL WG extra call on Fri 29th Jan. Regards Steve Hanson Programming Model Architect, WebSphere Message Broker, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 29/01/2010 17:27 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 26/01/2010 12:52 Subject: DFDL: Action 071 Changed unparsing behaviour - we must honour the property - the existing behaviour of always writing the initiator means we can not successfully re-parse if writing empty content and enum is 'suppress'. When reading, assume that section 15.13 has been updated to include complex as well as simple elements. No change to enums. missingValueInitiatorPolicy Enum Valid values ‘required', ‘prohibited' Specifies whether to expect an initiator when an element is missing. Ignored unless dfdl:initiator is specified and is not "" (empty string). 'required' - Indicates that the dfdl:initiator followed by empty content is the required syntax to indicate that the element is missing. 'prohibited' - Indicates that empty content is the required syntax to indicate that the element is missing. The presence of an initiator implies that real content must follow. Use of ‘prohibited’ implies an ordered sequence. If used on an initiated element of an unordered group it is a schema definition error. If the element is required, defaulting occurs as defined above. This property also applies on unparsing, when the data to be written (after nil value and default value processing) is empty content. Annotation: dfdl:element Unparsing. The branch of a choice output when a complex element is required but missing from the infoset is the first branch of the choice that does not result in a processing error. Regards Steve Hanson Programming Model Architect, WebSphere Message Broker, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Invitation: OGF DFDL Working Group extra call (2 Feb 13:00 GMT in DE3S09 for IBM Hursley participants (Peter - you have this room booked already))
by Steve Hanson 29 Jan '10

29 Jan '10

1 0

Agenda for OGF DFDL WG call 29 January 2010- 13:00 UK (8:00 ET)
by Alan Powell 29 Jan '10

29 Jan '10

1 Review Schedule OGF prereview is confirmed to take about 4 weeks assuming no document updates are required. We are behind schedule to be available for public review by March. Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 2. Go through Actions 3. TLog Confirm decision to allow dfdl:lengtKind=delimited for all binary representation 4 Action 071 Semantics of length=0, nil handling and defaults. 5. Go through Actions 6. Discriminators Review two options (attached) 7. Draft 037 review issues - Case of enumerations. We should follow the XSDL convention which is that enumerations are case sensitive - dfdl:lengthKind='Pattern scannability: A complex element with lengthKind=Pattern will use its dfdl:encoding property as the encoding when scanning its children irrespective of the child's encoding property. Go through unanswered issues in Mike's comments document Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Remaining 037 review issues for today WG call (20 Jan)
by Alan Powell 29 Jan '10

29 Jan '10

I have answered most of the issues and comments raised by Steve and Mike but some need further discussion. Issues from Steve H General. Although dfdl:encoding enums are case insensitive, we should stick to UC throughout in examples. 2. I agree with the existing comment that the RFC2119 key words should be upper case. 14.3.4. There are type/rep combinations where lengthKind="implicit" is not allowed - so saying that 'pattern' is replaced by 'implicit' on unparsing does not work. TBD 16.2. I'm not sure that scannability in this constant encoding sense is necessary for patterns. I can create a regular expression that extracts all characters up to hex value xXX or all characters up to xYY, thereby treating the content as an encoding in-sensitive black box. Issues from Mike B · Tracker issue: codepoints outside BMP, as literals and in data. · If I put in a value that requires use of a high/low surrogate pair, is that an error, does it require me to put in two separate %#...; thingys, one for each of the surrogates (in which case these are not really code points in ISO10646). If I put in a codepoint for one of the supplemental characters and the schema itself is written in UTF-16 then that has to translate into literal surrogate pair. Ok, but I?m very uncertain about all this stuffTracker Issue: illegal character encodings for parsing and unparsing. TBD: how do these make it into the infoset or are they replaced, and if so how TBD: can one represent these in the infoset for output? Ideally not, but? · Tracker Issue: Processing-time Schema Definition Errors This section (2.3.1 in this draft), is problematic as we?re trying to allow simple DFDL implementations to not do a bunch of static checking, yet if implementations differ on when Schema Definition errors are detected, then the second paragraph says they are converted to processing errors. This lets different implementations do very different things in terms of how the speculative parsing back-tracks around. Grammar ambiguity is a very tricky case. Unless a DFDL implementation can prove a grammar to be unambiguous, then it is very hard to say that any particular combinatino of delimiters make up a legal DFDL schema definition. If the parser simply fails because the grammar was ambiguous, there?s no way to tell the difference between this and just broken data without proving the grammar is unambiguous. In general it is formally undecidable whether a grammar is ambiguous or unambiguous. ( http://books.google.com/books?id=lIuu53IcKWoC&pg=PT217&lpg=PT217&dq=proving… ) Since DFDL v1.0 doesn?t allow recursive declarations/definitions, it may be possible to provide the ambiguity or unambiguity of a DFDL schema (or rather, the data syntax grammar described by it ? if you want to bother to distinguish the two), but recursion isn?t something we want to rule out for the future, so Type checking is decidable in DFDL?s expression language, so we could always detect type safety before run time; however, if we allow a simplistic DFDL implementation to just check types at run time then this would, by the definition in this section (2.3.1), issue processing errors when it detects these at run time, thereby allowing backtracking of the speculative parser to be driven off of type-checks in the expression language. It seems to me that we need to find a way to put this problem back into the hands of the user, and say that a schema where this actually matters (one where a type error causes a backtrack, which ultimately causes a successful parse) are illegal but implementations are allowed to not detect this particular illegality. It seems to me we need to put this problem back into the hands of the user. · Tracker Issue: "round trip" for infoset. Should we omit the whole point? · Tracker Issue: [schema] is an absolute or relative SCD. Why bother allowing absolute? · Tracker Issue: Glossary as the place for centralized definitions, or should they be repeated there, but also introduced at point of first use, or should we put the definitions only at the places where they are discussed, and xref from the glossary? · TBD: Issue - semantics of expressions containing relative paths that are inherited via ref to a dfdl:defineFormat. (also section 10.3) · TBD: Issue - XPath term - we are not consistent about using the term XPath, or "expression" when referring to our expression language. I prefer to call it our expression language, and then in the section that defines it state that it is a strict subset of XPath 2.0. · TBD: Issue - fn:position is unclear given that we've just said we don't support sequences in the expression language. · TBD: Issue - order of sections. Scoping rules section should come before variables section, which uses these concepts. TBD: Issue: Case sensitivity of enum names - did we say whether this is case sensitive or not? I believe it should be case sensitive. · Issue: dfdl:representation - Strings in binary rep. I see no reason why elements of type xs:string will examine dfdl:representation. They shouldn?t' care what it is, they are always "text". I should be able to specify a bunch of inter-mixed binary number and string elements without having to specify dfdl:representation="text' just to avoid an error on the string type elements. I believe xs:string type ignores dfdl:representation (always behaves as if dfdl:representation is 'text').(If we change this then the property precedence section for simpletypes changes slightly as representation="text" is implied if type is string.) That will make it impossible to introduce a binary representation of text later textStringPadCharacter textNumberPadCharacter - did we agree that this character must be a "minimum width" character if the char set encoding is variable width? (i.e., the pad char must be 1 byte if the encoding is UTF-8. numberInfinityRep numberNanRep - Is this applicable only to xs:double and xs:float? Also, what I've seen requires a distinction of sign. I.e., there are positive and negative infinities often printing as -inf and +inf. · TBD: Issue - \n in regular expressions - clarify relationship of this to entities like NL entity. Also, if I include an entity like WSP* in a regular expression (can I?) does it then match accordingly? It appears that some of our multi-valued entities like WSP+ create conditional "matching" behavior without having to use regular expressions, e.g., when WSP+ is used as a separator. But can you use entities like WSP+ in a regular expression? It seems you should be able to use regular "single valued" entities in a regular expression, its these multi-valued ones that have tricky semantics. Added Unicode values to /n, /t,/r. Disallow DFDL entities in regular expressions. 14.1 Alignment - TBD: Issue - zero-based thinking here. But all the bits stuff and everything else in DFDL uses 1-based reasoning. Need to revisit to make this sensible for 1 based world. Added implicit alignment table. TBD zero-based finalTerminatorCanBeMissing - spec is not clear. Also is there a finalSeparatorCanBeMissing Chaned to finalDocumentTerminatorCanBeMissing and finalDocumentSeparatorCanBeMissing. Not sure where finalDocumentSeparatorCanBeMissing should be specified. Looks odd on 'distinguished root'. These properties operate differently from other properties as they are defined on the 'distinguished root' but affect some lower down element. Effectively they are put in scope by a different mechanism Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Minutes for OGF DFDL Working Group Call, January-26 & 27-2010
by Alan Powell 28 Jan '10

28 Jan '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, January-26 & 27-2010 Attendees Mike Beckerle (Oco) (27) Steve Hanson (IBM) (26 & 27) Alan Powell (IBM) (26 & 27) Steve Marting (Progeny) (27) Suman Kalia (IBM) (27) Peter Lambros (IBM) (26) Tim Kimber(IBM) (26 & 27) Apologies Stephanie Fetzer (IBM) 1. Discriminators There was a long discussion of Alan's document which merged the 'Resolving Uncertainty' and 'Discriminators' documents. It proposes that discriminators 'confirm the existence of their parent sequence' and did not allow discriminators on simple elements to confirm the existence of that element based on its contents or other data. Mike pointed out that this did not allow the uses case of a previous field in the data stream (eg a flag) indicating the existence of a field without wrapping that field in a extra sequence. The uses cases are: - A sequence is 'known to exist' by testing one or more of its children - A sequence is 'known to exist' by testing some previous data - A simple element is 'known to exist' by testing some previous data - A simple element is 'known to exist' by testing its content An alternative proposal was discussed where a discriminator where allowed on particles and 'confirmed the existence of the a component' Alan will write up both proposals. Also mentioned was the meaning of minOccurs= 0 on a choice branch. Steve will ask Sandy Gao 2. Unparsing lengthKind=Pattern The table below was modified to make the text unparing length the minLength/outputMinLength. Mike suggested that lengtKind pattern should be limited to text fields but need to understand Tlog requirements. Representation text binary String minLength Not applicable Float outputMinLength 32 Double outputMinLength 64 Decimal/Integer outputMinLength Minimum number of bytes to represent significant digits and sign Long, UnsignedLong outputMinLength packed/bcd : as decimal binary: 64 Int, Unsignedint outputMinLength packed/bcd : as decimal binary: 32 Short, Unsignedshort outputMinLength packed/bcd : as decimal binary: 16 Byte, Unsignedbyte outputMinLength packed/bcd : as decimal binary: 8 DateTime outputMinLength packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Date outputMinLength packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Time outputMinLength packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Boolean outputMinLength 32 HexBinary Not applicable Length of infoset value 3. TLog Reviewed Steve's proposal Proposal 1) The 'variable length with a maximum' can not be handled using a post-timing assertion, because assertions only apply on parsing. Given that IBM MRM allows you to model this case, I think we should allow dfdl:length & dfdl:lengthUnits to be specified when dfdl:lengthKind is 'delimited' or 'pattern'. Extraction during parse is still by scanning. If the physical representation exceeds that length after extraction, it is a processing error. Similarly when unparsing, if the physical representation exceeds that length prior to output, it is a processing error. 2) dfdl:binaryNumberRep is extended with another value 'tlog'. Associated property dfdl:binaryDecimalVirtualPoint is applicable. Associated property dfdl:binaryPackedSignCodes is not applicable, there being only xD to indicate negative numbers, and no distinct nibbles for unsigned or zero. 3) dfdl:lengthKind 'delimited' is permitted for numbers when dfdl:representation is 'binary' and dfdl:binaryNumberRep is 'packed' or 'bcd' or 'tlog' because it is possible to know in advance the range of bytes being used, and therefore to choose suitable delimiters. 1) can be achieved on parsing with an assert but cannot be enforced on unparsing 2) Discussed the desirability of properties/enumerations specific particular formats versus providing more extensibility. Decided neither were needed for DFDL v1 3) Steve H felt that this was needed. It was agreed to add to spec (I need to confirm this) 4 Action 071 Semantics of length=0, nil handling and defaults. Not discussed 5. Go through Actions 6. Draft 037 review issues - Case of enumerations. We should follow the XSDL convention which is that enumerations are case sensitive - dfdl:lengthKind='Pattern scannability: A complex element with lengthKind=Pattern will use its dfdl:encoding property as the encoding when scanning its children irrespective of the child's encoding property. Go through unanswered issues in Mike's comments document 7 Review Schedule OGF prereview is confirmed to take about 4 weeks assuming no document updates are required. We are behind schedule to be available for public review by March. Draft 038 will be available at the end of this week. Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Meeting closed, 15:10 Next call Friday 29 January 2010 13:00 UK Next action: 077 Actions raised at this meeting No Action 076 SH semantics of minOccurs= 0 on choice branches Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 13/01: Stephaine took us through a description of WTX identifiers. Mike agreed to write up in DFDL terms. 20/01: Mike will write up 27/01: further discussion of disciminators 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progress 20/01: no progress 27/01: no progress 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 13/01: no update 20/01: no update 27/01: Session confirmed 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 13/01: SH proposal review. Minor updates to be made 20/01: Reviewed updated proposal. Need to agree on unparsing empty choices. 27/01: Steve H had sent update but not discussed due to lack of time 074 SH: Proposal for parsing TLog 27/01: Proposal discussed and agreed to allow delimited for binary packed/bcd fields 075 SH: rewrite empty sequences section 27/01: Steve provide written section 076 SH semantics of minOccurs= 0 on choice branches Closed actions No Action Work items: No Item target version status 005 Improvements on property descriptions not started 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 069 ICU fractional seconds 039 070 Write DFDL primer 071 Write test cases. 072 it is a processing error if the number of occurrences in the data does not match the value of the expression or prefix 039 073 Rename dfdl:separatorPolicy="required" to "always". 039 Defferred untilaction 071 agreed 078 document UPA checks 039 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Invitation: OGF DFDL Working Group extra call (29 Jan 13:00 GMT in Participant Passcode: 8190865 )
by Steve Hanson 28 Jan '10

28 Jan '10

1 0

Fw: [wg-all] OGF28 Registration is now open!
by Steve Hanson 27 Jan '10

27 Jan '10

----- Forwarded by Steve Hanson/UK/IBM on 27/01/2010 16:19 ----- From: Joel Replogle <replogle(a)ogf.org> To: Steve Hanson/UK/IBM@IBMGB Date: 27/01/2010 16:04 Subject: Re: [wg-all] OGF28 Registration is now open! Hi Steve - All the group sessions were accepted, including DFDL. We are working on the schedule and hope to have a draft soon for review. Thanks, Joel From: Joel Replogle <replogle(a)ogf.org> To: wg-all(a)ogf.org Date: 27/01/2010 05:47 Subject: [wg-all] OGF28 Registration is now open! Sent by: wg-all-bounces(a)ogf.org All - I'm pleased to announce that registration for OGF28 is now open! OGF28 will be held at the Ludwig-Maximilians-Universität München in Munich, Germany on March 15-19, 2010. The theme is "The Future of Distributed Computing Infrastructures (DCI)" and will prove to be the premier event relating to DCI, Clouds and Grids in Europe this spring. To register, please visit: http://www.ogf.org/OGF28/registration.php or go directly to the registration site: http://www.interplan.de/reg/2010/OGF10/ See you in Munich! Joel -------------------------------------------------------------------- Joel Replogle Executive Director, Open Grid Forum replogle(a)ogf.org http://www.ogf.org -- The wg-all mailing list is an umbrella list for all OGF mailing lists. If you are subscribed to any OGF mailing list, you will receive mails to wg-all. You can unsubscribe from this list on http://www.ogf.org/cgi-bin/perl/unsubscribe-wg-all.pl wg-all mailing list wg-all(a)ogf.org http://www.ogf.org/mailman/listinfo/wg-all Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------------------------------------------------------------- Joel Replogle Executive Director, Open Grid Forum replogle(a)ogf.org http://www.ogf.org Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Uncertainty and Discriminators
by Alan Powell 27 Jan '10

27 Jan '10

There was quite a lot of overlap between the Discriminators document and the previous Resolving Uncertainty document so I merged the two together and incorporated Suman's and Steve's comment. It is a bit of a rushed job but should help the discussion. I didn't have time for examples. Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com ----- Forwarded by Alan Powell/UK/IBM on 27/01/2010 12:10 ----- From: Alan Powell/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 26/01/2010 12:59 Subject: [DFDL-WG] Fw: discriminators - such as it is Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com ----- Forwarded by Alan Powell/UK/IBM on 26/01/2010 12:59 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Alan Powell/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 21/01/2010 16:12 Subject: discriminators - such as it is Alan, This is where I am with discriminators. I won't have an opportunity to work on this again for a bit, and it's close, though not perfect. In particular, there's a numbered list of points here, and the order of those points is probably not right. ...mikeb Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "dfdl-discriminators.doc" deleted by Alan Powell/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Fw: Fw: discriminators - such as it is
by Suman Kalia 26 Jan '10

26 Jan '10

- some comments / clarifications.. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 01/26/2010 06:12 PM ----- From: Alan Powell <alan_powell(a)uk.ibm.com> To: dfdl-wg(a)ogf.org Date: 01/26/2010 08:01 AM Subject: [DFDL-WG] Fw: discriminators - such as it is Sent by: dfdl-wg-bounces(a)ogf.org Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com ----- Forwarded by Alan Powell/UK/IBM on 26/01/2010 12:59 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Alan Powell/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 21/01/2010 16:12 Subject: discriminators - such as it is Alan, This is where I am with discriminators. I won't have an opportunity to work on this again for a bit, and it's close, though not perfect. In particular, there's a numbered list of points here, and the order of those points is probably not right. ...mikeb Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

Fw: Draft 038 available - update
by Alan Powell 26 Jan '10

26 Jan '10

To make it easier to identify changes in this draft I have extended the change history table and uploaded a new version. This is the only change. The new version can be access from the same link below. Latest entry at the top please Version Author/ Contributor History Date(yyyy-mm-dd) 038 Mike Beckerle Alan Powell Steve Hanson Changes from IBM implementation team review. See tracker items Changes from Steve Hansonerle review Changes from Mike Beckerle review Changed binaryFloatRepresentation to binaryFloatRep added implicit alignment table changed 'know length' to specified length' Significant changes to length section. Added lengthUnit='bits' Added resolving points of uncertainty and discriminators Removes occursCountKind 'useAvailableSpace' Case of enumerations. We should follow the XSDL convention which is that enumerations are case sensitive Restrictions on use of 'special' entities in regular expressions lengthKind=Pattern will use its dfdl:encoding property Last 'postFix' separator is not optional Terminators are mandatory. Added dfdl:documentFinalTerminatorCanBeMissing dfdl:documentFinalSeparatorCanBeMissing Added missingValueInitiatorPolicy (but not complete yet) 2010-01-05 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com ----- Forwarded by Alan Powell/UK/IBM on 26/01/2010 17:28 ----- From: Alan Powell/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 22/01/2010 17:47 Subject: [DFDL-WG] Draft 038 available All DFDL V1 draft 038 is available at http://forge.gridforum.org/sf/go/doc15851?nav=1 There are many changes as a result of the review of draft 037. Documents responding to the comments are attached. Some issues are still being resolved. There have been significant changes to the Length section that should be thoroughly reviewed. This draft should be the last one before the draft we submit to OGF. I do not think that it is ready for submission. 038 Mike Beckerle Alan Powell Steve Hanson Changes from IBM implementation team review. See tracker items Changes from Steve Hanson review Changes from Mike Beckerle review Changed binaryFloatRepresentation to binaryFloatRep added implicit alignment table changed 'know length' to specified length' Significant changes to length section. Added lengthUnit='bits' Added resolving points of uncertainty and discriminators but not complete Removed occursCountKind 'useAvailableSpace' 2010-01-05 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "V37_Implementation Team v1.doc" deleted by Alan Powell/UK/IBM] [attachment "ogf-dfdl-v1.0-Core-037.final.MB.Comments.doc" deleted by Alan Powell/UK/IBM] [attachment "ogf-dfdl-v1.0-Core-037.final.SH.Comments.doc" deleted by Alan Powell/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Agenda for OGF DFDL WG call 27 January 2010- 13:00 UK (8:00 ET)
by Alan Powell 26 Jan '10

26 Jan '10

As not many made the extra call today we will use the same agenda. On todays call we discussed the discriminator document. Alan will update it for this call. Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com ----- Forwarded by Alan Powell/UK/IBM on 26/01/2010 17:08 ----- From: Alan Powell/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 26/01/2010 12:11 Subject: [DFDL-WG] Agenda for OGF DFDL WG call 26 January 2010- 13:00 UK (8:00 ET) 1. Discriminators Review Mikes writeup 2. Unparsing lengthKind=Pattern Representation text binary String unpadded length Not applicable Float unpadded length 32 Double unpadded length 64 Decimal/Integer unpadded length Minimum number of bytes to represent significant digits and sign Long, UnsignedLong unpadded length packed/bcd : as decimal binary: 64 Int, Unsignedint unpadded length packed/bcd : as decimal binary: 32 Short, Unsignedshort unpadded length packed/bcd : as decimal binary: 16 Byte, Unsignedbyte unpadded length packed/bcd : as decimal binary: 8 DateTime unpadded length packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Date unpadded length packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Time unpadded length packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Boolean Length of textBooleanTrue or textBooleanFalse 32 HexBinary Not applicable Length of infoset value 3. TLog 4 Action 071 Semantics of length=0, nil handling and defaults. 5. Go through Actions 6. Draft 037 review issues - Case of enumerations. We should follow the XSDL convention which is that enumerations are case sensitive - dfdl:lengthKind='Pattern scannability: A complex element with lengthKind=Pattern will use its dfdl:encoding property as the encoding when scanning its children irrespective of the child's encoding property. Go through unanswered issues in Mike's comments doecumnet 7 Review Schedule OGF prereview is confirmed to take about 4 weeks assuming no document updates are required. We are behind schedule to be available for public review by March. Draft 038 will be available at the end of this week. Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: discriminators - such as it is
by Alan Powell 26 Jan '10

26 Jan '10

Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com ----- Forwarded by Alan Powell/UK/IBM on 26/01/2010 12:59 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Alan Powell/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 21/01/2010 16:12 Subject: discriminators - such as it is Alan, This is where I am with discriminators. I won't have an opportunity to work on this again for a bit, and it's close, though not perfect. In particular, there's a numbered list of points here, and the order of those points is probably not right. ...mikeb Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Action 071
by Steve Hanson 26 Jan '10

26 Jan '10

Proposal as per last week's minutes: We should be consistent with the similar property for missing separators. So I've changed the enums too - now 'require' and 'suppress', Also changed unparsing behaviour - we must honour the property - the existing behaviour of always writing the initiator means we can not successfully re-parse if writing empty content and enum is 'suppress'. When reading, assume that section 15.13 has been updated to include complex as well as simple elements. missingValueInitiatorPolicy Enum Valid values ‘require', ‘suppress' Specifies whether to expect an initiator when an element is missing. Ignored unless dfdl:initiator is specified and is not "" (empty string). 'require' - Indicates that the dfdl:initiator followed by empty content is the required syntax to indicate that the element is missing. 'suppress' - Indicates that empty content is the required syntax to indicate that the element is missing. The presence of an initiator implies that real content must follow. Use of ‘suppress’ implies an ordered sequence. If used on an initiated element of an unordered group it is a schema definition error. If the element is required, defaulting occurs as defined above. This property also applies on unparsing, when the data to be written (after nil value and default value processing) is empty content. Annotation: dfdl:element We should similarly change the enums for nilValueInitiatorPolicy to 'require' and 'suppress'. And we should back out work item 071 which suggests changing dfdl:separatorPolicy enumeration from 'require' to 'always'. That means 'always' is matched with 'suppress' - but always is not a verb. Unparsing. Which branch of a choice do we output when a complex element is required but missing from the infoset? I think it should be the first branch of the choice that does not result in a processing error. Regards Steve Hanson Programming Model Architect, WebSphere Message Broker, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: TLOG proposal
by Steve Hanson 26 Jan '10

26 Jan '10

As per minutes from last week's call - here is proposal for TLOG after more research. TLOG The individual fields are a mixture of ASCII strings, proprietary packed decimals, and the occasional pure binary data. All fields are delimited by a separator. Fields of all types can be fixed length or variable length with a maximum. Pure binary data is preceded by a field giving the actual length. All lengths in bytes. Packed decimals. Like a packed decimal in the IBM sense. These can carry negative numbers but use a leading xD sign nibble. No sign nibble if positive or unsigned. Odd number of digits (including sign if present) are padded with xF nibble. This is best illustrated using examples. 1234 => x12x34 123 => xF1x23 -1234 => xFDx12x34 -123 => xD1x23 Proposal 1) The 'variable length with a maximum' can not be handled using a post-timing assertion, because assertions only apply on parsing. Given that IBM MRM allows you to model this case, I think we should allow dfdl:length & dfdl:lengthUnits to be specified when dfdl:lengthKind is 'delimited' or 'pattern'. Extraction during parse is still by scanning. If the physical representation exceeds that length after extraction, it is a processing error. Similarly when unparsing, if the physical representation exceeds that length prior to output, it is a processing error. 2) dfdl:binaryNumberRep is extended with another value 'tlog'. Associated property dfdl:binaryDecimalVirtualPoint is applicable. Associated property dfdl:binaryPackedSignCodes is not applicable, there being only xD to indicate negative numbers, and no distinct nibbles for unsigned or zero. 3) dfdl:lengthKind 'delimited' is permitted for numbers when dfdl:representation is 'binary' and dfdl:binaryNumberRep is 'packed' or 'bcd' or 'tlog' because it is possible to know in advance the range of bytes being used, and therefore to choose suitable delimiters. Regards Steve Hanson Programming Model Architect, WebSphere Message Broker, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG call 26 January 2010- 13:00 UK (8:00 ET)
by Alan Powell 26 Jan '10

26 Jan '10

1. Discriminators Review Mikes writeup 2. Unparsing lengthKind=Pattern Representation text binary String unpadded length Not applicable Float unpadded length 32 Double unpadded length 64 Decimal/Integer unpadded length Minimum number of bytes to represent significant digits and sign Long, UnsignedLong unpadded length packed/bcd : as decimal binary: 64 Int, Unsignedint unpadded length packed/bcd : as decimal binary: 32 Short, Unsignedshort unpadded length packed/bcd : as decimal binary: 16 Byte, Unsignedbyte unpadded length packed/bcd : as decimal binary: 8 DateTime unpadded length packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Date unpadded length packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Time unpadded length packed/bcd : as decimal binarySeconds: 32, binaryMilliseconds:64 Boolean Length of textBooleanTrue or textBooleanFalse 32 HexBinary Not applicable Length of infoset value 3. TLog 4 Action 071 Semantics of length=0, nil handling and defaults. 5. Go through Actions 6. Draft 037 review issues - Case of enumerations. We should follow the XSDL convention which is that enumerations are case sensitive - dfdl:lengthKind='Pattern scannability: A complex element with lengthKind=Pattern will use its dfdl:encoding property as the encoding when scanning its children irrespective of the child's encoding property. Go through unanswered issues in Mike's comments doecumnet 7 Review Schedule OGF prereview is confirmed to take about 4 weeks assuming no document updates are required. We are behind schedule to be available for public review by March. Draft 038 will be available at the end of this week. Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Invitation: OGF DFDL Working Group extra call (26 Jan 13:00 GMT in Participant Passcode: 8190865 )
by Steve Hanson 26 Jan '10

26 Jan '10

1 0

Draft 038 available
by Alan Powell 22 Jan '10

22 Jan '10

All DFDL V1 draft 038 is available at http://forge.gridforum.org/sf/go/doc15851?nav=1 There are many changes as a result of the review of draft 037. Documents responding to the comments are attached. Some issues are still being resolved. There have been significant changes to the Length section that should be thoroughly reviewed. This draft should be the last one before the draft we submit to OGF. I do not think that it is ready for submission. 038 Mike Beckerle Alan Powell Steve Hanson Changes from IBM implementation team review. See tracker items Changes from Steve Hanson review Changes from Mike Beckerle review Changed binaryFloatRepresentation to binaryFloatRep added implicit alignment table changed 'know length' to specified length' Significant changes to length section. Added lengthUnit='bits' Added resolving points of uncertainty and discriminators but not complete Removed occursCountKind 'useAvailableSpace' 2010-01-05 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Minutes for OGF DFDL Working Group Call, January-20-2010
by Alan Powell 21 Jan '10

21 Jan '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, January-20-2010 Attendees Mike Beckerle (Oco) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Suman Kalia (IBM) Peter Lambros (IBM) Apologies Stephanie Fetzer (IBM) Tim Kimber(IBM) 1. 045 - Disciminators Mike has started to write proposal. 2 Unparsing lengthKind = 'pattern' The Specification currently says that on unparsing dfdl:lengthKind='pattern' behaves like dfdl:lenghtKind='implict' but since we limited 'implict' to certain logical/representation combinations this is no longer correct. Proposals 1. When unparsing complex elements with dfdl:lengthKind='pattern', the length is the length of the children (same as 'implicit') When unparsing simple elements with dfdl:lengthKind='pattern' the length is the implicit length for those that have one and length of the data supplied in the infoset converted to the representation with no padding/filling for the rest. For string/text and hexbinary/binary that is reasonable For number/text/standard and zoned it is governed by the numberPattern For number/binary/ binary use 'implicit' lengths For number/binary/packed or bcd use minimum number of bytes For Calendar/text it is governed by the calendarpattern For Calendar/binary/packed or bcd use minimum number of bytes For Calendar/binary/binarymilliseconds or binaryseconds use implicit lengths For Boolean/text use the length of the true/false rep. For Boolean/binary use implicit length (32) 2. A new property dfdl:patternOutputLengthKind 3. We could limit dfdl:lenghtKind 'pattern' to complex elements so that there is always a lengthKind for child simple elements but this would introduce 'unnecessary' complex items in the infoset when there is only one child. The WG decided that option 1 was preferable but it needs to be written in terms of a set of 'base' unparsing rules which are augmented depending on the dfdl:lengthKind. There was some discussion about whether the pattern should be reapplied to the data stream produced to ensure that it could be parsed by the same schema. Perhaps this should be part of validation. There was no agreement. MB suggested that dfdl:lengthKind 'pattern' should only be allowed for elements with representation 'text' and complex elements with children all 'text'. Steve H felt that it was needed for complex elements containing binary data. Alan will write up 'base' unparsing length rules. 3. TLOG Steve H took us through his further investigation of TLog format Finally had some time to look at the TLOG stuff again (the format emitted by IBM 4680 & 4690 POS controllers). This time I've looked at the MRM code, & spoken with domain expert David Bennett, to establish the full behaviour. Unfortunately there are some issues which mean DFDL 1.0 is not capable of handling it. 1. The individual fields are a mixture of ASCII strings, proprietary packed strings (rare), proprietary packed decimals, binary integers (rare). All fields are delimited by a separator. 2. The fields are all defined with a length in bytes, but most of the string and decimal fields are actually variable length. If the data exceeds the length in bytes when parsing or unparsing the MRM throws an error. However, talking to David B, the length is really intended to be used if there was ever a fixed length equivalent format, so is really for validation only. To parse the current TLOG formats it is sufficient to use the delimiter. Note to WG: Validation using the specified length only works for strings. Should we allow dfdl:length to be specified when dfdl:lengthKind=delimited or pattern, and use it as an extra constraint when parsing/unparsing? 3. Scanning for the separator, or maybe use of a data pattern, is needed to extract the variable length data, including packed decimal data (which we would consider a binary type). Note to WG: Should we allow binary scanning when the users says it is safe to do so? 4. Packed strings. This is a packed data type used when the range of possible chars in the string is limited to 0-9, A-F. You can view this as a BCD that can also carry A-F. MRM does not try to turn this into an integer, instead it treats it as character data. That is x12x34 would result in string '1234'. Odd numbers of digits are padded with a x0 nibble. On unparsing the MRM throws an error if a character other than 0-9, A-F is encountered. In practice, these packed strings are rare and invariably only carry 0-9 anyway. I'm still trying to establish why this data type is needed, as it could be treated as a logical integer and modelled as BCD. 5. Packed decimals. Like a packed decimal in the IBM sense. These can carry negative numbers but use a leading xD sign nibble. No sign nibble if positive. Odd number of digits (including sign if present) are padded with xF nibble. This is best illustrated using examples. 1234 => x12x34 123 => xF1x23 -1234 => xFDx12x34 -123 => xD1x23 Note to WG: Should we support this data type natively? 6. Some of the packed decimals are interpreted as an array of bit flags. I assume that DFDL would model these using dfdl:hidden. Steve H is going to make a proposal about how DFDL can parse TLog 4, Empty Sequences Is the current section correct 1.1 Empty Sequences A sequence having no children is syntactically legal in DFDL; however, a sequence having no children must have content length zero. It can still have non-zero length Prefix and Suffix regions, but the SequenceContent region in between must be of length zero. It is a schema definition error if the SequenceContent region of an empty sequence is not length zero. Steve H will re-word. 5. Mike/Steve review issues Mike B had to leave but a number a issues were discussed - Case of enumerations. We should follow the XSDL convention which is that enumerations are case sensitive - dfdl:lengthKind='Pattern: dfdl entities will be allowed in the lengthPattern regular expression except for generic entities such as WSP and NL - dfdl:lengthKind='Pattern scannability: A complex element with lengthKind=Pattern will use its dfdl:encoding property as the encoding when scanning its children irrespective of the child's encoding property. Mike to comment on remaining review issues. 6. Go through remaining actions Updated below Action 071 Semantics of length=0, nil handling and defaults. 1) Steve H to propose new name for dfdl:defaultValueInitiatorPolicy Looking more at this, we should be consistent with the similar property for missing separators. So I've changed the enums too - now 'require' and 'suppress', Also changed unparsing behaviour - we must honour the property - the existing behaviour of always writing the initiator means we can not successfully re-parse if writing empty content and enum is 'suppress'. When reading, assume that section 15.13 has been updated to include complex as well as simple elements. missingValueInitiatorPolicy Enum Valid values ?require', ?suppress' Specifies whether to expect an initiator when an element is missing. Ignored unless dfdl:initiator is specified and is not "" (empty string). 'require' - Indicates that the dfdl:initiator followed by empty content is the required syntax to indicate that the element is missing. 'suppress' - Indicates that empty content is the required syntax to indicate that the element is missing. The presence of an initiator implies that real content must follow. Use of ?suppress? implies an ordered sequence. If used on an initiated element of an unordered group it is a schema definition error. If the element is required, defaulting occurs as defined above. This property also applies on unparsing, when the data to be written (after nil value and default value processing) is empty content. Annotation: dfdl:element We should similarly change the enums for nilValueInitiatorPolicy to 'require' and 'suppress'. 2) Not recorded in minutes, but there was a discussion around my bullet on choices. Worth noting that the concept of 'required' for the elements of a choice does not apply. Even if minOccurs > 0. The issue was on unparsing. Which branch of a choice do we output when a complex element is required but missing from the infoset? I think it should be the first branch of the choice that does not result in a processing error. NOTE: Work Item 071 changes dfdl:separatorPolicy enumeration from require to always. Need to make sure that is consistent with this proposal. 7 Review Schedule OGF prereview is confirmed to take about 4 weeks assuming no document updates are required. We are behind schedule to be available for public review by March. Draft 038 will be available at the end of this week. Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Meeting closed, 14:30 Next call Tuesday 26 January 2010 13:00 UK and Wednesday 27th January 2010 13:00 UK. Next action: 076 Actions raised at this meeting No Action 074 SH: Proposal for parsing TLog 075 SH: rewrite empty sequences section Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 13/01: Stephaine took us through a description of WTX identifiers. Mike agreed to write up in DFDL terms. 20/01: Mike will write up 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progess 20/01: no progess 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 13/01: no update 20/01: no update 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 13/01: SH proposal review. Minor updates to be made 20/01: Reviewed updated proposal. Need to agree on unparsing empty choices. 074 SH: Proposal for parsing TLog 075 SH: rewrite empty sequences section Closed actions No Action 056 MB Resolve lengthUnits=bits including fillbytes 12/08: No Progress ... 28/10: no progress 04/11: MB to look at lengthUnits = bits 11/11: no update 18/11: no update 25/11: no update 04/12: no update. ALan will set up a separate call to progress this action. 09/12: no update. ALan will set up a separate call to progress this action. 16/12: MB, SH and AP had a separate call. MB to distribute proposal 23/12: Discussed proposal. MB will updated 06/01: V4 discussed and approved 13/01: Mike updated proposal. Closed 073 SH: Control of overpunching zoned positive sign 13/01: no update 20/01: Proposal agreed. Closed Work items: No Item target version status 005 Improvements on property descriptions not started 011 How speculative parsing works (combining choice and variable-occurence - currently these are separate) (from action 045) awaiting completion of actions 045 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 038 Improve length section including bit handling 038 some improvement in 036 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 069 ICU fractional seconds 038 070 Write DFDL primer 071 Write test cases. 072 it is a processing error if the number of occurrences in the data does not match the value of the expression or prefix 038 073 Rename dfdl:separatorPolicy="required" to "always". 038 074 - Last 'postFix' separator is not optional - Terminators are mandatory. - dfdl:documentFinalTerminatorCanBeMissing - dfdl:documentFinalSeparatorCanBeMissing (Action (70)) 038 075 Remove occursCountKind="useAvailableSpace". 038 076 dfdl:documentRoot, will be defined that can only be on global elements. The DFDL spec does not have to define the format of parameters to the DFDL processor but will indicate that it must be possible to adresss any element. Agreed that ANY element within the schema cane be the starting point for parsing or unparsing. dfdl:documentRoot no longer required 038 077 'delimited' means the item is delimited by the item?s terminator (if specified) or an enclosing construct?s separator or end of the enclosing construct designated by its known length or its terminator. The definition of EndOfParent also needs improving. 038 078 document UPA checks 038 079 Restrictions on use of 'special' entities in regular expressions 038 080 LengthUnit=bits (A056) 038 081 Case of enumerations. We should follow the XSDL convention which is that enumerations are case sensitive 038 082 dfdl:lengthKind='Pattern: dfdl entities will be allowed in the lengthPattern regular expression except for generic entities such as WSP and NL 038 083 dfdl:lengthKind='Pattern scannability: A complex element with lengthKind=Pattern will use its dfdl:encoding property as the encoding when scanning its children irrespective of the child's encoding property. 038 084 Control of overpunching zoned positive sign 038 Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG call 20 January 2010- 13:00 UK (8:00 ET)
by Alan Powell 20 Jan '10

20 Jan '10

1. 045 - Disciminators Mike was writing up proposal 2 Unparsing lengthKind = 'pattern' Review email from Alan 3. TLOG SteveH has been investigating 4, Empty Sequences Is the current section correct 1.1 Empty Sequences A sequence having no children is syntactically legal in DFDL; however, a sequence having no children must have content length zero. It can still have non-zero length Prefix and Suffix regions, but the SequenceContent region in between must be of length zero. It is a schema definition error if the SequenceContent region of an empty sequence is not length zero. 5. Mike review issues Attached document gives replies to comments/issues. Ones without replies need to be discussed 6. Go through remaining actions 7 Review Schedule Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 13/01: Stephaine took us through a description of WTX identifiers. Mike agreed to write up in DFDL terms. 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progess 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 13/01: no update 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 13/01: closed 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 13/01: SH proposal review. Minor updates to be made 073 SH: Control of overpunching zoned positive sign 13/01: no update Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Minutes for OGF DFDL Working Group Call, January-13-2010
by Alan Powell 15 Jan '10

15 Jan '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, January-13-2010 Attendees Mike Beckerle (Oco) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Stephanie Fetzer (IBM) Suman Kalia (IBM) Peter Lambros (IBM) Tim Kimber(IBM) Apologies 1. 045 - Disciminators Stephanie took us through her email Subject: [DFDL-WG] Bob & Steph's WTX 'Discriminators' write-up WTX Identifiers are similar to DFDL discriminators -Discriminators may only be placed on the physical representation of a group. That is why we see them on partition groups and sequence groups but not on choice groups (or unordered groups ? covered below). In partitioned groups we have a subtype of each possible group ? so each possible group may have a discriminator. When WTX expresses choice groups it expresses them as a group containing all of the possible child groups ? so at the top level ?choice group? there is no component of the actual group content- so no use for a discriminator. But each choice which may itself be a group may have a discriminator. Choice groups are special in that the choice model construct simply lists the components and only one may occur...at this level a discriminator on one of the choices may not be very useful. Inside of each choice?s components a discriminator could be used to indicate the existence of that choice. -The WTX UI does not allow discriminators on the components of Unordered Groups. This may be due to the fact that the position of the discriminator has significance (all rules at or above the discriminator must evaluate to true). If the group is unordered it would be difficult to enforce. Will need to discuss for DFDL. -A group may have either zero or one discriminators. No group may have more than one discriminator. -The discriminator may have two significant parts o it?s location (mandatory). The discriminator is placed on a component of a group and makes all of the cardinality and rules at that point and above become part of it's concept. o it?s rule (optional) A group with a component which has a discriminator should have some ?rule? associated with it. In WTX if there is no explicit rule then the implicit rule is ?PRESENT($)?. We will need to decide if such implied rules will be allowed in DFDL. -A group may only have a discriminator on a mandatory component. Once again, this impacts a choice group where by definition all components are optional ? which will not have a discriminator. This has been an issue of debate in WTX. We could have implemented checking on optional elements quite easily Over the years this has been questioned (as our UI allows them to be placed on optional elements) but once we explained the way the engine worked no customers perceived this as a deficiency. In DFDL we will need to determine if this is needed. -In WTX we do allow a discriminator to be placed on a mandatory fixed size array (a repeating mandatory component with n:n cardinality). It?s component rule can either refer to the entirety of the array (PRESENT($) meaning the whole of the array is present) or can call out a specific rule against one if the iterations. This is not done often in practice. -In WTX it is common to have multiple levels of discriminators when we are working with nested groups. We discussed whether DFDL should not allow discriminators on unordered groups or groups with floating elements. Agree that discriminators should be allowed Also discussed whether timing 'before/after' was required are WTX only has after. Decided to keep timing property. Suggested should not be allowed on variable length arrays to be consistent with not being allowed on optional elements. Mike agreed to write up rules in dfdl terms and extent to cover other points of uncertainty besides choices. 2. Zero length elements Steve H took us through his email subject: [DFDL] zero length (was Re: Fw: TDS length reference) ** updated ** This proposes that zero length fields should not be a processing error Proposal: 1. Parsing Simple elements 1) It is not a schema definition error nor a processing error if a length is being used to extract data and it is zero. This covers dfdl:lengthKind implicit, explicit, prefixed and endOfParent (when parent length is known). The result is 'empty content'. (Note that for implicit, XSDL allows maxLength/length facet to be 0, so disallowing it for others is not consistent). 2) It is not a processing error if scanning for data and the length of the returned bytes is zero. This applies to dfdl:lengthKind delimited, pattern and endOfParent (when parent length is not known). The result is 'empty content'. (This is just stating the obvious). (The above two rules ensure that it is possible to apply empty content to trigger optional, nil value or default value processing regardless of data type and dfdl:lengthKind). 3) Optional, nil and default processing are applied as per spec. 4) If the element is required, and nil value or default value is not used, and empty string is not in the lexical space of the element's type, then it is a processing error. The two initiator related properties dfdl:nilValueInitiatorPolicy and dfdl:defaultValueInitiatorPolicy define whether nils and defaults are applied when initiated empty content is found, they don't affect the definition of empty content or what it means for the type. [Note: If you recall, this discussion was triggered by a customer that was using an expression to calculate the length of a standard text decimal. He wanted 0 length to mean 0 ended up in the infoset. He can achieve this by making the element required with a default value of 0.] Complex elements It is possible to get returned empty content for a complex element for cases 1) and 2) above. 1) If the complex element is optional then it is not added to the infoset. 2) If the complex element does not have an initiator specified & is required then it is added to the infoset. 3) If the element has an initiator specified then dfdl:defaultValueInitiatorPolicy applies - required => element is added to infoset only if initiator is present (processing error if no initiator & empty content) - prohibited => element is added to infoset only if initiator is not present (initiator implies real content follows so processing error if initiator & empty content) 4) If the complex element is added to the infoset, then the parser processes the child content of the complex type. This may or may not cause a processing error. <tk>I presume a processing error would be caused by - any group having an initiator or terminator ( same as 5. below ) - any group having a prefix or postfix delimiter - any group with more than one member having an infix delimiter - any required element within the complex element having an initiator and dfdl:defaultValueInitiatorPolicy="required" - any required element within the complex element having a terminator - any required element which does not have a default value specified, and for which a zero-length representation is illegal - other error scenarios? </tk> <smh>Correct. Basically you are going through the element's content (model group plus children) and attempting to parse. When you extract the data you get back empty content. This may or not cause a processing error. This was agreed on the call as the correct behaviour. In summary, for empty content to be valid for the complex element then it must also be valid for at least one content model</smh> If it doesn't then default value processing applies for required child elements. If we don't do this then we will not create default values for all missing required simple elements, and that would be wrong. 5) If the contained sequence or choice has an initiator or terminator then it is a processing error. <tk> So it's OK to have a choice among the children of the complex element? If so, the specification should define the rules for picking a branch of the choice. The DFDL processer *could* always pick the first branch, but what if the first branch triggers a processing error and a different branch would not have done? </tk> <smh>I think it's the same as with real content. Parser will start against the first branch of the choice and see where it gets. Usual speculative parsing rules apply. If it has not discriminated successfully and a processing error occurs it will cause backtracking and the next branch will be tried. If it finds a valid content model for the empty content we are ok. If it doesn't it's a processing error.</smh> 2. Unparsing Simple elements Data in the infoset can result in empty content being added to the bit stream (ie, nothing), with an accompanying 0 value in any length prefix or length expression field, if appropiate to the dfdl:lengthKind. Complex elements The absence from the infoset of a required complex element will cause any specified initiator to be output, plus if there are required children then default values will be output for those children. If we don't do this then we will not create default values for nested missing required simple elements, and that would be wrong. This enables creation of a sparse infoset containing just the elements with explicit values, with the rest defaulting regardless of nesting. 3. Choices Worth noting that the concept of 'required' for the elements of a choice does not apply. Even if minOccurs > 0. 4. Outstanding Issues Is it ok to reuse dfdl:defaultValueInitiatorPolicy for complex elements? Should it be renamed? Should we add a separate property for complex elements? Steve H to propose new name for dfdl:defaultValueInitiatorPolicy 3. Difference between dfdl:lenghtKind= Delimited and endOfParent 'delimited' means the item is delimited by the item?s terminator (if specified) or an enclosing construct?s separator or the end of the enclosing construct designated by its known length or its terminator. the only difference with dfdl:lentghKind='endOfParent' is that the latter includes the 'end of the data stream' and applies to binary fields. We should either Add 'end of data stream' to delimited and remove 'endOfParent' Make 'endOfParent' be specifically for only 'end of data stream' Short discussion. Alan agreed to try to write up description of endOfParent for review 4. Go through remaining actions No enough time 5 Draft 037 review >From comments: a DFDL Subset of XML Schema (TBD: need means for an implementation to indicate it is using non-standard extensions?) Believe that this was to allow users to indicate they are using unsupported schema components. Agreed to defer fron DFDL v1 b. Question whether infoset MUST be in schema order. Request for 'bitstream order' Short discussion. Main reason for schema order is allow the infoset to be validated against a schema. Agree to leave as schema order c. Dealing with 'Grammar ambiguity' errors Not discussed 6 Review Schedule Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Meeting closed, 15:20 Next call 20 January 2010 13:00 UK Next action: 074 Actions raised at this meeting No Action Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 13/01: Stephaine took us through a description of WTX identifiers. Mike agreed to write up in DFDL terms. 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progess 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 13/01: no update 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 13/01: closed 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 13/01: SH proposal review. Minor updates to be made 073 SH: Control of overpunching zoned positive sign 13/01: no update Closed actions No Action 056 MB Resolve lengthUnits=bits including fillbytes 12/08: No Progress ... 28/10: no progress 04/11: MB to look at lengthUnits = bits 11/11: no update 18/11: no update 25/11: no update 04/12: no update. ALan will set up a separate call to progress this action. 09/12: no update. ALan will set up a separate call to progress this action. 16/12: MB, SH and AP had a separate call. MB to distribute proposal 23/12: Discussed proposal. MB will updated 06/01: V4 discussed and approved 13/01: Mike updated proposal. Closed 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 13/01: closed Work items: No Item target version status 005 Improvements on property descriptions not started 011 How speculative parsing works (combining choice and variable-occurence - currently these are separate) (from action 045) awaiting completion of actions 045 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 038 Improve length section including bit handling some improvement in 036 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 069 ICU fractional seconds 070 Write DFDL primer 071 Write test cases. 072 it is a processing error if the number of occurrences in the data does not match the value of the expression or prefix 073 Rename dfdl:separatorPolicy="required" to "always". 074 - Last 'postFix' separator is not optional - Terminators are mandatory. - dfdl:documentFinalTerminatorCanBeMissing - dfdl:documentFinalSeparatorCanBeMissing (Action (70)) 075 Remove occursCountKind="useAvailableSpace". 076 dfdl:documentRoot, will be defined that can only be on global elements. The DFDL spec does not have to define the format of parameters to the DFDL processor but will indicate that it must be possible to adresss any element. Agreed that ANY element within the schema cane be the starting point for parsing or unparsing. dfdl:documentRoot no longer required 077 'delimited' means the item is delimited by the item?s terminator (if specified) or an enclosing construct?s separator or end of the enclosing construct designated by its known length or its terminator. The definition of EndOfParent also needs improving. 078 document UPA checks 079 Restrictions on use of 'special' entities in regular expressions 080 LengthUnit=bits (A056) Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Unparsing dfdl:lenghtKind='pattern'
by Alan Powell 14 Jan '10

14 Jan '10

The Specification currently says that on unparsing dfdl:lengthKind='pattern' behaves like dfdl:lenghtKind='implict' but since we limited 'implict' to certain logical/representation combinations this is no longer correct. Proposals 1. When unparsing complex elements with dfdl:lengthKind='pattern', the length is the length of the children (same as 'implicit') When unparsing simple elements with dfdl:lengthKind='pattern' the length is the implicit length for those that have one and length of the data supplied in the infoset converted to the representation with no padding/filling for the rest. For string/text and hexbinary/binary that is reasonable For number/text/standard and zoned it is governed by the numberPattern For number/binary/ binary use 'implicit' lengths For number/binary/packed or bcd use minimum number of bytes For Calendar/text it is governed by the calendarpattern For Calendar/binary/packed or bcd use minimum number of bytes For Calendar/binary/binarymilliseconds or binaryseconds use implicit lengths For Boolean/text use the length of the true/false rep. For Boolean/binary use implicit length (32) 2. A new property dfdl:patternOutputLengthKind 3. We could limit dfdl:lenghtKind 'pattern' to complex elements so that there is always a lengthKind for child simple elements but this would introduce 'unnecessary' complex items in the infoset when there is only one child. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] dfdl-wg Digest, Vol 41, Issue 15
by Tim Kimber 14 Jan '10

14 Jan '10

A correction re: the meaning of lengthKind="delimited". Current text: 'delimited' means the item is delimited by the item’s terminator (if specified) or an enclosing construct’s separator or the end of the enclosing construct designated by its known length or its terminator. the only difference with dfdl:lentghKind='endOfParent' is that the latter includes the 'end of the data stream' and applies to binary fields. My proposed text: 'delimited' means the item is delimited by any of - the item’s terminator (if specified) - an enclosing construct’s separator or terminator - the end of an enclosing construct designated by its known length - the end of the data stream If applied to a field with representation='text', lengthKind='endOfParent' means the same as 'delimited' If applied to a field with representation='binary', lengthKind='endOfParent' means the item is terminated by either of - the end of an enclosing construct designated by its known length - the end of the data stream We must not exclude the 'end of data stream' case from delimited, otherwise some fairly basic infix-delimited scenarios will not work. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: dfdl-wg-request(a)ogf.org To: dfdl-wg(a)ogf.org Date: 14/01/2010 11:04 Subject: dfdl-wg Digest, Vol 41, Issue 15 Send dfdl-wg mailing list submissions to dfdl-wg(a)ogf.org To subscribe or unsubscribe via the World Wide Web, visit http://www.ogf.org/mailman/listinfo/dfdl-wg or, via email, send a message with subject or body 'help' to dfdl-wg-request(a)ogf.org You can reach the person managing the list at dfdl-wg-owner(a)ogf.org When replying, please edit your Subject line so it is more specific than "Re: Contents of dfdl-wg digest..." Today's Topics: 1. Minutes for OGF DFDL Working Group Call, January-13-2010 (Alan Powell) ----- Message from Alan Powell <alan_powell(a)uk.ibm.com> on Thu, 14 Jan 2010 11:03:42 +0000 ----- To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] Minutes for OGF DFDL Working Group Call, January-13-2010 Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, January-13-2010 Attendees Mike Beckerle (Oco) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Stephanie Fetzer (IBM) Suman Kalia (IBM) Peter Lambros (IBM) Tim Kimber(IBM) Apologies 1. 045 - Disciminators Stephanie took us through her email Subject: [DFDL-WG] Bob & Steph's WTX 'Discriminators' write-up WTX Identifiers are similar to DFDL discriminators -Discriminators may only be placed on the physical representation of a group. That is why we see them on partition groups and sequence groups but not on choice groups (or unordered groups – covered below). In partitioned groups we have a subtype of each possible group – so each possible group may have a discriminator. When WTX expresses choice groups it expresses them as a group containing all of the possible child groups – so at the top level ‘choice group’ there is no component of the actual group content- so no use for a discriminator. But each choice which may itself be a group may have a discriminator. Choice groups are special in that the choice model construct simply lists the components and only one may occur...at this level a discriminator on one of the choices may not be very useful. Inside of each choice’s components a discriminator could be used to indicate the existence of that choice. -The WTX UI does not allow discriminators on the components of Unordered Groups. This may be due to the fact that the position of the discriminator has significance (all rules at or above the discriminator must evaluate to true). If the group is unordered it would be difficult to enforce. Will need to discuss for DFDL. -A group may have either zero or one discriminators. No group may have more than one discriminator. -The discriminator may have two significant parts o it’s location (mandatory). The discriminator is placed on a component of a group and makes all of the cardinality and rules at that point and above become part of it's concept. o it’s rule (optional) A group with a component which has a discriminator should have some ‘rule’ associated with it. In WTX if there is no explicit rule then the implicit rule is ‘PRESENT($)’. We will need to decide if such implied rules will be allowed in DFDL. -A group may only have a discriminator on a mandatory component. Once again, this impacts a choice group where by definition all components are optional – which will not have a discriminator. This has been an issue of debate in WTX. We could have implemented checking on optional elements quite easily Over the years this has been questioned (as our UI allows them to be placed on optional elements) but once we explained the way the engine worked no customers perceived this as a deficiency. In DFDL we will need to determine if this is needed. -In WTX we do allow a discriminator to be placed on a mandatory fixed size array (a repeating mandatory component with n:n cardinality). It’s component rule can either refer to the entirety of the array (PRESENT($) meaning the whole of the array is present) or can call out a specific rule against one if the iterations. This is not done often in practice. -In WTX it is common to have multiple levels of discriminators when we are working with nested groups. We discussed whether DFDL should not allow discriminators on unordered groups or groups with floating elements. Agree that discriminators should be allowed Also discussed whether timing 'before/after' was required are WTX only has after. Decided to keep timing property. Suggested should not be allowed on variable length arrays to be consistent with not being allowed on optional elements. Mike agreed to write up rules in dfdl terms and extent to cover other points of uncertainty besides choices. 2. Zero length elements Steve H took us through his email subject: [DFDL] zero length (was Re: Fw: TDS length reference) ** updated ** This proposes that zero length fields should not be a processing error Proposal: 1. Parsing Simple elements 1) It is not a schema definition error nor a processing error if a length is being used to extract data and it is zero. This covers dfdl:lengthKind implicit, explicit, prefixed and endOfParent (when parent length is known). The result is 'empty content'. (Note that for implicit, XSDL allows maxLength/length facet to be 0, so disallowing it for others is not consistent). 2) It is not a processing error if scanning for data and the length of the returned bytes is zero. This applies to dfdl:lengthKind delimited, pattern and endOfParent (when parent length is not known). The result is 'empty content'. (This is just stating the obvious). (The above two rules ensure that it is possible to apply empty content to trigger optional, nil value or default value processing regardless of data type and dfdl:lengthKind). 3) Optional, nil and default processing are applied as per spec. 4) If the element is required, and nil value or default value is not used, and empty string is not in the lexical space of the element's type, then it is a processing error. The two initiator related properties dfdl:nilValueInitiatorPolicy and dfdl:defaultValueInitiatorPolicy define whether nils and defaults are applied when initiated empty content is found, they don't affect the definition of empty content or what it means for the type. [Note: If you recall, this discussion was triggered by a customer that was using an expression to calculate the length of a standard text decimal. He wanted 0 length to mean 0 ended up in the infoset. He can achieve this by making the element required with a default value of 0.] Complex elements It is possible to get returned empty content for a complex element for cases 1) and 2) above. 1) If the complex element is optional then it is not added to the infoset. 2) If the complex element does not have an initiator specified & is required then it is added to the infoset. 3) If the element has an initiator specified then dfdl:defaultValueInitiatorPolicy applies - required => element is added to infoset only if initiator is present (processing error if no initiator & empty content) - prohibited => element is added to infoset only if initiator is not present (initiator implies real content follows so processing error if initiator & empty content) 4) If the complex element is added to the infoset, then the parser processes the child content of the complex type. This may or may not cause a processing error. <tk>I presume a processing error would be caused by - any group having an initiator or terminator ( same as 5. below ) - any group having a prefix or postfix delimiter - any group with more than one member having an infix delimiter - any required element within the complex element having an initiator and dfdl:defaultValueInitiatorPolicy="required" - any required element within the complex element having a terminator - any required element which does not have a default value specified, and for which a zero-length representation is illegal - other error scenarios? </tk> <smh>Correct. Basically you are going through the element's content (model group plus children) and attempting to parse. When you extract the data you get back empty content. This may or not cause a processing error. This was agreed on the call as the correct behaviour. In summary, for empty content to be valid for the complex element then it must also be valid for at least one content model</smh> If it doesn't then default value processing applies for required child elements. If we don't do this then we will not create default values for all missing required simple elements, and that would be wrong. 5) If the contained sequence or choice has an initiator or terminator then it is a processing error. <tk> So it's OK to have a choice among the children of the complex element? If so, the specification should define the rules for picking a branch of the choice. The DFDL processer *could* always pick the first branch, but what if the first branch triggers a processing error and a different branch would not have done? </tk> <smh>I think it's the same as with real content. Parser will start against the first branch of the choice and see where it gets. Usual speculative parsing rules apply. If it has not discriminated successfully and a processing error occurs it will cause backtracking and the next branch will be tried. If it finds a valid content model for the empty content we are ok. If it doesn't it's a processing error.</smh> 2. Unparsing Simple elements Data in the infoset can result in empty content being added to the bit stream (ie, nothing), with an accompanying 0 value in any length prefix or length expression field, if appropiate to the dfdl:lengthKind. Complex elements The absence from the infoset of a required complex element will cause any specified initiator to be output, plus if there are required children then default values will be output for those children. If we don't do this then we will not create default values for nested missing required simple elements, and that would be wrong. This enables creation of a sparse infoset containing just the elements with explicit values, with the rest defaulting regardless of nesting. 3. Choices Worth noting that the concept of 'required' for the elements of a choice does not apply. Even if minOccurs > 0. 4. Outstanding Issues Is it ok to reuse dfdl:defaultValueInitiatorPolicy for complex elements? Should it be renamed? Should we add a separate property for complex elements? Steve H to propose new name for dfdl:defaultValueInitiatorPolicy 3. Difference between dfdl:lenghtKind= Delimited and endOfParent 'delimited' means the item is delimited by the item’s terminator (if specified) or an enclosing construct’s separator or the end of the enclosing construct designated by its known length or its terminator. the only difference with dfdl:lentghKind='endOfParent' is that the latter includes the 'end of the data stream' and applies to binary fields. We should either Add 'end of data stream' to delimited and remove 'endOfParent' Make 'endOfParent' be specifically for only 'end of data stream' Short discussion. Alan agreed to try to write up description of endOfParent for review 4. Go through remaining actions No enough time 5 Draft 037 review From comments: a DFDL Subset of XML Schema (TBD: need means for an implementation to indicate it is using non-standard extensions?) Believe that this was to allow users to indicate they are using unsupported schema components. Agreed to defer fron DFDL v1 b. Question whether infoset MUST be in schema order. Request for 'bitstream order' Short discussion. Main reason for schema order is allow the infoset to be validated against a schema. Agree to leave as schema order c. Dealing with 'Grammar ambiguity' errors Not discussed 6 Review Schedule Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items – 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec – 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Meeting closed, 15:20 Next call 20 January 2010 13:00 UK Next action: 074 Actions raised at this meeting No Action Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 13/01: Stephaine took us through a description of WTX identifiers. Mike agreed to write up in DFDL terms. 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progess 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 13/01: no update 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 13/01: closed 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 13/01: SH proposal review. Minor updates to be made 073 SH: Control of overpunching zoned positive sign 13/01: no update Closed actions No Action 056 MB Resolve lengthUnits=bits including fillbytes 12/08: No Progress ... 28/10: no progress 04/11: MB to look at lengthUnits = bits 11/11: no update 18/11: no update 25/11: no update 04/12: no update. ALan will set up a separate call to progress this action. 09/12: no update. ALan will set up a separate call to progress this action. 16/12: MB, SH and AP had a separate call. MB to distribute proposal 23/12: Discussed proposal. MB will updated 06/01: V4 discussed and approved 13/01: Mike updated proposal. Closed 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 13/01: closed Work items: No Item target version status 005 Improvements on property descriptions not started 011 How speculative parsing works (combining choice and variable-occurence - currently these are separate) (from action 045) awaiting completion of actions 045 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 038 Improve length section including bit handling some improvement in 036 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 069 ICU fractional seconds 070 Write DFDL primer 071 Write test cases. 072 it is a processing error if the number of occurrences in the data does not match the value of the expression or prefix 073 Rename dfdl:separatorPolicy="required" to "always". 074 - Last 'postFix' separator is not optional - Terminators are mandatory. - dfdl:documentFinalTerminatorCanBeMissing - dfdl:documentFinalSeparatorCanBeMissing (Action (70)) 075 Remove occursCountKind="useAvailableSpace". 076 dfdl:documentRoot, will be defined that can only be on global elements. The DFDL spec does not have to define the format of parameters to the DFDL processor but will indicate that it must be possible to adresss any element. Agreed that ANY element within the schema cane be the starting point for parsing or unparsing. dfdl:documentRoot no longer required 077 'delimited' means the item is delimited by the item’s terminator (if specified) or an enclosing construct’s separator or end of the enclosing construct designated by its known length or its terminator. The definition of EndOfParent also needs improving. 078 document UPA checks 079 Restrictions on use of 'special' entities in regular expressions 080 LengthUnit=bits (A056) Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: [DFDL] zero length (was Re: Fw: TDS length reference) ** updated **
by Steve Hanson 13 Jan '10

13 Jan '10

For discussion on today'call..... Regards Steve Hanson Programming Model Architect, WebSphere Message Brokers, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/01/2010 12:46 ----- ** Complex element design added. Please review ** ------------------------------------------------------------------------------------------------------ Proposal extends the earlier work done in this area & described by spec section 15.13 and 5.7. To paraphrase those sections: 5.7 The 'default' attribute is used to provide the logical value of a required element while parsing when the representation is empty (content length is zero). 15.13 When we get 'empty content' from an element, and the element is optional, then it is not present and is not added to the infoset. When we get empty content from an element, and the element is required, then we start to look at nil handling and default handling properties. - If the properties are such that the empty string is a nil value then the infoset value is the special value nil. - If the properties are such that there is a default value specified then the infoset value is the default value. - Otherwise if empty string is valid for the type (ie, is derived from xs:string) then the infoset value is a zero length string. So we know what empty content is and how it is applied to simple elements. We need to define when it is possible to get empty content and what it means to elements of complex type or of non-string simple type. Proposal: 1. Parsing Simple elements 1) It is not a schema definition error nor a processing error if a length is being used to extract data and it is zero. This covers dfdl:lengthKind implicit, explicit, prefixed and endOfParent (when parent length is known). The result is 'empty content'. (Note that for implicit, XSDL allows maxLength/length facet to be 0, so disallowing it for others is not consistent). 2) It is not a processing error if scanning for data and the length of the returned bytes is zero. This applies to dfdl:lengthKind delimited, pattern and endOfParent (when parent length is not known). The result is 'empty content'. (This is just stating the obvious). (The above two rules ensure that it is possible to apply empty content to trigger optional, nil value or default value processing regardless of data type and dfdl:lengthKind). 3) Optional, nil and default processing are applied as per spec. 4) If the element is required, and nil value or default value is not used, and empty string is not in the lexical space of the element's type, then it is a processing error. The two initiator related properties dfdl:nilValueInitiatorPolicy and dfdl:defaultValueInitiatorPolicy define whether nils and defaults are applied when initiated empty content is found, they don't affect the definition of empty content or what it means for the type. [Note: If you recall, this discussion was triggered by a customer that was using an expression to calculate the length of a standard text decimal. He wanted 0 length to mean 0 ended up in the infoset. He can achieve this by making the element required with a default value of 0.] Complex elements It is possible to get returned empty content for a complex element for cases 1) and 2) above. 1) If the complex element is optional then it is not added to the infoset. 2) If the complex element does not have an initiator specified & is required then it is added to the infoset. 3) If the element has an initiator specified then dfdl:defaultValueInitiatorPolicy applies - required => element is added to infoset only if initiator is present (processing error if no initiator & empty content) - prohibited => element is added to infoset only if initiator is not present (initiator implies real content follows so processing error if initiator & empty content) 4) If the complex element is added to the infoset, then the parser processes the child content of the complex type. This may or may not cause a processing error. If it doesn't then default value processing applies for required child elements. If we don't do this then we will not create default values for all missing required simple elements, and that would be wrong. 5) If the contained sequence or choice has an initiator or terminator then it is a processing error. 2. Unparsing Simple elements Data in the infoset can result in empty content being added to the bit stream (ie, nothing), with an accompanying 0 value in any length prefix or length expression field, if appropiate to the dfdl:lengthKind. Complex elements The absence from the infoset of a required complex element will cause any specified initiator to be output, plus if there are required children then default values will be output for those children. If we don't do this then we will not create default values for nested missing required simple elements, and that would be wrong. This enables creation of a sparse infoset containing just the elements with explicit values, with the rest defaulting regardless of nesting. 3. Choices Worth noting that the concept of 'required' for the elements of a choice does not apply. Even if minOccurs > 0. 4. Outstanding Issues Is it ok to reuse dfdl:defaultValueInitiatorPolicy for complex elements? Should it be renamed? Should we add a separate property for complex elements? Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Bob & Steph's WTX 'Discriminators' write-up
by Stephanie Fetzer 13 Jan '10

13 Jan '10

All: Here are the WTX identification/discrimination behaviours 'on-paper' written with the help of Bob Connolly our WTX core engine expert. I had attempted to create a single description of the concept but ended up needing to split the core behaviour from the modelling options in order to get some of the wrinkles discussed. The attempt here is to share information on the use of this concept within WTX with the (outside of IBM) DFDL WG in such a way that it does not go into the specifics of the WTX implementation. ___________________________________________________________________________________________________ WTX use of IDENTIFIERS for distinguishing data This is a brief description of the use of WTX identifiers to describe limitations that could be imposed on the implementation of DFDL discriminators. We will attempt to word this in terms that are not specific to WTX and are a bit more XML-centric than we would normally describe WTX processing. This may dilute some of the specificity of the descriptions – but the spirit of the concepts is the important thing here. WHAT IS A DISCRIMINATOR?: Discriminators allow us to evaluate a data component and determine if it 'is known to exist'. The distinction that we are making is that there are situations when parsing when we need to determine if some data is an 'invalid instance' of one data component as opposed to 'not that data component'. This concept has multiple advantages: it lets us be more specific when modeling our data than we could be without the concept, and it allows us to terminate one branch of speculative parsing quicker than we might otherwise be able to without the concept. HOW ARE DISCRIMINATORS USED IN WTX CORE: The core of WTX uses discriminators in the parsing of data in three main ways: 1-We are parsing a group which is identifiable in some other way – for example, it is a group partitioned by initiator. In this case we functionally have a choice between two or more groups. So the decision here is which path or partition/choice group (if any) will be found to exist. If we don’t find the initiator in the data then this partition/choice group is not ‘known to exist’. If we find the initiator in the data and... -If we find the initiator in the data and we do not have a discriminator on a component of the partition/choice group then this type is ‘know to exist’ -If we find the initiator in the data and if we have a discriminator set on a component of the partition group and if the rule on the discriminator and all rules on earlier components evaluate to true then this group is 'known to exist' Note: There is no requirement in WTX that the component on which the identifier attribute is set has to have a specified rule - all mandatory instances of components have an implied rule of PRESENT($) in WTX. In this document when I say that the rule evaluates to true there is really more to this concept - we could say that everything up to that point is 'valid' meaning that all rules and cardinality constraints and other facets enforced (restrictions, size, presentation checks) pass. In WTX the lines between validation and parsing are blurred so these distinctions may be a bit different in a DFDL implementation. But we will at minimum need to work this in such a way as we can specify that not only the rules evaluate to true but that the cardinality checks also pass. In DFDL we may want to make having a rule a requirement if we choose to use the currently documented discriminator construct. -If we find the initiator in the data and if we have a discriminator set on a component of the partition group and if the rule on the discriminator evaluates to false or if any rule on a component prior to the component carrying the discriminator evaluates to false then this group is ‘known to not exist’ 2-We are evaluating a group and have determined that something is wrong with it and the group we are evaluating has a discriminator and we have not found (or evaluated its rule) yet - so we say that it is ‘known to not exist’. 3-A rule on a component in the group has evaluated to false and that rule is on or above the component with the discriminator. This group is ‘known to not exist’. Note: this is similar to number 2 but number 2 is another reason for failure such as missing a mandatory component. 4-A rule on a component in the group has evaluated to true and that rule is on the component with the discriminator. This group is ‘known to exist’. Note: as we are processing the component with the discriminator the assumption is that we would not be processing this rule if all previous occurring checks and rules checks had not evaluated to true. 5-A rule on a component in the group has evaluated to false and that rule is after the component with the discriminator. The rules on the component with the discriminator and above rules evaluated to true. This group is ‘known to exist’ but is invalid. 6-All rules on all components in the group have evaluated to true including the rule on the component with the discriminator. This group is ‘known to exist’ and is valid. 7-All rules on all components in the group have evaluated to true – there is no component with a discriminator. This group is ‘known to exist’ and is valid. Without the identifier we process the group to the end (last component) before determining that it is ‘known to exist’. HOW ARE DISCRIMINATORS IMPLEMENTED IN WTX MODELS?: WTX imposes many limitations to the expression of identifiers on the model. -Discriminators may only be placed on the physical representation of a group. That is why we see them on partition groups and sequence groups but not on choice groups (or unordered groups – covered below). In partitioned groups we have a subtype of each possible group – so each possible group may have a discriminator. When WTX expresses choice groups it expresses them as a group containing all of the possible child groups – so at the top level ‘choice group’ there is no component of the actual group content- so no use for a discriminator. But each choice which may itself be a group may have a discriminator. Choice groups are special in that the choice model construct simply lists the components and only one may occur...at this level a discriminator on one of the choices may not be very useful. Inside of each choice’s components a discriminator could be used to indicate the existence of that choice. -The WTX UI does not allow discriminators on the components of Unordered Groups. This may be due to the fact that the position of the discriminator has significance (all rules at or above the discriminator must evaluate to true). If the group is unordered it would be difficult to enforce. Will need to discuss for DFDL. -A group may have either zero or one discriminators. No group may have more than one discriminator. -The discriminator may have two significant parts o it’s location (mandatory). The discriminator is placed on a component of a group and makes all of the cardinality and rules at that point and above become part of it's concept. o it’s rule (optional) A group with a component which has a discriminator should have some ‘rule’ associated with it. In WTX if there is no explicit rule then the implicit rule is ‘PRESENT($)’. We will need to decide if such implied rules will be allowed in DFDL. -A group may only have a discriminator on a mandatory component. Once again, this impacts a choice group where by definition all components are optional – which will not have a discriminator. This has been an issue of debate in WTX. We could have implemented checking on optional elements quite easily Over the years this has been questioned (as our UI allows them to be placed on optional elements) but once we explained the way the engine worked no customers perceived this as a deficiency. In DFDL we will need to determine if this is needed. -In WTX we do allow a discriminator to be placed on a mandatory fixed size array (a repeating mandatory component with n:n cardinality). It’s component rule can either refer to the entirety of the array (PRESENT($) meaning the whole of the array is present) or can call out a specific rule against one if the iterations. This is not done often in practice. -In WTX it is common to have multiple levels of discriminators when we are working with nested groups. Stephanie Fetzer WebSphere Common Transformation Industry Packs - Software Engineer

1 0

Comments on DFDL Spec V0.37 - Remove dfdl:format annotation from complex type
by Suman Kalia 13 Jan '10

13 Jan '10

Chapter 1 : When the scoping rules were simplified in V 0.37 of spec, we removed dfd:format annotation from complex types but most of our examples in chapter 1 haven't been updated to to reflect this.. The examples should be updated to show this annotation at schema level.. <xs:complexType name="example1"> <xs:annotation> <xs:appinfo> <dfdl:format representation="binary" byteOrder="bigEndian" lengthKind="implicit" binaryFloatRepresentation="ieee" /> </xs:appinfo> </xs:annotation> <xs:sequence> <xs:element name="w" type="int"/> <xs:element name="x" type="int "/> <xs:element name="y" type="double"/> <xs:element name="z" type="float" /> </xs:sequence> </xs:complexType> Section 7.1 ( Page 47) -- Table should be updated to remove dfdl:format annotation from complex type.. Schema object DFDL annotation xs:choice dfdl:choice xs:complexType dfdl:format Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com

1 0

Agenda for OGF DFDL WG call 13 January 2010- 13:00 UK (8:00 ET)
by Alan Powell 12 Jan '10

12 Jan '10

1. 045 - Disciminators Discussion with Stephanie 2. Zero length elements Review Steve H email 3. Difference between dfdl:lenghtKind= Delimited and endOfParent 'delimited' means the item is delimited by the item?s terminator (if specified) or an enclosing construct?s separator or the end of the enclosing construct designated by its known length or its terminator. the only difference with dfdl:lentghKind='endOfParent' is that the latter includes the 'end of the data stream' and applies to binary fields. We should either Add 'end of data stream' to delimited and remove 'endOfParent' Make 'endOfParent' be specifically for only 'end of data stream' 4. Go through remaining actions 5 Draft 037 review >From comments: a DFDL Subset of XML Schema (TBD: need means for an implementation to indicate it is using non-standard extensions?) b. Question whether infoset MUST be in schema order. Request for 'bitstream order' c. Dealing with 'Grammar ambiguity' errors 6 Review Schedule Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 056 MB Resolve lengthUnits=bits including fillbytes 12/08: No Progress ... 28/10: no progress 04/11: MB to look at lengthUnits = bits 11/11: no update 18/11: no update 25/11: no update 04/12: no update. ALan will set up a separate call to progress this action. 09/12: no update. ALan will set up a separate call to progress this action. 16/12: MB, SH and AP had a separate call. MB to distribute proposal 23/12: Discussed proposal. MB will updated 06/01: V4 discussed and approved 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 073 SH: Control of overpunching zoned positive sign Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Floating elements and unordered groups
by Tim Kimber 12 Jan '10

12 Jan '10

Hi all, I know this area of the specification was only recently resolved, and I think there may be an inconsistency in the v0.37 wording. Section 16, re: sequenceKind says: "The children of an unordered sequence must be xs:element." Section 16.5 Floating Elements says: "An ordered sequence of n element children with either n or n-1 of those children with dfdl:floating="true" is equivalent to an unordered sequence with the same n element children with dfdl:floating="false". A complex element with dfdl:floating="true" can have as its content model a sequence with elements that also have dfdl:floating="true". " Now suppose that, instead of N element children, there are N-1 floating element children + one non-floating group. This group will be equivalent to an unordered group with a non-element member. If the specification was intending to make life easy for implementers, then it should probably disallow groups in any non-ordered context, including when sequenceKind='ordered' and there is at least one floating component. But I think that would be too restrictive. I would be happy for the restriction to be lifted entirely. Given that unordered groups can have dfdl:initiated="false", it will sometimes be necessary to find the correct member by trial and error ( speculative parsing ) anyway. I don't think it's any more difficult to speculatively parse a group than to speculatively parse a complex element. If I've missed something, and it turns out that the restriction is useful, then we should a) tighten up the wording to say that if a group with N members has N or N-1 floating members, then it must be validated as if it was an unordered group. b) consider lifting the restriction in cases where dfdl:initiated="true" ( because it makes things so much easier for the DFDL processor ) regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

Re: [DFDL-WG] dfdl-wg Digest, Vol 41, Issue 8
by Tim Kimber 08 Jan '10

08 Jan '10

My understanding was different from what is quoted below. 'delimited' can always be terminated by the end of the data or the end of an enclosing known-length element. Without that rule, the simplest scenario involving an infix delimiter will not work correctly ( or will force the user to set lengthKind='endOfParent' on the final group member, which would be very unintuitive ). There is a difference between endOfParent and delimited though. Delimited is not allowed when representation='binary'. 'endOfParent' is. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: dfdl-wg-request(a)ogf.org To: dfdl-wg(a)ogf.org Date: 08/01/2010 18:02 Subject: dfdl-wg Digest, Vol 41, Issue 8 Send dfdl-wg mailing list submissions to dfdl-wg(a)ogf.org To subscribe or unsubscribe via the World Wide Web, visit http://www.ogf.org/mailman/listinfo/dfdl-wg or, via email, send a message with subject or body 'help' to dfdl-wg-request(a)ogf.org You can reach the person managing the list at dfdl-wg-owner(a)ogf.org When replying, please edit your Subject line so it is more specific than "Re: Contents of dfdl-wg digest..." Today's Topics: 1. dfdl:lentgKind= 'delimited' and 'endOfParent' (Alan Powell) ----- Message from Alan Powell <alan_powell(a)uk.ibm.com> on Fri, 8 Jan 2010 17:10:51 +0000 ----- To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] dfdl:lentgKind= 'delimited' and 'endOfParent' Since we extended the meaning of dfdl:lengthKind= 'delimited' to include 'delimited' means the item is delimited by the item’s terminator (if specified) or an enclosing construct’s separator or the end of the enclosing construct designated by its known length or its terminator. the only difference with dfdl:lentghKind='endOfParent' is that the latter includes the 'end of the data stream'. We should either Add 'end of data stream' to delimited and remove 'endOfParent' Make 'endOfParent' be specifically for only 'end of data stream' Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

dfdl:lentgKind= 'delimited' and 'endOfParent'
by Alan Powell 08 Jan '10

08 Jan '10

Since we extended the meaning of dfdl:lengthKind= 'delimited' to include 'delimited' means the item is delimited by the item?s terminator (if specified) or an enclosing construct?s separator or the end of the enclosing construct designated by its known length or its terminator. the only difference with dfdl:lentghKind='endOfParent' is that the latter includes the 'end of the data stream'. We should either Add 'end of data stream' to delimited and remove 'endOfParent' Make 'endOfParent' be specifically for only 'end of data stream' Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Minutes for OGF DFDL Working Group Call, January-06-2010
by Alan Powell 07 Jan '10

07 Jan '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, January-06-2010 Attendees Mike Beckerle (Oco) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Stephanie Fetzer (IBM) Suman Kalia (IBM) Apologies Peter Lambros (IBM) 1. 045 - Disciminators Bob Connolly was not available to discuss WTX. Stephanie discussed some of the WTX behaviour and agreed to provide more for the next call. We need a definition of the restricted positions where discriminators can be used and the semantics 2. LengthUnits = 'bits' Reviewed v4 of Mike's proposal and suggested editorial changes. If possible would like some words to describe how 'little-endian' is dealt with. Approved. 3. Selector support in V1 Implementation of selectors is proving more difficult than expected so IBM requested that selectors were deferred. In order to give reviewers and indiction of how the function might look in a future release and the opportunity to say if it is a higher priority the description will be move to an appendix named 'Features under consideration for a future version' 4. Go through remaining actions updated below 5 Draft 037 review Mike has reviewed and raised many comments -Significant issues: The spec is not consistent about how different types of errors are dealt with. Schema definition errors can occur at design and runtime and stop processing 'Grammar ambiguity' errors may not be statically detectable so are processing errors. - Use of entities in regular expressions. 'Special' entities such as WSP+ will not be allowed. 5A. AOB Steve H raised an issue with zoned number support. Customers have requested the ability control whether the sign is overpunched for positive numbers. Action raised 6 Review Schedule Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Meeting closed, 15:00 Next call 13 January 2010 13:00 UK Next action: 074 Actions raised at this meeting No Action 073 SH: Control of overpunching zoned positive sign Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deffered 056 MB Resolve lengthUnits=bits including fillbytes 12/08: No Progress ... 28/10: no progress 04/11: MB to look at lengthUnits = bits 11/11: no update 18/11: no update 25/11: no update 04/12: no update. ALan will set up a separate call to progress this action. 09/12: no update. ALan will set up a separate call to progress this action. 16/12: MB, SH and AP had a separate call. MB to distribute proposal 23/12: Discussed proposal. MB will updated 06/01: V4 discussed and approved 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 073 SH: Control of overpunching zoned positive sign Closed actions No Action 072 Byte Order Mark and Unicode signature 23/12: Confirm BOM will be treated and part of the content. 06/01: MB v37 comments suggest additional description. closed Work items: No Item target version status 005 Improvements on property descriptions not started 011 How speculative parsing works (combining choice and variable-occurence - currently these are separate) (from action 045) awaiting completion of actions 045 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 038 Improve length section including bit handling some improvement in 036 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 069 ICU fractional seconds 070 Write DFDL primer 071 Write test cases. 072 it is a processing error if the number of occurrences in the data does not match the value of the expression or prefix 073 Rename dfdl:separatorPolicy="required" to "always". 074 - Last 'postFix' separator is not optional - Terminators are mandatory. - dfdl:documentFinalTerminatorCanBeMissing - dfdl:documentFinalSeparatorCanBeMissing (Action (70)) 075 Remove occursCountKind="useAvailableSpace". 076 dfdl:documentRoot, will be defined that can only be on global elements. The DFDL spec does not have to define the format of parameters to the DFDL processor but will indicate that it must be possible to adresss any element. Agreed that ANY element within the schema cane be the starting point for parsing or unparsing. 077 'delimited' means the item is delimited by the item?s terminator (if specified) or an enclosing construct?s separator or end of the enclosing construct designated by its known length or its terminator. The definition of EndOfParent also needs improving. 078 document UPA checks 079 Restrictions on use of 'special' entities in regular expressions Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Trackers posted from 037 review
by Mike Beckerle 06 Jan '10

06 Jan '10

I've posted several new tracker artifacts from my review of draft 037. Major issues have separate artifacts, and there is a big consolidated list of 61 smaller (In my opinion that is) items. http://forge.gridforum.org/sf/go/projects.dfdl-wg/tracker.specification_ver…

1 0

zoned having leading overpunched sign - evidence for existence of
by Mike Beckerle 06 Jan '10

06 Jan '10

This google search: http://www.google.com/search?rlz=1C1CHMA_enUS361US361&sourceid=chrome&ie=UT… has a number of pages that hit, Only http://www.uni.edu/coboldoc/cobrm_042.htm suggests there really is a leading overpunched sign. It suggests that CDO (an Oracle Cobol technology) supports SIGNED NUMERIC LEFT OVERPUNCHED l s, equivalent to Compaq Cobol S9(m)V9(n) LEADING, both of which mean zoned with leading overpunched sign, from what I can infer. I've never seen such data, but this suggests there are Cobol systems, or were, which can create this format. Whether anyone using such compilers has ever exercised this format is also in question of course.

2 1

DFDL bits - version 5
by Mike Beckerle 06 Jan '10

06 Jan '10

I made the title corrections below, and cleaned up the "pseudo code" a little, changing fieldWidth to bitStringLength. This is the best I can do for the v1.0 draft. On Wed, Jan 6, 2010 at 5:47 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > > Mike > > Minor comments before today's call: > > I'm happy for 1.0 to restrict the use of lengthUnits='bits' to a specific > set of logical types as per your doc. > > Section *Bits within Bit Strings of Length < 8* should really be *Bits > within Bit Strings of Length <= 8* > > Section *Numeric Value of a Bit String* is better titled *Bits within Bit > Strings of Length > 8 *for symmetry > > Regards > > Steve Hanson > Programming Model Architect, WebSphere Message Brokers, > OGF DFDL WG Co-Chair, > Hursley, UK, > Internet: smh(a)uk.ibm.com, > Phone (+44)/(0) 1962-815848 > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org Date: 06/01/2010 > 04:53 Subject: [DFDL-WG] DFDL bits - version 4 Sent by: > dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > I've fixed the issues identified by Suman and Alan. > > RE: > > 1. On the last WG call Steve H suggested allowing lengthUnit='bits' for any > logical type as long as the length was a multiple of 8. Did you decide > against that? > > > I did not decide on this either way. Which ever requires the least writing > in the spec is fine with me. > > Consider: dfdl:lengthUnits="bits" for dfdl:representation="text". This is > exactly equivalent to dfdl:lengthUnits="bytes" with the length, however it > is determined (prefix, or explicit) to be divided by 8. Now, what if the > length comes back from the expression or prefix as not a multiple of 8. Then > we have to detect a processing error..... length in bits not a multiple of > 8, and we have to explain why this is an error, etc. > > Unless there is a use case that requires this, I'd rather not put it in > just for uniformity's sake. I'd rather say that lengthKind="bits" is > available only for a specific narrow set of types. > > [attachment "DFDL Bits-bigEndian-bias.doc" deleted by Steve Hanson/UK/IBM] > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/dfdl-wg > > > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > >

1 0

DFDL bits - version 4
by Mike Beckerle 06 Jan '10

06 Jan '10

I've fixed the issues identified by Suman and Alan. RE: > > 1. On the last WG call Steve H suggested allowing lengthUnit='bits' for any > logical type as long as the length was a multiple of 8. Did you decide > against that? > > > I did not decide on this either way. Which ever requires the least writing in the spec is fine with me. Consider: dfdl:lengthUnits="bits" for dfdl:representation="text". This is exactly equivalent to dfdl:lengthUnits="bytes" with the length, however it is determined (prefix, or explicit) to be divided by 8. Now, what if the length comes back from the expression or prefix as not a multiple of 8. Then we have to detect a processing error..... length in bits not a multiple of 8, and we have to explain why this is an error, etc. Unless there is a use case that requires this, I'd rather not put it in just for uniformity's sake. I'd rather say that lengthKind="bits" is available only for a specific narrow set of types.

1 0

Information Update - Description has changed: OGF DFDL Working Group weekly call
by Steve Hanson 05 Jan '10

05 Jan '10

IBM Hursley attendees - DE3S09 Added Hursley location info

1 0

DFDL bits - version 3
by Mike Beckerle 05 Jan '10

05 Jan '10

Attached is revised per today's WG call.

3 2

Agenda for OGF DFDL WG call 06 January 2010- 13:00 UK (8:00 ET)
by Alan Powell 05 Jan '10

05 Jan '10

A Happy New Year to all WG participants. 1. 045 - Disciminators Continue discussion with Stephanie and Bob Connolly 2. LengthUnits = 'bits' Review Mike's updated proposal 3. Selector support in V1 4. Go through remaining actions 5 Draft 037 review No more comments received 6 Review Schedule Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 056 MB Resolve lengthUnits=bits including fillbytes 12/08: No Progress ... 28/10: no progress 04/11: MB to look at lengthUnits = bits 11/11: no update 18/11: no update 25/11: no update 04/12: no update. ALan will set up a separate call to progress this action. 09/12: no update. ALan will set up a separate call to progress this action. 16/12: MB, SH and AP had a separate call. MB to distribute proposal 23/12: Discussed proposal. MB will updated 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 072 Byte Order Mark and Unicode signature 23/12: Confirm BOM will be treated and part of the content. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0