Finally had some time to look at the TLOG stuff again (the format emitted by IBM 4680 & 4690 POS controllers). This time I've looked at the MRM code, & spoken with domain expert David Bennett, to establish the full behaviour. Unfortunately there are some issues which mean DFDL 1.0 is not capable of handling it. 1. The individual fields are a mixture of ASCII strings, proprietary packed strings (rare), proprietary packed decimals, binary integers (rare). All fields are delimited by a separator. 2. The fields are all defined with a length in bytes, but most of the string and decimal fields are actually variable length. If the data exceeds the length in bytes when parsing or unparsing the MRM throws an error. However, talking to David B, the length is really intended to be used if there was ever a fixed length equivalent format, so is really for validation only. To parse the current TLOG formats it is sufficient to use the delimiter. Note to WG: Validation using the specified length only works for strings. Should we allow dfdl:length to be specified when dfdl:lengthKind=delimited or pattern, and use it as an extra constraint when parsing/unparsing? 3. Scanning for the separator, or maybe use of a data pattern, is needed to extract the variable length data, including packed decimal data (which we would consider a binary type). Note to WG: Should we allow binary scanning when the users says it is safe to do so? 4. Packed strings. This is a packed data type used when the range of possible chars in the string is limited to 0-9, A-F. You can view this as a BCD that can also carry A-F. MRM does not try to turn this into an integer, instead it treats it as character data. That is x12x34 would result in string '1234'. Odd numbers of digits are padded with a x0 nibble. On unparsing the MRM throws an error if a character other than 0-9, A-F is encountered. In practice, these packed strings are rare and invariably only carry 0-9 anyway. I'm still trying to establish why this data type is needed, as it could be treated as a logical integer and modelled as BCD. 5. Packed decimals. Like a packed decimal in the IBM sense. These can carry negative numbers but use a leading xD sign nibble. No sign nibble if positive. Odd number of digits (including sign if present) are padded with xF nibble. This is best illustrated using examples. 1234 => x12x34 123 => xF1x23 -1234 => xFDx12x34 -123 => xD1x23 Note to WG: Should we support this data type natively? 6. Some of the packed decimals are interpreted as an array of bit flags. I assume that DFDL would model these using dfdl:hidden. Regards Steve Hanson Programming Model Architect, WebSphere Message Brokers, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh@uk.ibm.com, Phone (+44)/(0) 1962-815848 From: Alan Powell/UK/IBM@IBMGB To: dfdl-wg@ogf.org Date: 19/01/2010 18:40 Subject: [DFDL-WG] Agenda for OGF DFDL WG call 20 January 2010- 13:00 UK (8:00 ET) Sent by: dfdl-wg-bounces@ogf.org 1. 045 - Disciminators Mike was writing up proposal 2 Unparsing lengthKind = 'pattern' Review email from Alan 3. TLOG SteveH has been investigating 4, Empty Sequences Is the current section correct 1.1 Empty Sequences A sequence having no children is syntactically legal in DFDL; however, a sequence having no children must have content length zero. It can still have non-zero length Prefix and Suffix regions, but the SequenceContent region in between must be of length zero. It is a schema definition error if the SequenceContent region of an empty sequence is not length zero. 5. Mike review issues Attached document gives replies to comments/issues. Ones without replies need to be discussed 6. Go through remaining actions 7 Review Schedule Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items – 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec – 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 13/01: Stephaine took us through a description of WTX identifiers. Mike agreed to write up in DFDL terms. 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progess 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 13/01: no update 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 13/01: closed 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 13/01: SH proposal review. Minor updates to be made 073 SH: Control of overpunching zoned positive sign 13/01: no update Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell@uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU