Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, March 03-2010 Attendees Suman Kalia (IBM) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Peter Lambros (IBM) Stephanie Fetzer (IBM) Tim Kimber(IBM) Apologies Mike Beckerle (Oco) 1. 16.2 scannablility with lengthKind pattern: In summary, you can use a data pattern on any element (complex, simple text, simple binary) as long as the bytes are legal in the stated encoding, which where binary data is involved in practice means an 8-bit ASCII encoding. Binary data can be handled using some of the conveniences of text by way of treating it as text with encoding="iso-8859-1". In this case literal text, such as length patterns, is interpreted as in the iso-8859-1 character encoding, and the correspondence of byte values in the data to a string in the DFDL infoset is one to one. That is, byte with value N, produces an infoset character with character code N. The WG agreed to the solution of recommending the use of "iso-8859-1" for binary fields. The write-up should note that he main use case is for a complex element that contains binary fields. 2. Current Actions: Updated below 3 Steve H issues with draft 039 1) Name of property dfdl:textNumberRepresentation is not consistent with dfdl:binaryNumberRep, dfdl:binaryFloatRep, etc. Agreed 2) The dfdl:numberPattern etc properties that have been moved from the defunct dfdl:textNumberFormat annotation to dfdl:element etc should be called dfdl:textNumberPattern etc. Otherwise users will think they apply to binary numbers too. Agreed 3) In section 14.3 on sequences, there are several sub-sections that talk about parsing according to different ways of specifying length (ie, lengthKind). But dfdl:sequence no longer carries dfdl:lengthKind so I think these sub-sections are not in the right place. I think they should be in section 12, under the correct 12.3.x lengthKind sub-section. MB has moved these sections 4) Section 19 on built-in specifications. Given that we don't have any for public comment phase we should reword this section. Agreed 4 Tim's (major) issues with draft 039 12.2 Delimiters: Text Markup - The term 'Delimiters' is not accurate. Most readers will not think of an initiator as a 'delimiter'. - It's not 'Text' markup any more - especially since v0.39 has allowed lengthKind="delimited" for elements with binary representation. Title should be 'Markup' and explanation can then deal with what it really is, rather than justifying the innaccurate title :-) Will be changed to Delimiters - Add delimiters to glossary (DFDLlimits delimiters to terminators and separator when dfdl:lengthKind=delimited) Syntax for specifying markup: It's not clear from this description that each item in the space-separated list is a DFDL string literal. Section 6.3.1 will be revised to separate string literals and dfdl expressions etc. If possible will add the list of property types (listof DFDL string literals, etc) and update property tables initiator ( and all other space-separated properties ) It is not clear whether the order of the space-separated properties matters. Must the parser test them in the order in which they are specified? ( Q: What if %ES; is the first in the list? ) The parser must look for the longest first. Unparsing will output the first in the list. The %ES entity should be not be allowed for initiator, separator and terminator. Should probably be limited to use in nilValues but will check whether it is useful in other properties.. occursStopValue should a value in the value space of the base type of the array. terminator: is it OK if the final terminator is missing within the scope of a known-length parent? Seems like a reasonable extension of the rule ( in all other scenarios, the end of a known-length parent acts like the end of the data stream for items with its scope ). documentFinalTerminatorCanBeMissing: Let's try to avoid creating another property for the postfix separator scenario. I think this property provides a way of modelling the data naturally. We can recommend use of infix-with-a-terminator rather than 'postfix' if the final terminator can be missing. outputNewLine Should we validate that the 'characterOrCharacters' are all newline characters from the set described by the %NL; mnemonic? Otherwise the DFDL serializer will output data which cannot be parsed by the DFDL parser. Agreed dfdl:lengthKind endOfParent 'endOfParent' has almost the same meaning as 'delimited' so should have the same semantics. · the item?s terminator (if specified) · an enclosing construct?s separator or terminator · the end of an enclosing construct designated by its known length · the end of the data stream The effect would be the the element could be ended by the nearest known length parent not just the immediate parent. Also the immediate parent could have lengthKind 'implicit' Not discussed but emails indicate Agreed choiceKind 'Fixed' When lengthKind='implicit' all alternative branches of the choice are padded to the fixed length of the largest one so that overall the entire choice construct is fixed length There must be a restriction that the length of at least one choice must be statically defined. Not discussed but emails indicate Agreed 5 DFDL v1 Specification completion. Draft 40 will be published early next week. Will be sent to OGF next week prior to OGF 28 Meeting closed, 14:10 Next call Wednesday 10 March January 2010 13:00 UK Next action: 084 Actions raised at this meeting No Action Current Actions: No Action 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 24/03: No progress 03/03: Discussions have been taking place on the subset of tests that will be provided. 080 AP:Clarify semantics of fn:poisition and fn:count 17/02: no progress 24/03: No progress 03/03: no progress. There are other functions which return a duration which need investgating Closed actions No Action 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progress 20/01: no progress 27/01: no progress 29/01: No progress. The predefined formats do not need to be available when the spec is published. Suman said that he had been mapping COBOL structures to DFDL and it didn't look as though the way text numbers are define is very usable. He will document for next call 03/02: No progress 10/02: No progress 17/03: No progress 24/03: No progress 03/03: The wording of this section will be changed to say that 'implementations may choose to provide predefined formats. The DFDL WG intends to supply a limited set' Closed 079 MB:Encoding for binary fields when lenghtkind is pattern 17/02: Discussed but no conclusion 24/03: Mike has found an encoding that matches the first 255 codepoints of iso 10646. Will document its use for binary fields. 03/03: Wording in minutes agreed. Closed 083 MB:To correct syntax diagram for FinalUnused and suggest wording for the Sequence section 03/03: Mike has supplied updates to Sytax, length and Sequence sections. Closed Work items: No Item target version status 005 Improvements on property descriptions not started 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 070 Write DFDL primer 071 Write test cases. 083 Implement RFC2116 084 MB:Encoding for binary fields when lenghtkind is pattern 085 MB:To correct syntax diagram for FinalUnused and suggest wording for the Sequence section 086 dfdl:textNumberRepresentation to dfdl:textNumberRep 087 dfdl:NumberPattern etc to dfdl:textNumberPattern (not for calendarformat but check individual properties) 088 Update built-in specifications section. 089 12.2 Will be changed to Delimiters - Add delimiters to glossary (DFDLlimits delimiters to terminators and separator when dfdl:lengthKind=delimited) 090 Section 6.3.1 will be revised to separate string literals and dfdl expressions etc. If possible will add the list of property types (listof DFDL string literals, etc) and update property tables 091 The parser must look for the longest first. Unparsing will output the first in the list. The %ES entity should be not be allowed for initiator, separator and terminator. Should probably be limited to use in nilValues but will check whether it is useful in other properties.. 092 occursStopValue should a value in the value space of the base type of the array. 093 outputNewLine MUST be one of newline characters from the set described by the %NL; mnemonic 094 dfdl:lengthKind endOfParent 'endOfParent' has almost the same meaning as 'delimited' so should have the same semantics. · the item?s terminator (if specified) · an enclosing construct?s separator or terminator · the end of an enclosing construct designated by its known length · the end of the data stream The effect would be the the element could be ended by the nearest known length parent not just the immediate parent. Also the immediate parent could have lengthKind 'implicit' 095 ChoiceKinf fixed:There must be a restriction that the length of at least one choice must be statically defined. 096 The WG agreed to the solution of recommending the use of "iso-8859-1" for binary fields. The write-up should note that he main use case is for a complex element that contains binary fields. Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell@uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU