- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] dfdl-wg Digest, Vol 43, Issue 2
by Tim Kimber 03 Mar '10

03 Mar '10

I need to attend the sprint planning meeting from 2pm GMT, so I may not be around when these items are discussed. Comments added below: 12.2 Delimiters: Text Markup - The term 'Delimiters' is not accurate. Most readers will not think of an initiator as a 'delimiter'. - It's not 'Text' markup any more - especially since v0.39 has allowed lengthKind="delimited" for elements with binary representation. Title should be 'Markup' and explanation can then deal with what it really is, rather than justifying the innaccurate title :-) I dislike the use of the term "markup" for something not written by people, and most data formats of the DFDL kind are written by computers, so nothing is getting "marked up" by anyone. Initiators certainly are delimiters in the situations where they are not tags. I.e., initiator="[" terminator="]". Only tags will not be thought of as delimiters. Even then I think it is a stretch to say that nobody will think of an introductory tag as a delimiter. These definitions found online: de·lim·it·er (d?-l?m'?-t?r) n. Computer Science A character or sequence of characters marking the beginning or end of a unit of data. Computing Dictionary delimiter character A character or string used to separate, or mark the start and end of, items of data in, e.g., a database, source code, or text file. See also: record. (2001-03-16) These definitions are consistent with our usage of the term. I suggest no change in our terminology here. <TK> Point taken re: the term 'delimiter'. I still have reservations about calling it 'Text Markup' in the title, though. I think the intro paragraph should explain the common usage ( intiators, separators, terminators for text formats ) and the exceptional usage ( handling delimited binary data and other non-text markup ) </TK> Syntax for specifying markup: It's not clear from this description that each item in the space-separated list is a DFDL string literal. These have always bugged me. Any better solution is welcome. XML/XSD does tend to make space separated the standard way to specify more than one thing. <TK> In a future revision of the spec we need a list of property value types which can then be used consistently in the tables which describe properties. - Enumeration - DFDL string literal - List of DFDL string literals - DFDL expression - DFDL regular expression - Boolean - Non-negative integer - any more? In some cases it will be necessary to place restrictions on the type of content allowed in the string literal ( disallow raw byte values / raw byte values must represent a character / etc ) </TK initiator ( and all other space-separated properties ) It is not clear whether the order of the space-separated properties matters. Must the parser test them in the order in which they are specified? ( Q: What if %ES; is the first in the list? ) I think the order should not matter, and it should test them longest first. <TK> Good idea. I have another related suggestion below. </TK> terminator: is it OK if the final terminator is missing within the scope of a known-length parent? Seems like a reasonable extension of the rule ( in all other scenarios, the end of a known-length parent acts like the end of the data stream for items with its scope ). I believe this should be true. "Final" is relative in my mind. <TK> Good - it's much easier to implement if end of known length parent is always equivalent to end of data stream, from the point of view of enclosed elements. But see next point... </TK> documentFinalTerminatorCanBeMissing: Let's try to avoid creating another property for the postfix separator scenario. I think this property provides a way of modelling the data naturally. We can recommend use of infix-with-a-terminator rather than 'postfix' if the final terminator can be missing. Copasetic. <TK> Had to look up 'copasetic'. I'm amazed that my Mum never came out with that one - she's a walking dictionary. This property has caused problems with naming and interpretation all along the line. Last time we discussed it, I don't think we considered this option ( we did talk about something like it ): - If %ES; is included in the list of values for separator or terminator then a) The parser ignores it while performing ordinary scanning ( otherwise it would always cause a zero-length string to be scanned ). b) The parser accepts 'end of data stream' as a match for the %ES; mnemonic. That makes this property ( and the equivalent one for separators ) redundant. c) Other usages of %ES; remain unchanged. </TK> outputNewLine Should we validate that the 'characterOrCharacters' are all newline characters from the set described by the %NL; mnemonic? Otherwise the DFDL serializer will output data which cannot be parsed by the DFDL parser. Nice catch. dfdl:lengthKind endOfParent 'endOfParent' has almost the same meaning as 'delimited' so should have the same semantics. · the item?s terminator (if specified) · an enclosing construct?s separator or terminator · the end of an enclosing construct designated by its known length · the end of the data stream The effect would be the the element could be ended by the nearest known length parent not just the immediate parent. Also the immediate parent could have lengthKind 'implicit' Agreed. choiceKind 'Fixed' When lengthKind='implicit' all alternative branches of the choice are padded to the fixed length of the largest one so that overall the entire choice construct is fixed length There must be a restriction that the length of at least one choice must be statically defined. Also good catch. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG call 3 March 2010- 13:00 UK (8:00 ET)
by Alan Powell 02 Mar '10

02 Mar '10

1. 16.2 scannablility with lengthKind pattern: 2. Current Actions: 3 Steve H issues with draft 039 4 Tim's (major) issues with draft 039 5 Status of specification (for OGF28) 1. 16.2 scannablility with lengthKind pattern: In summary, you can use a data pattern on any element (complex, simple text, simple binary) as long as the bytes are legal in the stated encoding, which where binary data is involved in practice means an 8-bit ASCII encoding. Binary data can be handled using some of the conveniences of text by way of treating it as text with encoding="iso-8859-1". In this case literal text, such as length patterns, is interpreted as in the iso-8859-1 character encoding, and the correspondence of byte values in the data to a string in the DFDL infoset is one to one. That is, byte with value N, produces an infoset character with character code N. 2. Current Actions: No Action 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progress 20/01: no progress 27/01: no progress 29/01: No progress. The predefined formats do not need to be available when the spec is published. Suman said that he had been mapping COBOL structures to DFDL and it didn't look as though the way text numbers are define is very usable. He will document for next call 03/02: No progress 10/02: No progress 17/03: No progress 24/03: No progress 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 24/03: No progress 079 MB:Encoding for binary fields when lenghtkind is pattern 17/02: Discussed but no conclusion 24/03: Mike has found an encoding that matches the first 255 codepoints of iso 10646. Will document its use for binary fields. 080 AP:Clarify semantics of fn:poisition and fn:count 17/02: no progress 24/03: No progress 083 MB:To correct syntax diagram for FinalUnused and suggest wording for the Sequence section 3 Steve H issues with draft 039 1) Name of property dfdl:textNumberRepresentation is not consistent with dfdl:binaryNumberRep, dfdl:binaryFloatRep, etc. 2) The dfdl:numberPattern etc properties that have been moved from the defunct dfdl:textNumberFormat annotation to dfdl:element etc should be called dfdl:textNumberPattern etc. Otherwise users will think they apply to binary numbers too. 3) In section 14.3 on sequences, there are several sub-sections that talk about parsing according to different ways of specifying length (ie, lengthKind). But dfdl:sequence no longer carries dfdl:lengthKind so I think these sub-sections are not in the right place. I think they should be in section 12, under the correct 12.3.x lengthKind sub-section. 4) Section 19 on built-in specifications. Given that we don't have any for public comment phase we should reword this section. 4 Tim's (major) issues with draft 039 12.2 Delimiters: Text Markup - The term 'Delimiters' is not accurate. Most readers will not think of an initiator as a 'delimiter'. - It's not 'Text' markup any more - especially since v0.39 has allowed lengthKind="delimited" for elements with binary representation. Title should be 'Markup' and explanation can then deal with what it really is, rather than justifying the innaccurate title :-) Syntax for specifying markup: It's not clear from this description that each item in the space-separated list is a DFDL string literal. initiator ( and all other space-separated properties ) It is not clear whether the order of the space-separated properties matters. Must the parser test them in the order in which they are specified? ( Q: What if %ES; is the first in the list? ) terminator: is it OK if the final terminator is missing within the scope of a known-length parent? Seems like a reasonable extension of the rule ( in all other scenarios, the end of a known-length parent acts like the end of the data stream for items with its scope ). documentFinalTerminatorCanBeMissing: Let's try to avoid creating another property for the postfix separator scenario. I think this property provides a way of modelling the data naturally. We can recommend use of infix-with-a-terminator rather than 'postfix' if the final terminator can be missing. outputNewLine Should we validate that the 'characterOrCharacters' are all newline characters from the set described by the %NL; mnemonic? Otherwise the DFDL serializer will output data which cannot be parsed by the DFDL parser. dfdl:lengthKind endOfParent 'endOfParent' has almost the same meaning as 'delimited' so should have the same semantics. · the item?s terminator (if specified) · an enclosing construct?s separator or terminator · the end of an enclosing construct designated by its known length · the end of the data stream The effect would be the the element could be ended by the nearest known length parent not just the immediate parent. Also the immediate parent could have lengthKind 'implicit' choiceKind 'Fixed' When lengthKind='implicit' all alternative branches of the choice are padded to the fixed length of the largest one so that overall the entire choice construct is fixed length There must be a restriction that the length of at least one choice must be statically defined. Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

DFDL V1 Draft 039 is available
by Alan Powell 02 Mar '10

02 Mar '10

All DFDL V1 draft 038 is available at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… There have again been many changes listed below including many editorial updates to make the specification more readable. The DFDL expression language and DFDL regular expression language sections have been moved towards the end as they are for more advanced users. All highlighted changes are for the changes made in this draft only and do not include changes made in previous drafts. Unfortunately I have had significant problems with MS Word crashes while creating this draft which have resulted the formatting going haywire in some places. I have corrected the major problems but many minor ones still exist. In particular all the cross references have lost the number of the section they reference. This will be fixed in the next draft. Please send comments to me by March 3rd Latest entry at the top please Version Author/ Contributor History Date(yyyy-mm-dd) 039 Alan Powell Added numberPattern section. Added defaulting of complex types Changed missingValueInitiatorPolicy to 'required' and 'prohibited' Changed separatorPolicy to 'required' 'supressed' 'supressedAtEnd' Added defaulting choices. Many editorial changes Added lengthKind delimited for 'binary' 'packed/bcd' Clarified empty sequqnce wording Reworded infoset round tripping. Moved DFDL Expression and regular expression section to near back os spec. Added treatment of unrepresentable characaters to infoset section. Clatified relative path expressions on global declarations. Representation assumed to be text for string and binary for hexBinary Change alignment to be 1 based. docmentFinalTerninatorCanBeMissing and docmentFinalSeparatorCanBeMissing change to dfdl:format only Added UPA checks Added Ambiguity checks Removed defineTextNumberFormat, TextNumberFormat and textNumberFormatRef and moved properties in line Revised 'Resolving points of uncertainty' and discriminators Clarified Nan and infinity values 2010-02-24 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

Minutes for OGF DFDL Working Group Call, February 24-2010
by Alan Powell 24 Feb '10

24 Feb '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, February 24-2010 Attendees Mike Beckerle (Oco) Suman Kalia (IBM) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Peter Lambros (IBM) Stephanie Fetzer (IBM) Apologies Tim Kimber(IBM) 1. Remaining 037 review issues A: 16.2 scannablility with lengthKind pattern: Confirm that this is what we agreed In summary, you can use a data pattern on any element (complex, simple text, simple binary) as long as the bytes are legal in the stated encoding, which where binary data is involved in practice means an 8-bit ASCII encoding. Mike B I found an official reference which has no "greyed out" codepoints. All 256 values are "mapped". The following ftp table (see URL below) officially defines the mapping for 8859-1 to unicode/iso10646. The table includes all 256 codepoints - some are specified as just <control> i.e., have no specific meaning, but their 8859 codepoint maps one-to-one and onto a unicode/10646 codepoint with the same value. Note that this property holds for 8859-1. It does not hold for 8859-2 to 8859-16, as these have character codes substituted into them that map to other places in the iso10646 codepoint space. Here's the correspondence table: ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-1.TXT If we reference this mapping table in the references of the DFDL spec, then I believe we can say that using encoding="iso-8859-1", you can treat binary data as textual, use patterns, etc., and the relationship to/from the infoset always insures preservation of the values of the bytes (parsing), and creation of bytes whose values exactly match the string codepoints (unparsing). This language can be added to the section on lengthKind="pattern" and binary data: Binary data can be handled using some of the conveniences of text by way of treating it as text with encoding="iso-8859-1". In this case literal text, such as length patterns, is interpreted as in the iso-8859-1 character encoding, and the correspondence of byte values in the data to a string in the DFDL infoset is one to one. That is, byte with value N, produces an infoset character with character code N. [reference to above FTP site]. B: Glossary Variable-Occurrence Item - Optional elements have a variable number of occurrences (0 or 1) and arrays also can have a variable number of occurrences (when minOccurs < maxOccurs). So when we say an item with a variable number of occurrences, this can mean either an optional element, or an array where minOccurs < maxOccurs. In either array or optional elements, we have the additional constraint that the DFDL representation properties do not preclude a variable number of occurrences. When dfdl:occursCountKind='explicit' and dfdl:occursCount has a literal constant as its value, or an expression that statically evaluates to a constant, then the DFDL properties are specifying exactly the number of occurrences for all instances and so are said to preclude a variable number of occurrences. If dfdl:occursCount has a formula as its expressed value, then the DFDL properties do not preclude a variable number of occurrences. MikeB Comment: This idea that you can have minOccurs < maxOccurs, but dfdl:occurs is equal to a constant and dfdl:occursKind="explicit" is causing us a bunch of grief in these definitions. Can we be conservative and just say it is a schema definition error if minOccurs < maxOccurs but the length is static, i.e., an explicit constant-valued expression? WG decided that the wording can remain as currently written C: DFDL Schema Component Model Only changes needed are: remove wildcards. Add following to describe shading: The shaded boxes have a direct corresponding element syntax and therefore appear in DFDL schema D: Sequence Groups Mike B: To correct syntax diagram for FinalUnused and suggest wording for the Sequence section E: Check other comments in document. Please look at the remaining comments in draft 039 and suggest solutions 2. Go through Actions Updated below 3 DFDL v1 Specification completion. Draft 039 will be publish today. WG review and Comments by 3 March Draft 40 with updates for OGF submission - available 5 March Meeting closed, 14:00 Next call Wednesday 3 March January 2010 13:00 UK Next action: 084 Actions raised at this meeting No Action 083 MB:To correct syntax diagram for FinalUnused and suggest wording for the Sequence section Current Actions: No Action 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progress 20/01: no progress 27/01: no progress 29/01: No progress. The predefined formats do not need to be available when the spec is published. Suman said that he had been mapping COBOL structures to DFDL and it didn't look as though the way text numbers are define is very usable. He will document for next call 03/02: No progress 10/02: No progress 17/03: No progress 24/03: No progress 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 24/03: No progress 079 MB:Encoding for binary fields when lenghtkind is pattern 17/02: Discussed but no conclusion 24/03: Mike has found an encoding that matches the first 255 codepoints of iso 10646. Will document its use for binary fields. 080 AP:Clarify semantics of fn:poisition and fn:count 17/02: no progress 24/03: No progress 081 AP: Inf and Nan The description is the way ICU behaves but need clarification. It isn't clear how inf and Nan are represented in the infoset. Need to investigate if XML allows these values 17/02: XML allows Nan and inf for float and double Dfdl will do the same. Requires more investigation of ICU. 24/03: Alan send clarification that Inf and Nan are limited to float and Double. Closed. 083 MB:To correct syntax diagram for FinalUnused and suggest wording for the Sequence section Closed actions No Action 081 AP: Inf and Nan The description is the way ICU behaves but need clarification. It isn't clear how inf and Nan are represented in the infoset. Need to investigate if XML allows these values 17/02: XML allows Nan and inf for float and double Dfdl will do the same. Requires more investigation of ICU. 24/03: Alan send clarification that Inf and Nan are limited to float and Double. Closed. Work items: No Item target version status 005 Improvements on property descriptions not started 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 069 ICU fractional seconds 039 070 Write DFDL primer 071 Write test cases. 072 it is a processing error if the number of occurrences in the data does not match the value of the expression or prefix 039 073 Rename dfdl:separatorPolicy="required" to "always". 039 Defferred untilaction 071 agreed 078 document UPA checks 039 079 Semantics of length=0, nil handling and defaults. (A071) 039 080 Tlog: Allow LengthKind delimited for packed/bcd (A074) 039 081 Update empty sequence section (A075) 039 082 semantics of minOccurs= 0 on choice branches (A076) 039 083 Implement RFC2116 084 Length|Kind pattern scanability rules 039 085 Invalid character substitution 039 086 infoset round tripi: Rephrase sentence 'It is possible to define a schema so that when infoset unparsed and the datastream reparsed, the same infoset will be produced' 039 087 Clarify use of relative paths in global components. 039 088 'DFDL expression' 039 089 Ageed that dfdl:represetnation 'text' is implied for strings and dfdl:represetnation 'binary' is implied for hexbinary 039 091 textStringPadCharacter textNumberPadCharacter must be a 1 byte character if the char set encoding is variable width? 039 092 finalDocumentTerminatorCanBeMissing and finalDocumentSeparatorCanBeMissing allowed only in 'default' format 039 093 remove textNumberFormat and textCalendarFormat. 039 094 Alignment should be 1 based 039 095 AP: Inf and Nan 039 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] 8-bit-ascii for dealing with binary data in text-like manner - problematic
by Mike Beckerle 24 Feb '10

24 Feb '10

I think we've got a fix for this. I found an official reference which has no "greyed out" codepoints. All 256 values are "mapped". The following ftp table (see URL below) officially defines the mapping for 8859-1 to unicode/iso10646. The table includes all 256 codepoints - some are specified as just <control> i.e., have no specific meaning, but their 8859 codepoint maps one-to-one and onto a unicode/10646 codepoint with the same value. Note that this property holds for 8859-1. It does not hold for 8859-2 to 8859-16, as these have character codes substituted into them that map to other places in the iso10646 codepoint space. Here's the correspondence table: ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-1.TXT If we reference this mapping table in the references of the DFDL spec, then I believe we can say that using encoding="iso-8859-1", you can treat binary data as textual, use patterns, etc., and the relationship to/from the infoset always insures preservation of the values of the bytes (parsing), and creation of bytes whose values exactly match the string codepoints (unparsing). This language can be added to the section on lengthKind="pattern" and binary data: Binary data can be handled using some of the conveniences of text by way of treating it as text with encoding="iso-8859-1". In this case literal text, such as length patterns, is interpreted as in the iso-8859-1 character encoding, and the correspondence of byte values in the data to a string in the DFDL infoset is one to one. That is, byte with value N, produces an infoset character with character code N. [reference to above FTP site]. On Thu, Feb 11, 2010 at 5:32 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > > Mike > > In the wikipedia entry for ISO 10646 it says "The system deliberately > leaves many code points not assigned to characters, even in the BMP. It does > this to allow for future expansion or to minimize conflicts with other > encoding forms." If those code points are below 256 then we have the same > problem as 8859? I can't find an actual map of the 10646 code points - you > have to buy it from ISO. > > Regards > > Steve Hanson > Programming Model Architect, WebSphere Message Broker, > OGF DFDL WG Co-Chair, > Hursley, UK, > Internet: smh(a)uk.ibm.com, > Phone (+44)/(0) 1962-815848 > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org Date: 11/02/2010 > 00:38 Subject: [DFDL-WG] 8-bit-ascii for dealing with binary data in > text-like manner - problematic Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > > Every "8-bit-ascii" encoding I can find has holes in the code page. That > is, values that don't have a corresponding character codepoint assigned. > > Example: iso-8859-X are a bunch of 8-bit ascii-based encodings that are > popular. > > If you lookup iso-8859-1 it has this language: > > Code values 00–1F, 7F–9F are not assigned to characters by ISO/IEC 8859-1. > > The lower range 20 to 7E (the G0 subset) maps exactly to the same coded G0 > subset of the ISO 646 US variant (commonly known as *ASCII*<http://en.wikipedia.org/wiki/ASCII>), > ... > > They're saying 7-bit ascii is included, and some other codes are there, but > they don't assign a codepoint generally. > > So, to me suggesting use of any particular code page for this purpose is > somewhat ambiguous. E.g., what does &#x01 mean in a string if the encoding > is iso-8859-1? There appears to be a set of translation tables that assign > this to unicode in standard ways that one can find on the web. But the > codepoint doesn't have an assigned meaning in iso-8859-X standards. > > Two possible clarifications: > 1) for all ascii-based character sets, we say that bytes 0x00 to 0xFF all > map to exactly those codepoints in ISO 10646 for the infoset, and vice > versa. > > 2) define dfdl:encoding="bytes" as a special character set name which has > the above property. > > Personally, I prefer 2. It is simpler to explain what is going on, and when > people are depending on bytes it will be clearer that they are. > > ...mike > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/dfdl-wg > > > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > >

1 0

Agenda for OGF DFDL WG call 24 February 2010- 13:00 UK (8:00 ET)
by Alan Powell 24 Feb '10

24 Feb '10

1. Remaining 037 review issues A: 16.2 scannablility with lengthKind pattern: Confirm that this is what we agreed In summary, you can use a data pattern on any element (complex, simple text, simple binary) as long as the bytes are legal in the stated encoding, which where binary data is involved in practice means an 8-bit ASCII encoding. By 8-bit ASCII we really mean an encoding where all the codepoints from 0-255 map to the equivalent value. Subsequent investigation indicates that 'all' 8-bit ASCII encodings have gaps so there isn't a valid character. Mike has suggested 1) for all ascii-based character sets, we say that bytes 0x00 to 0xFF all map to exactly those codepoints in ISO 10646 for the infoset, and vice versa. 2) define dfdl:encoding="bytes" as a special character set name which has the above property. Briefly discussed but no conclusion. B: Glossary Variable-Occurrence Item - Optional elements have a variable number of occurrences (0 or 1) and arrays also can have a variable number of occurrences (when minOccurs < maxOccurs). So when we say an item with a variable number of occurrences, this can mean either an optional element, or an array where minOccurs < maxOccurs. In either array or optional elements, we have the additional constraint that the DFDL representation properties do not preclude a variable number of occurrences. When dfdl:occursCountKind='explicit' and dfdl:occursCount has a literal constant as its value, or an expression that statically evaluates to a constant, then the DFDL properties are specifying exactly the number of occurrences for all instances and so are said to preclude a variable number of occurrences. If dfdl:occursCount has a formula as its expressed value, then the DFDL properties do not preclude a variable number of occurrences. MikeB Comment: This idea that you can have minOccurs < maxOccurs, but dfdl:occurs is equal to a constant and dfdl:occursKind="explicit" is causing us a bunch of grief in these definitions. Can we be conservative and just say it is a schema definition error if minOccurs < maxOccurs but the length is static, i.e., an explicit constant-valued expression? C: DFDL Schema Component Model What needs to be changed in the UML diagram? D: Sequence Groups Mike B: TBD: rewrite these property descriptions in terms of the grammar for sequences Specifically, this is where the FinalUnused Region must be described. E: Check other comments in document. 2. Go through Actions Current Actions: No Action 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progress 20/01: no progress 27/01: no progress 29/01: No progress. The predefined formats do not need to be available when the spec is published. Suman said that he had been mapping COBOL structures to DFDL and it didn't look as though the way text numbers are define is very usable. He will document for next call 03/02: No progress 10/02: No progress 17/03: No progress 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 079 MB:Encoding for binary fields when lenghtkind is pattern 17/02: Discussed but no conclusion 080 AP:Clarify semantics of fn:poisition and fn:count 17/02: no progress 081 AP: Inf and Nan The description is the way ICU behaves but need clarification. It isn't clear how inf and Nan are represented in the infoset. Need to investigate if XML allows these values 17/02: XML allows Nan and inf for float and double Dfdl will do the same. Requires more investigation of ICU. 3 DFDL v1 Specification completion. Draft 039 will be publish today. WG review and Comments by 3 March Draft 40 with updates for OGF submission - available 5 March Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Minutes for OGF DFDL Working Group Call, February 17-2010
by Alan Powell 18 Feb '10

18 Feb '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, February 17-2010 Attendees Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Peter Lambros (IBM) Stephanie Fetzer (IBM) Apologies Mike Beckerle (Oco) Suman Kalia (IBM) Tim Kimber(IBM) 1. Comments of latest discriminators doc v5. Steve questioned: It is a processing error if none of the choice branches are known to exist Steve: I know we agreed on this but I?m concerned that I am now unable to model the situation where I have a header, a footer, and in between either nothing or exactly 1 of a choice of n records It was agreed that the body could be an optional complex element with the choice as its content. Also confirmed dfdl:discriminator can be an annotation on · a local xs:element declaration · an xs:element reference · an xs:group reference (when the top level of a choice branch) · an xs:sequence (when the top level of a choice branch) · an xs:choice (when the top level of a choice branch) Action 045 will be closed. 2. Remaining 037 review issues 16.2 scannablility with lengthKind pattern: Confirm that this is what we agreed In summary, you can use a data pattern on any element (complex, simple text, simple binary) as long as the bytes are legal in the stated encoding, which where binary data is involved in practice means an 8-bit ASCII encoding. By 8-bit ASCII we really mean an encoding where all the codepoints from 0-255 map to the equivalent value. Subsequent investigation indicates that 'all' 8-bit ASCII encodings have gaps so there isn't a valid character. Mike has suggested 1) for all ascii-based character sets, we say that bytes 0x00 to 0xFF all map to exactly those codepoints in ISO 10646 for the infoset, and vice versa. 2) define dfdl:encoding="bytes" as a special character set name which has the above property. Briefly discussed but no conclusion. 3. Go through Actions Updated below 4. Biztalk comparison Short discussion which noted that DFDL could match Biztalk representation properties but was missing support for attributes and some simple types. Decided that we don't need to do anything in DFDL v1. 5 DFDL v1 Specification completion. It was agreed that the spec should be in the OGF review process by OGF 28 in March. Alan stated it needed 2 more drafts. Draft 39 with all the updates completed for a final WG review - available 24 Feb Comments by 3 March Draft 40 with updates for OGF submission - available 5 March Meeting closed, 14:00 Next call Wednesday 24 February January 2010 13:00 UK Next action: 083 Actions raised at this meeting No Action Current Actions: No Action 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progress 20/01: no progress 27/01: no progress 29/01: No progress. The predefined formats do not need to be available when the spec is published. Suman said that he had been mapping COBOL structures to DFDL and it didn't look as though the way text numbers are define is very usable. He will document for next call 03/02: No progress 10/02: No progress 17/03: No progress 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 079 MB:Encoding for binary fields when lenghtkind is pattern 17/02: Discussed but no conclusion 080 AP:Clarify semantics of fn:poisition and fn:count 17/02: no progress 081 AP: Inf and Nan The description is the way ICU behaves but need clarification. It isn't clear how inf and Nan are represented in the infoset. Need to investigate if XML allows these values 17/02: XML allows Nan and inf for float and double Dfdl will do the same. Requires more investigation of ICU. Closed actions No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 13/01: Stephaine took us through a description of WTX identifiers. Mike agreed to write up in DFDL terms. 20/01: Mike will write up 27/01: further discussion of discriminators 29/01: Alan had emailed both proposals but not enough time to discuss 02/02: Agreed to adopt 'component exists' semantics for discriminators 10/02: 'component exists' proposal updated. comments by next call. 17/02: reviewed needs minor updates. Closed 082 MB: Should alignment be 0 or 1 based 17/02: Mike has reworded the section to be 1 based and Agreed this was OK Closed Work items: No Item target version status 005 Improvements on property descriptions not started 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 069 ICU fractional seconds 039 070 Write DFDL primer 071 Write test cases. 072 it is a processing error if the number of occurrences in the data does not match the value of the expression or prefix 039 073 Rename dfdl:separatorPolicy="required" to "always". 039 Defferred untilaction 071 agreed 078 document UPA checks 039 079 Semantics of length=0, nil handling and defaults. (A071) 039 080 Tlog: Allow LengthKind delimited for packed/bcd (A074) 039 081 Update empty sequence section (A075) 039 082 semantics of minOccurs= 0 on choice branches (A076) 039 083 Implement RFC2116 084 Length|Kind pattern scanability rules 085 Invalid character substitution 086 infoset round tripi: Rephrase sentence 'It is possible to define a schema so that when infoset unparsed and the datastream reparsed, the same infoset will be produced' 087 Clarify use of relative paths in global components. 088 'DFDL expression' 089 Ageed that dfdl:represetnation 'text' is implied for strings and dfdl:represetnation 'binary' is implied for hexbinary 091 textStringPadCharacter textNumberPadCharacter must be a 1 byte character if the char set encoding is variable width? 092 finalDocumentTerminatorCanBeMissing and finalDocumentSeparatorCanBeMissing allowed only in 'default' format 093 remove textNumberFormat and textCalendarFormat. 094 Alignment should be 1 based Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 081: Clarification of NaN and Inf
by Alan Powell 18 Feb '10

18 Feb '10

DFDL support for Infinity and Nan need clarification. There are two aspects Representation of infinity and Nan in the infoset Representation of Infinity and Nan in the datastream Representation of infinity and Nan in the infoset DFDL should follow XML which allows +inf, -inf and Nan for Float and Double >From XML Schema Part 2: Datatypes Second Edition [Definition:] float is patterned after the IEEE single-precision 32-bit floating point type [IEEE 754-1985]. The basic ·value space· of float consists of the values m × 2^e, where m is an integer whose absolute value is less than 2^24, and e is an integer between -149 and 104, inclusive. In addition to the basic ·value space· described above, the ·value space· of float also contains the following three special values: positive and negative infinity and not-a-number (NaN). The ·order-relation· on float is: x < y iff y - x is positive for x and y in the value space. Positive infinity is greater than all other non-NaN values. NaN equals itself but is ·incomparable· with (neither greater than nor less than) any other value in the ·value space·. The special values positive and negative infinity and not-a-number have lexical representations INF, -INF and NaN, respectively. Lexical representations for zero may take a positive or negative sign. Similarly description for Double Minor changes to infoset [dataValue] description 1. [dataValue] The value in the value space (as defined by XML Schema Part 2: Datatypes [XSDLV1] ) of the [datatype] member or special value nil. In a complex element information item this member has no value. For information items of datatype xs:string, the value will be the ISO10646 character codes of the string and 'implicit' (also known as logical), left-to-right bidirectional ordering and orientation. Is it sufficient just to refer to XML schema rather than spell out the value spaces in this specification? Representation of Infinity and Nan in the data stream ICU allows a single character for Nan and infinity. The infinity character can be prefixed by the positive or negative prefix (ie the prefix part of he pattern) This is too restrictive for DFDL as we need to be able to support at least 'INF' '-INF' and 'NaN' Only minor changes to current description. numberInfinityRep String The value used to represent infinity. Infinity is represented as string with the positive or negative prefixes and suffixes from the numberPattern applied This property is applicable when dfdl: textNumberRepresentation is 'standard' and the simple type type is float or double. Annotation: dfdl:textNumberFormat numberNaNRep String The value used to represent NaN. NaN is represented as string and the positive or negative prefixes and suffixes from the numberPattern are not used This property is applicable when dfdl: textNumberRepresentation is 'standard' and the simple type type is float or double. Annotation: dfdl:textNumberFormat Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG call 17 February 2010- 13:00 UK (8:00 ET)
by Alan Powell 16 Feb '10

16 Feb '10

1. Comments of latest discriminators doc v5. 2. Remaining 037 review issues 16.2 scannablility with lengthKind pattern: Confirm that this is what we agreed In summary, you can use a data pattern on any element (complex, simple text, simple binary) as long as the bytes are legal in the stated encoding, which where binary data is involved in practice means an 8-bit ASCII encoding. By 8-bit ASCII we really mean an encoding where all the codepoints from 0-255 map to the equivalent value. Subsequent investigation indicates that 'all' 8-bit ASCII encodings have gaps so there isn't a valid character. Mike has suggested 1) for all ascii-based character sets, we say that bytes 0x00 to 0xFF all map to exactly those codepoints in ISO 10646 for the infoset, and vice versa. 2) define dfdl:encoding="bytes" as a special character set name which has the above property. Action rasied 3. Go through Actions see below 4. Biztalk comparison Discuss analysis of Biztalk function. Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 13/01: Stephaine took us through a description of WTX identifiers. Mike agreed to write up in DFDL terms. 20/01: Mike will write up 27/01: further discussion of discriminators 29/01: Alan had emailed both proposals but not enough time to discuss 02/02: Agreed to adopt 'component exists' semantics for discriminators 10/02: 'component exists' proposal updated. comments by next call. 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deferred 13/01:no progress 20/01: no progress 27/01: no progress 29/01: No progress. The predefined formats do not need to be available when the spec is published. Suman said that he had been mapping COBOL structures to DFDL and it didn't look as though the way text numbers are define is very usable. He will document for next call 03/02: No progress 10/02: No progress 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 079 AP:Encoding for binary fields when lenghtkind is pattern 080 AP:Clarify semantics of fn:poisition and fn:count 081 AP: Inf and Nan The description is the way ICU behaves but need clarification. It isn't clear how inf and Nan are represented in the infoset. Need to investigate if XML allows these values 082 MB: Should alignment be 0 or 1 based Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL discriminators v5
by Alan Powell 16 Feb '10

16 Feb '10

The latest, and hopefully the last, discriminators write-up which incorporates comments from Steve H and Stephanie. Steve questioned: It is a processing error if none of the choice branches are known to exist. Steve: I know we agreed on this but I?m concerned that I am now unable to model the situation where I have a header, a footer, and in between either nothing or exactly 1 of a choice of n records Also the WG agreed that: Discriminators are not allowed on - Global groups and the top level sequence or choice of a global group. - Global element decalrations - The top level group of a complex type. - Anonymous groups other than when it is the top level of a choice branch. Which I think reduces to dfdl:discriminator can be an annotation on · a local xs:element declaration · an xs:element reference · an xs:group reference · an xs:sequence (when the top level of a choice branch) · an xs:choice (when the top level of a choice branch) Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0