- dfdl-wg - lists.ogf.org

Fw: OGF DFDL WG Call Agenda 2011-06-28 - resending
by Steve Hanson 28 Jun '11

28 Jun '11

Due to an issue with CutePDF truncating text, here is the agenda for today. 1. Spec issues from IBM As implementation has progressed, a number of issues have been spotted in the DFDL 1.0 spec by the IBM implementation team. List so far for resolution. 6.3. Bullet for logical value. State that the string must obey the lexical representation of the type. 6.3.1. Explicit statement of white space rules is missing. Just white space for a DFDL string literal property is an error. When a property can be a list of string literals, white space between values is collapsed. 7.7. What is the scope of a variable with respect to included/imported xsds? Can I refer from xsd A to a variable defined in parent xsd B? 9.2. Grammar. Does not allow for a prefix length itself having a prefixed length, so does not match property description (ie, the change to permit this is draft 045 omitted to update the grammar). 13. Be explicit on the DFDL literals allowed for text properties. See separate e-mail. 13.6. When textNumberRep is zoned, property description should state that base is assumed to be 10. 13.6. State that dfdl:ignoreCase not honoured by textStandardExponentCharacter, textStandardInfinityRep, textStandardNanRep as ICU does not offer that flexibility. 13.6 Allow multiple characters for textStandardExponentCharacter to handle reps like 1.23x10^4 as ICU allows that. Note that property name will have to change, eg, to textStandardExponent.. 13.6. Change meaning of textNumberCheckPolicy 'lax' to align more closely with ICU: "If ‘lax' and dfdl:textNumberRep is 'standard' then grouping separators are ignored, leading and trailing whitespace is ignored, leading zeros are ignored." 13.6. Disallow the use of empty string for textStandardExponentCharacter. State property must be set if the pattern contains an 'E' 13.6. Disallow the use of empty string for textStandardDecimalSeparator, State property must be set if the pattern contains a '.' 13.6 Disallow the use of empty string for textStandardGroupingSeparator, State property must be set if the pattern contains a ',' 13.6. Allow decimal separator to be a List of DFDL String Literals or a DFDL expression. This allows modelling of EDIFACT where user can choose decimal separator in ISA header but '.' is also allowed. 13.6. When textNumberPadCharacter is '0' which it commonly is, a value of say '00000' will get trimmed to the empty string, whereas the intent would surely be to trim to '0'. Need a rule for this. 13.6.Should textStandardDecimalSeparator ignored when the logical type is not decimal/float/double? 13.6.1.1. Erroneously mentions sigDigits in reference to the BNF for textNumberPattern. 13.6.1.1. Formatting. Uses terms 'minimum/maximum integer/fraction digits' but does not define them. 13.9. Should state that textBooleanTrue/FalseRep properties are used after trimming when parsing, and before padding when unparsing. If lengthKind is explicit or implicit and either textPadKind or textTrimKind is none then the properties must have the same length else schema definition error. 13.16. Nil literal character. When representation is binary, the encoding property is not used, which implies that a nil value must only contain DFDL hex literals (%rXX;) 13.16. Clarify nilKind 'literalCharacter' and 'literalValue' and whether nil test is applied before or after trimming. 15. Using a discriminator to resolve each branch of a choice with a very large number of branches has a performance impact as each branch must be evaluated in turn. In cases where the name of the branch element is known or derivable in advance (ie, each discriminator expression is of the form {..\tag eq "elem1"} or similar) then consider adding a property to dfdl:choice that provides an expression that returns a QName, being the name of the branch element. Note: IBM MRM has something like this. 16.1. Revise entire section. 16.2. State it is a processing error if the stop value is missing when parsing. 2. Spec issues from Mike Beckerle A number of issues have been spotted by Mike. Some have been resolved by action 139 below, but others remain. See separate e-mail. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/06/2011 14:46 ----- From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 28/06/2011 13:50 Subject: RE: [DFDL-WG] OGF DFDL WG Call Agenda 2011-06-28 Steve, When I download this agenda PDF file, it’s all cut off on the right edge of the page, making it hard to figure out what some of these line items are about. Something is up with the way you converted this teamroom page to a pdf. If this was paper, I’d suspect the metric vs. US paper size issue, but this is all bits…. J …mikeb From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Steve Hanson Sent: Monday, June 27, 2011 12:18 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] OGF DFDL WG Call Agenda 2011-06-28 Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Spec issues from IBM: Restrictions on properties of type 'DFDL string literal'
by Steve Hanson 27 Jun '11

27 Jun '11

For discussion on DFDL WG call. Please find proposal from IBM to clarify the allowable characters that be used in some of the DFDL properties that are defined as type 'DFDL String Literal'. It is proposed that the properties in question do not need the full power of DFDL String Literal. escaping ( single character ) escapeCharacter escapeEscapeCharacter escapeBlockStart escapeBlockEnd Property value must resolve to a single character DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed padding ( single character or byte ) textStringPadCharacter textNumberPadCharacter textBooleanPadCharacter textCalendarPadCharacter Property value must resolve to a single character or a single byte value DFDL character entities are allowed The raw byte entity ( %#r ) is allowed subject to the restrictions already documented for these properties DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Number separators ( single character ) textStandardDecimalSeparator textStandardGroupingSeparator textStandardExponentCharacter Property value must resolve to a single character DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Number representations ( multiple characters ) textStandardInfinityRep textStandardNanRep textStandardZeroRep DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Boolean representations ( multiple characters ) textBooleanTrueRep textBooleanFalseRep DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Nil representations ( multiple characters/bytes ) nilValue DFDL character entities are allowed The raw byte entity ( %#r ) is allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are allowed Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Spec issues from Mike Beckerle
by Steve Hanson 27 Jun '11

27 Jun '11

For discussion on DFDL WG calls. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 27/06/2011 17:26 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 15/06/2011 01:59 Subject: Issues to add to work items list Steve, Below is the list I have so far of items in the spec that go beyond just typos - where re-wording is required or advisable. All but the last of these is tied up with the length & delimiters issue, but separate of the "known to exist" vs. "missing" topic. I've not bothered to provide anything about the section on "known to exist" and such. That section is already the discussion of a work item/topic. The last one is just a nit about BOMs. ...mikeb -------------------------------------------------------------------------------------------- 5.2.2 MinLength, MaxLength These facets are used: When dfdl:lengthKind=”implicit”. In that case the length is given by the value of xs:maxLength. In this case minLength if specified is required to be equal to maxLength (schema definition error otherwise). For validation of variable length string elements. It is a processing error when a fixed-length string is found to have a number of characters not equal to the fixed number. For example, if a fixed-length string also has delimiters we might be able to successfully separate it from the surrounding elements depending on the delimiter specifications; however, if the length of the fixed-length string is not equal to the number specified as the fixed length then it is a processing error (not simply a validation error).[MB1] [MB1]Contradicts statement that scanning for delimiters is off.(Discussed where dfdl:lengthKind=’explicit’ is described) What is a fixed length string? Clearly if it has lengthKind=”explicit” it is fixed length. What if it has lengthKind=”implicit” and maxLength=”10”. Is that a fixed length string which shuts off delimiter scanning also? If so then this paragraph is erroneous and misleading. <SMH>Yes, Tim and I have noted that this paragraph needs revising depending on the outcome of action 139 <SMH> -------------------------------------------------------------------------------------- 9.2 DFDL Syntax Grammar Change to introduce concept of EnclosedItem or ChildItem (I used EnclosedItem below): Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedItem [ Separator EnclosedItem ]* PostfixSeparator ] FinalUnusedRegion EnclosedItem = Element | Array | ComplexContent[MB1] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedItem ] FinalUnusedRegion[MB2] [MB1]Refactored to share the EnclosedItem concept to Choices also. Should perhaps be named ChildItem. This is useful when discussing how parsing, defaulting, etc. work as well. <SMH>Seems fine, it is effectively just a renaming<SMH> --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Table 14 Implicit Alignment in bits Note: Specifying the implicit alignment in bits does not imply that dfdl:lengthUnits 'bits' can be specified for all simple types.[MB1] [MB1]I do not understand this comment. What exactly is the restriction? <SMH>It is really saying that alignmentUnits and lengthUnits are independent and have their own rules for when they are applicable. <SMH> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- lengthKind Enum Controls how the representation length of the component is determined. Valid values are: 'explicit', 'delimited', 'prefixed', 'implicit', 'pattern', 'endOfParent' A full description of each enumeration is given in the later sections. 'explicit' means the length of the item is given by the dfdl:length property 'delimited' means the item is delimited by a terminator or separator[MB1] ‘prefixed’ means the length of the item is given by an immediately preceding prefix field specified using prefixLengthType. ‘implicit means the length is to be determined in terms of the type of the element and its schema-specified properties if any. ‘pattern’ means the length of the item is given by a regular expression specified using the dfdl:lengthPattern property. ‘endOfParent’ means that the item is terminated by the termination of the containing construct. Annotation: dfdl:element, dfdl:simpleType [MB1]To me this is a very strong statement. It means that an outside-in parse is allowed where for a sequence, we can scan and determine its end, and then parse the children. It requires that “scan” is a well-defined concept for the contents of the sequence. It means there can be nothing inside which requires the suspension of scanning. It means contained elements that have length explicit and representation binary, are simply not allowed. <SMH>Not entirely true, we allow some binary types to have delimited lengthKind (eg, BCD and Packed Decimals). There are formats out there that require this. <SMH> To me it also means you cannot change the character set encoding, or have a contained element that itself uses an overlapping set of delimiters with the enclosing group’s delimiters. – Is that going too far? – Delimited is like pattern. It restricts what is in the data stream substantially. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The rules for resolving ambiguity between delimiters are: 1. When two delimiters have a common prefix, the longest delimiter has precedence. 2. When two delimiters have exactly the same value, the innermost (most deeply nested) delimiter has precedence. 3. When the separator and terminator on a group have the same value, the separator has precedence.[MB1] [MB1]By precedence, this must mean it is tried first, but the parser might backtrack and assume it to be a terminator instead. This seems problematic to me. I’d like to either rule this out and say they can’t be the same, or see a use case where this is needed, a backtracking parser can parse it, and there’s no more reasonable way to structure the schema. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 12.3.5.1 Pattern-Based Lengths - Scanability Any element (complex, simple text, simple binary) may have a dfdl:lengthKind 'pattern' as long as the bytes in the content region of the element are legal in the stated encoding of that element. Where a complex element has children with binary representation in practice this means an 8-bit ASCII encoding. [MB1] [MB1]Not necessarily ASCII. An 8-bit encoding such that any byte value is valid is the real requirement. The point is no single byte value is invalid, and no combinations of adjacent byte values are invalid so that any binary data won’t trip up a character conversion and subsequent scan. Hmmm. The 8-bit character set used must have a transformation into Unicode which is bijective and information preserving. I.e., a unique Unicode character for each code point, and no “invalid” chracters which have no corresponding Unicode value. However, Ascii-based sets like 8859-1 are not strictly speaking, required. ----------------------------------------------------------------------------------------------------------------------------------------------- 12.3.7.1.3 Byte Order Mark If a byte-order mark codepoint appears at the start of a UTF-8, [MB1] UTF-16 or UTF-32 encoded string then the byte-order mark will be included as part of the string payload[1]. That is, for the UTF-8, UTF-16 and UTF-32 character encodings, a byte-order-mark codepoint is treated as a character of the string in DFDL and contributes to the length. A way of eliminating the byte-order mark so that it does not end up in the infoset is that the byte-order mark can be modeled as a separate element before the string. This BOM element can be either required or optional depending on whether one is expected or optional at the beginning of the string. [1] Byte-order marks are explicitly stated to be “not characters” in the Unicode standard. [MB1]No such thing as a BOM codepoint in a UTF-8 string. A UTF-8 byte sequence might encode the character code for a BOM, but this would be a meaningless inclusion of a BOM character code in a context where it will never be interpreted. I suggest that we drop the term UTF-8 here, and BOM’s that get encoded when they are interpreted as character codes, and translated by the UTF-8 encoding algorithm into a multi-byte UTF-8 byte sequence, is handled the same way as other non-characters, i.e., what do we do when a high or low surrogate codepoint is present and we’re to encode as UTF-8. I think the answer is we run the UTF-8 encode/decode algorithm, and whatever Unicode character code it creates is what it creates, and if that happens to come out as any of the non-characters (BOMs, surrogates, others perhaps), so be it. The topic is about Unicode non-characters, not specifically BOMs. The general topic is encoding/decoding our infoset Unicode character codes which have no real representation in the specified encoding. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2011-06-28
by Steve Hanson 27 Jun '11

27 Jun '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Rescheduled: OGF DFDL Working Group weekly call (21 Jun 15:00 GDT in Hursley DE3S09)
by Steve Hanson 21 Jun '11

21 Jun '11

Note new dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. Sorry to keep rescheduling....

1 0

Rescheduled: OGF DFDL Working Group weekly call (21 Jun 17:00 GDT in Hursley DE3S09)
by Steve Hanson 21 Jun '11

21 Jun '11

Note new dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. Rescheduling from Wed to Tues due to clashing calendars. Affects all meetings from this week inclusive.

1 0

OGF DFDL WG Call Minutes 2011-06-15
by Steve Hanson 16 Jun '11

16 Jun '11

Please find minutes of the above meeting on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] A selection of example data formats
by Steve Hanson 15 Jun '11

15 Jun '11

Hi Mike More replies but this time I'll keep them together here as the Word doc would get hard to read.... Tim and I have been thinking on similar lines as your "have enough properties to determine that the length is zero". In addition to your examples there are also: - lengthKind="prefixed" and prefix length is 0 - lengthKind="explicit" and lengthCount expression evaluates to 0 Using the same sectioning as the document... ------------------------------------------------- a) Fixed length, no delimiters We agree that there should be no defaulting when the length is > 0. Need to decide whether the length = 0 case implies defaulting, we think it does as the property determines that the length is zero b) Fixed length, only parent has delimiters This boils down to whether we need to detect early termination. Spec and yourself are clear that scanning is off when parsing fixed length. I'd like to hear what Steph has to say on this. c) Fixed length, initiators You want to treat this the same as un-initiated fixed length. OK, but more on this later under i) e) Delimited, separators required We agree that defaults should be applied when adjacent separators encountered f) Delimited, separators suppressed at end We agree that defaults should be applied when adjacent separators encountered and at the end g) Delimited, initiators, separators required We agree that defaults should be applied when adjacent separators encountered i) Delimited, initiators, separators suppressed You want defaults to be applied when an element is entirely absent (B in the example) Tim and I struggle to differentiate this case from c). At the start of B processing, there is nothing in the data to indicate B and the next thing is C's initiator. So why is the defaulting rule different? Take this one step further - my data is fixed length, initiated and the parent has a suppressed separator - so which of c) and i) applies? How does the parser know when a group has ended? One of Tim's rules was when an enclosing delimiter is found. That is not always the case. Tim suggested that if the immediate parent had lengthKind="implicit" then we would not be looking for delimiters. I believe your YES was agreeing with that? We would say it is also true if the immediate parent had lengthKind = "explicit" or "pattern" too. What is the algorithm for selecting the next occurrence? Tim and I discussed this, and there is not an issue here. The occursCountKind always tells you the number to expect (which might be 'don't know' if occursCountKind = "parsed" in which case we just speculatively parse). When parsing a group with separatorPolicy=suppressed, is every group member a 'point of uncertainty'? Agree with your statement. ---------------------------------- Other things to discuss: Defaulting complex elements when parsing The spec says that if zero length content is obtained for a complex element then it is defaulted, which means the element's complex type is walked and default values are sent to the infoset for required elements. It is an error if any required elements do not have a default value. A simpler alternative is to create just the element in the infoset with no children, but this would fail validation if switched on. Separator position Any rules that we agree on must take into account infix v prefix v postfix. In practice this determines how an element is 'bound' to a separator. Prefix it is bound to the beginning, postfix it is bound to the end, infix it is bound to the beginning except for the first element (need to check with Steph is that is how WTX does it). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Tim Kimber/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 11/06/2011 01:56 Subject: RE: A selection of example data formats My comments on your examples. I had to turn it into a word doc to reasonably put my commentary inline into this. I think the concept of an element declaration being classified into: · Can be defaulted from nothing · Can be defaulted from empty content (but requires some framing to determine that the content is empty) · Cannot be defaulted (requires at least some content bits, possibly also some framing) … I think this is something we’re in need of in the spec. If the element can be defaulted from nothing, and it is required, and we have nothing, i.e., no bits meaning that we have enough properties to determine that the length is zero, then we default it to get the infoset value. If it’s optional, then we don’t default it, and nothing goes into the infoset. This begs the question of “have enough properties to determine that the length is zero”. E.g., of this: end of data, end of parent, this element has no delimiters, but lengthKind=delimited and a parent delimiter was immediately encountered which terminates the element after zero bits. lengthKind=”pattern”, lengthPattern=”a*”, and the data has no “a” characters, so the length comes out zero, and no bits are consumed. Recursively, length is zero for a group requires same properties to hold inductively and for the group itself. I’m not sure I’ve got all the cases here, but it’s something like this. That’s all for my brain on DFDL today….. From: Tim Kimber [mailto:KIMBERT@uk.ibm.com] Sent: Thursday, June 02, 2011 4:41 PM To: mbeckerle.dfdl(a)gmail.com Cc: Steve Hanson Subject: A selection of example data formats Mike, Steve asked me to forward this text file that I have put together. I put it together as background material for our discussions about the parsing of DFDL elements and groups. Key issues: - The specification uses the terms 'empty', 'missing' and 'known not to exist' in reference to elements. We need to work out what these terms mean so that the spec can be made clearer. - In my opinion, the terms 'missing' and 'known not to exist' should not have different meanings - it invites criticism. If 'missing' means something different from 'known not to exist' then we need a different word or phrase. - The application of default values for missing required elements in the parser is problematic. I think Steve may have sent you an email about this, so I won't outline the issues here ( Steve, please can you forward your email to me ). Disclaimer : This set of data formats does not highlight all of the unresolved questions around the parsing of groups - only the ones that were in play at the time I produced the document. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] A selection of example data formats #1
by Steve Hanson 15 Jun '11

15 Jun '11

Mike I realise that this is the first of your two e-mails on this subject, but I've added some comments anyway to keep things in context...more to come on your 2nd mail. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Tim Kimber/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 11/06/2011 01:56 Subject: RE: A selection of example data formats Tim, I need schemas in order to interpret these puzzles. I can propose some, but I’m not sure they would match the scenarios you are trying to illustrate. You mention somewhere lengthKind=”explicit”….. which I think makes some of these puzzles very easy. To me, <element name=”a” type=”string” default=”AAAAA” dfdl:lengthKind=”explicit” dfdl:length=”5”/> is really easy. The default will ONLY be used on unparsing to fill in something missing from the infoset. When parsing, 5 characters are consumed. If they’re not available because end of data/parent occurred, then it’s a processing error, but the default would never be used. <smh> Tim and I agree on this when the dfdl:length property (which can be an expression) evaluates to > 0 </smh>. Our current draft spec says that when consuming these 5 characters, delimiter scanning is not even occurring. (though we have a separate topic to consider if that is right or not. I think it is right, and if you want to look for a terminator, as well as check for a fixed length, you have to use a dfdl:assert to check the length, but use lengthKind=”delimited”. But I digress.) <smh> Digression. Using dfdl:assert only helps with the last element in a sequence of fixed length elements. </smh> I do think we need to consolidate and clarify terminology somewhat, and there’s an important special case we need to clarify. Missing = Known Not To Exist = Not Existing Existing = Not Missing The trick is this: Missing means there is no place where the element could appear. The reason for this stilted language is that we’re trying to allow elements whose declarations admit representations of exactly zero bits, to be classified as Existing, not missing. So the evidence in the bit stream has to go on the other side, toward showing something is NOT there. Empty implies Existing, but where there is no grammar content region bits. The usual examples would be adjacent delimiters: “AAAAA,,CCCCC”, or initiators: “A:AAAAA, B:, C:CCCCC” This very special case where an element, including framing, legitimately occupies zero bits of the data stream is the trick. This is the ambiguity that some of your examples are about. This is why there’s this emphasis in the spec language Alan worked so hard on, about whether we can show things *don’t exist* in the data stream. It’s because we want this special case to be allowed to be “existing” even though there are no bits showing evidence of existence at all. To make things work, I think this special case must be classified as Existing, not Missing. Alternatively we can introduce a special term for this situation. (Call it a Ghost element, for example. It exists, but you can’t see it.). The trick here is resolving the point of uncertainty, and determining that the length is zero without unbounded speculation. We have to be very careful how much we require speculation to do for us. Specifically, we do *not* want to say that something’s length is zero because speculation forward after this element led to a successful parse. That’s the so-called “squeeze” algorithm for determining where something ends (its length), and I really worry about it. I think there are possibly other places where our draft spec implies this kind of squeeze algorithm for length. (Ex: I am worried about when separator and terminator are the same character. But I digress.) An element’s length is zero if it has no bits of framing (no initiator nor terminator of its own, no alignment or skip crud) and we immediately encounter terminating markup ( parent separator, parent terminator….), or we encounter end-of-parent/data. Assuming the element is required, has a default, and the length dfdl properties admit a zero-length representation, then in this case we would generate the default as its infoset value, and consume zero bits of the input stream. Example: <smh> We need to be very careful when using the term 'length'. Length in DFDL only ever means content region. The function dfdl:representationLength() is defined as omitting framing (see spec 23.5.3). If want to talk about total absence of framing as well then we need a new term. Tim and I agree with your second sentence if it uses length to mean content, but not if it includes framing, as it does not cover the case (eg) where I have a dfdl:initiator, zero length content, and a dfdl:separator in force. Note that you allow this case in your 2nd email, so maybe you changed your mind as you progressed through Tim's puzzles? </smh>. <sequence dfdl:terminator=”;”> <element name=”speedLimit” type=”int” dfdl:initiator=”” dfdl:terminator=”” default=”100” dfdl:lengthKind=”delimited” /> </sequence> So the data stream “;” would produce an infoset containing element speedLimit with value 100. (I intentionally avoided the string case, because of the empty string being a legitimate value, which disables defaulting really.) <smh> Tim and I have discussed what to do with empty content for xs:string. Spec section 13 currently implies that for this to mean empty string in the infoset, a default value would have to be supplied, set to empty string. I believe this was to make optional and required elements consistent, ie, for an optional element, empty content does not mean empty string, it means 'missing' (as currently defined). Ergo same for a required element, and defaulting then takes place. </smh> I’ll send an annotated version of your puzzles in a separate message. From: Tim Kimber [mailto:KIMBERT@uk.ibm.com] Sent: Thursday, June 02, 2011 4:41 PM To: mbeckerle.dfdl(a)gmail.com Cc: Steve Hanson Subject: A selection of example data formats Mike, Steve asked me to forward this text file that I have put together. I put it together as background material for our discussions about the parsing of DFDL elements and groups. Key issues: - The specification uses the terms 'empty', 'missing' and 'known not to exist' in reference to elements. We need to work out what these terms mean so that the spec can be made clearer. - In my opinion, the terms 'missing' and 'known not to exist' should not have different meanings - it invites criticism. If 'missing' means something different from 'known not to exist' then we need a different word or phrase. - The application of default values for missing required elements in the parser is problematic. I think Steve may have sent you an email about this, so I won't outline the issues here ( Steve, please can you forward your email to me ). Disclaimer : This set of data formats does not highlight all of the unresolved questions around the parsing of groups - only the ones that were in play at the time I produced the document. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Issues to add to work items list
by Steve Hanson 15 Jun '11

15 Jun '11

Thanks Mike. I have added a couple of comments below...we can discuss fully on the calls. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 15/06/2011 01:59 Subject: Issues to add to work items list Steve, Below is the list I have so far of items in the spec that go beyond just typos - where re-wording is required or advisable. All but the last of these is tied up with the length & delimiters issue, but separate of the "known to exist" vs. "missing" topic. I've not bothered to provide anything about the section on "known to exist" and such. That section is already the discussion of a work item/topic. The last one is just a nit about BOMs. ...mikeb -------------------------------------------------------------------------------------------- 5.2.2 MinLength, MaxLength These facets are used: When dfdl:lengthKind=”implicit”. In that case the length is given by the value of xs:maxLength. In this case minLength if specified is required to be equal to maxLength (schema definition error otherwise). For validation of variable length string elements. It is a processing error when a fixed-length string is found to have a number of characters not equal to the fixed number. For example, if a fixed-length string also has delimiters we might be able to successfully separate it from the surrounding elements depending on the delimiter specifications; however, if the length of the fixed-length string is not equal to the number specified as the fixed length then it is a processing error (not simply a validation error).[MB1] [MB1]Contradicts statement that scanning for delimiters is off.(Discussed where dfdl:lengthKind=’explicit’ is described) What is a fixed length string? Clearly if it has lengthKind=”explicit” it is fixed length. What if it has lengthKind=”implicit” and maxLength=”10”. Is that a fixed length string which shuts off delimiter scanning also? If so then this paragraph is erroneous and misleading. <SMH>Yes, Tim and I have noted that this paragraph needs revising depending on the outcome of action 139 <SMH> -------------------------------------------------------------------------------------- 9.2 DFDL Syntax Grammar Change to introduce concept of EnclosedItem or ChildItem (I used EnclosedItem below): Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedItem [ Separator EnclosedItem ]* PostfixSeparator ] FinalUnusedRegion EnclosedItem = Element | Array | ComplexContent[MB1] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedItem ] FinalUnusedRegion[MB2] [MB1]Refactored to share the EnclosedItem concept to Choices also. Should perhaps be named ChildItem. This is useful when discussing how parsing, defaulting, etc. work as well. <SMH>Seems fine, it is effectively just a renaming<SMH> --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Table 14 Implicit Alignment in bits Note: Specifying the implicit alignment in bits does not imply that dfdl:lengthUnits 'bits' can be specified for all simple types.[MB1] [MB1]I do not understand this comment. What exactly is the restriction? <SMH>It is really saying that alignmentUnits and lengthUnits are independent and have their own rules for when they are applicable. <SMH> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- lengthKind Enum Controls how the representation length of the component is determined. Valid values are: 'explicit', 'delimited', 'prefixed', 'implicit', 'pattern', 'endOfParent' A full description of each enumeration is given in the later sections. 'explicit' means the length of the item is given by the dfdl:length property 'delimited' means the item is delimited by a terminator or separator[MB1] ‘prefixed’ means the length of the item is given by an immediately preceding prefix field specified using prefixLengthType. ‘implicit means the length is to be determined in terms of the type of the element and its schema-specified properties if any. ‘pattern’ means the length of the item is given by a regular expression specified using the dfdl:lengthPattern property. ‘endOfParent’ means that the item is terminated by the termination of the containing construct. Annotation: dfdl:element, dfdl:simpleType [MB1]To me this is a very strong statement. It means that an outside-in parse is allowed where for a sequence, we can scan and determine its end, and then parse the children. It requires that “scan” is a well-defined concept for the contents of the sequence. It means there can be nothing inside which requires the suspension of scanning. It means contained elements that have length explicit and representation binary, are simply not allowed. <SMH>Not entirely true, we allow some binary types to have delimited lengthKind (eg, BCD and Packed Decimals). There are formats out there that require this. <SMH> To me it also means you cannot change the character set encoding, or have a contained element that itself uses an overlapping set of delimiters with the enclosing group’s delimiters. – Is that going too far? – Delimited is like pattern. It restricts what is in the data stream substantially. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The rules for resolving ambiguity between delimiters are: 1. When two delimiters have a common prefix, the longest delimiter has precedence. 2. When two delimiters have exactly the same value, the innermost (most deeply nested) delimiter has precedence. 3. When the separator and terminator on a group have the same value, the separator has precedence.[MB1] [MB1]By precedence, this must mean it is tried first, but the parser might backtrack and assume it to be a terminator instead. This seems problematic to me. I’d like to either rule this out and say they can’t be the same, or see a use case where this is needed, a backtracking parser can parse it, and there’s no more reasonable way to structure the schema. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 12.3.5.1 Pattern-Based Lengths - Scanability Any element (complex, simple text, simple binary) may have a dfdl:lengthKind 'pattern' as long as the bytes in the content region of the element are legal in the stated encoding of that element. Where a complex element has children with binary representation in practice this means an 8-bit ASCII encoding. [MB1] [MB1]Not necessarily ASCII. An 8-bit encoding such that any byte value is valid is the real requirement. The point is no single byte value is invalid, and no combinations of adjacent byte values are invalid so that any binary data won’t trip up a character conversion and subsequent scan. Hmmm. The 8-bit character set used must have a transformation into Unicode which is bijective and information preserving. I.e., a unique Unicode character for each code point, and no “invalid” chracters which have no corresponding Unicode value. However, Ascii-based sets like 8859-1 are not strictly speaking, required. ----------------------------------------------------------------------------------------------------------------------------------------------- 12.3.7.1.3 Byte Order Mark If a byte-order mark codepoint appears at the start of a UTF-8, [MB1] UTF-16 or UTF-32 encoded string then the byte-order mark will be included as part of the string payload[1]. That is, for the UTF-8, UTF-16 and UTF-32 character encodings, a byte-order-mark codepoint is treated as a character of the string in DFDL and contributes to the length. A way of eliminating the byte-order mark so that it does not end up in the infoset is that the byte-order mark can be modeled as a separate element before the string. This BOM element can be either required or optional depending on whether one is expected or optional at the beginning of the string. [1] Byte-order marks are explicitly stated to be “not characters” in the Unicode standard. [MB1]No such thing as a BOM codepoint in a UTF-8 string. A UTF-8 byte sequence might encode the character code for a BOM, but this would be a meaningless inclusion of a BOM character code in a context where it will never be interpreted. I suggest that we drop the term UTF-8 here, and BOM’s that get encoded when they are interpreted as character codes, and translated by the UTF-8 encoding algorithm into a multi-byte UTF-8 byte sequence, is handled the same way as other non-characters, i.e., what do we do when a high or low surrogate codepoint is present and we’re to encode as UTF-8. I think the answer is we run the UTF-8 encode/decode algorithm, and whatever Unicode character code it creates is what it creates, and if that happens to come out as any of the non-characters (BOMs, surrogates, others perhaps), so be it. The topic is about Unicode non-characters, not specifically BOMs. The general topic is encoding/decoding our infoset Unicode character codes which have no real representation in the specified encoding. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0