- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Agenda 2011-07-12
by Steve Hanson 12 Jul '11

12 Jul '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… As per action 144 an errata to the spec has been created here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2011-07-05
by Steve Hanson 05 Jul '11

05 Jul '11

Please find minutes of the above meeting on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

DFDL WG action 142: Proposed resolution
by Steve Hanson 05 Jul '11

05 Jul '11

Following is the resolution from action 142 An errata document accompaniment to the DFDL 1.0 specification is under production and this will be added to it. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/07/2011 18:51 ----- 9.2. Grammar. Does not allow for a prefix length itself having a prefixed length, so does not match property description (ie, the change to permit this is draft 045 omitted to update the grammar). Proposed change to grammar in 9.2: PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent This is sufficient to allow the grammar to describe the needed “one more level” of prefix (as required for modeling an ASN.1 format) without allowing recursion. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL WG action 139: Proposed resolution
by Steve Hanson 05 Jul '11

05 Jul '11

Following is the resolution from action 139. An errata document accompaniment to the DFDL 1.0 specification is under production and this will be added to it. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/07/2011 18:26 ----- WG resolution for action 139 a) In-scope delimiters Clarification: Using lengthKind 'explicit', 'implicit', 'prefixed' or 'pattern' means that delimiter scanning is turned off and in-scope delimiters are not looked for within or between elements. b) Policing a physical constraint without terminating a parse The example cited was using a dfdl:assert to police a physical length constraint when lengthKind is 'delimited'. Addition: - A new failure type is introduced, 'Recoverable Error', after which the parser will continue. - Importantly, it does not cause backtracking to take place when speculating. - It can be raised via a new enum attribute on dfdl:assert called 'failureType' - see below. - An error occurring during evaluation of a dfdl:assert remains a Processing Error. - All existing stated Processing Errors remain as such. - Discriminators remain unchanged. - The issuing of Recoverable Errors is independent of whether validation is enabled. Property Name Description failureType Enum (optional) Valid values are 'processingError', 'recoverableError'. Default value is 'processingError'. Specifies the type of failure that occurs when the dfdl:assert is unsuccessful. When 'processingError', a processing error is raised. When 'recoverableError', a recoverable error is raised. Annotation: dfdl:assert The WG believe the new failure type is genuinely useful going forward as it allows the parser or unparser to continue in situations where it was unable to do so. The WG considered extending Validation Error to cover this, but the spec is quite clear that these are logical checks performed on the infoset and the behaviour of the DFDL processor is unspecified, so it seemed best not to change this and to introduce a new failure type. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2011-07-05
by Steve Hanson 04 Jul '11

04 Jul '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2011-06-28
by Steve Hanson 28 Jun '11

28 Jun '11

Please find minutes of the above meeting on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: OGF DFDL WG Call Agenda 2011-06-28 - resending
by Steve Hanson 28 Jun '11

28 Jun '11

Due to an issue with CutePDF truncating text, here is the agenda for today. 1. Spec issues from IBM As implementation has progressed, a number of issues have been spotted in the DFDL 1.0 spec by the IBM implementation team. List so far for resolution. 6.3. Bullet for logical value. State that the string must obey the lexical representation of the type. 6.3.1. Explicit statement of white space rules is missing. Just white space for a DFDL string literal property is an error. When a property can be a list of string literals, white space between values is collapsed. 7.7. What is the scope of a variable with respect to included/imported xsds? Can I refer from xsd A to a variable defined in parent xsd B? 9.2. Grammar. Does not allow for a prefix length itself having a prefixed length, so does not match property description (ie, the change to permit this is draft 045 omitted to update the grammar). 13. Be explicit on the DFDL literals allowed for text properties. See separate e-mail. 13.6. When textNumberRep is zoned, property description should state that base is assumed to be 10. 13.6. State that dfdl:ignoreCase not honoured by textStandardExponentCharacter, textStandardInfinityRep, textStandardNanRep as ICU does not offer that flexibility. 13.6 Allow multiple characters for textStandardExponentCharacter to handle reps like 1.23x10^4 as ICU allows that. Note that property name will have to change, eg, to textStandardExponent.. 13.6. Change meaning of textNumberCheckPolicy 'lax' to align more closely with ICU: "If ‘lax' and dfdl:textNumberRep is 'standard' then grouping separators are ignored, leading and trailing whitespace is ignored, leading zeros are ignored." 13.6. Disallow the use of empty string for textStandardExponentCharacter. State property must be set if the pattern contains an 'E' 13.6. Disallow the use of empty string for textStandardDecimalSeparator, State property must be set if the pattern contains a '.' 13.6 Disallow the use of empty string for textStandardGroupingSeparator, State property must be set if the pattern contains a ',' 13.6. Allow decimal separator to be a List of DFDL String Literals or a DFDL expression. This allows modelling of EDIFACT where user can choose decimal separator in ISA header but '.' is also allowed. 13.6. When textNumberPadCharacter is '0' which it commonly is, a value of say '00000' will get trimmed to the empty string, whereas the intent would surely be to trim to '0'. Need a rule for this. 13.6.Should textStandardDecimalSeparator ignored when the logical type is not decimal/float/double? 13.6.1.1. Erroneously mentions sigDigits in reference to the BNF for textNumberPattern. 13.6.1.1. Formatting. Uses terms 'minimum/maximum integer/fraction digits' but does not define them. 13.9. Should state that textBooleanTrue/FalseRep properties are used after trimming when parsing, and before padding when unparsing. If lengthKind is explicit or implicit and either textPadKind or textTrimKind is none then the properties must have the same length else schema definition error. 13.16. Nil literal character. When representation is binary, the encoding property is not used, which implies that a nil value must only contain DFDL hex literals (%rXX;) 13.16. Clarify nilKind 'literalCharacter' and 'literalValue' and whether nil test is applied before or after trimming. 15. Using a discriminator to resolve each branch of a choice with a very large number of branches has a performance impact as each branch must be evaluated in turn. In cases where the name of the branch element is known or derivable in advance (ie, each discriminator expression is of the form {..\tag eq "elem1"} or similar) then consider adding a property to dfdl:choice that provides an expression that returns a QName, being the name of the branch element. Note: IBM MRM has something like this. 16.1. Revise entire section. 16.2. State it is a processing error if the stop value is missing when parsing. 2. Spec issues from Mike Beckerle A number of issues have been spotted by Mike. Some have been resolved by action 139 below, but others remain. See separate e-mail. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/06/2011 14:46 ----- From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 28/06/2011 13:50 Subject: RE: [DFDL-WG] OGF DFDL WG Call Agenda 2011-06-28 Steve, When I download this agenda PDF file, it’s all cut off on the right edge of the page, making it hard to figure out what some of these line items are about. Something is up with the way you converted this teamroom page to a pdf. If this was paper, I’d suspect the metric vs. US paper size issue, but this is all bits…. J …mikeb From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Steve Hanson Sent: Monday, June 27, 2011 12:18 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] OGF DFDL WG Call Agenda 2011-06-28 Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Spec issues from IBM: Restrictions on properties of type 'DFDL string literal'
by Steve Hanson 27 Jun '11

27 Jun '11

For discussion on DFDL WG call. Please find proposal from IBM to clarify the allowable characters that be used in some of the DFDL properties that are defined as type 'DFDL String Literal'. It is proposed that the properties in question do not need the full power of DFDL String Literal. escaping ( single character ) escapeCharacter escapeEscapeCharacter escapeBlockStart escapeBlockEnd Property value must resolve to a single character DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed padding ( single character or byte ) textStringPadCharacter textNumberPadCharacter textBooleanPadCharacter textCalendarPadCharacter Property value must resolve to a single character or a single byte value DFDL character entities are allowed The raw byte entity ( %#r ) is allowed subject to the restrictions already documented for these properties DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Number separators ( single character ) textStandardDecimalSeparator textStandardGroupingSeparator textStandardExponentCharacter Property value must resolve to a single character DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Number representations ( multiple characters ) textStandardInfinityRep textStandardNanRep textStandardZeroRep DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Boolean representations ( multiple characters ) textBooleanTrueRep textBooleanFalseRep DFDL character entities are allowed The raw byte entity ( %#r ) is not allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are not allowed Nil representations ( multiple characters/bytes ) nilValue DFDL character entities are allowed The raw byte entity ( %#r ) is allowed DFDL Character classes ( NL, WSP, WSP+, WSP*, ES ) are allowed Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Spec issues from Mike Beckerle
by Steve Hanson 27 Jun '11

27 Jun '11

For discussion on DFDL WG calls. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 27/06/2011 17:26 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 15/06/2011 01:59 Subject: Issues to add to work items list Steve, Below is the list I have so far of items in the spec that go beyond just typos - where re-wording is required or advisable. All but the last of these is tied up with the length & delimiters issue, but separate of the "known to exist" vs. "missing" topic. I've not bothered to provide anything about the section on "known to exist" and such. That section is already the discussion of a work item/topic. The last one is just a nit about BOMs. ...mikeb -------------------------------------------------------------------------------------------- 5.2.2 MinLength, MaxLength These facets are used: When dfdl:lengthKind=”implicit”. In that case the length is given by the value of xs:maxLength. In this case minLength if specified is required to be equal to maxLength (schema definition error otherwise). For validation of variable length string elements. It is a processing error when a fixed-length string is found to have a number of characters not equal to the fixed number. For example, if a fixed-length string also has delimiters we might be able to successfully separate it from the surrounding elements depending on the delimiter specifications; however, if the length of the fixed-length string is not equal to the number specified as the fixed length then it is a processing error (not simply a validation error).[MB1] [MB1]Contradicts statement that scanning for delimiters is off.(Discussed where dfdl:lengthKind=’explicit’ is described) What is a fixed length string? Clearly if it has lengthKind=”explicit” it is fixed length. What if it has lengthKind=”implicit” and maxLength=”10”. Is that a fixed length string which shuts off delimiter scanning also? If so then this paragraph is erroneous and misleading. <SMH>Yes, Tim and I have noted that this paragraph needs revising depending on the outcome of action 139 <SMH> -------------------------------------------------------------------------------------- 9.2 DFDL Syntax Grammar Change to introduce concept of EnclosedItem or ChildItem (I used EnclosedItem below): Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedItem [ Separator EnclosedItem ]* PostfixSeparator ] FinalUnusedRegion EnclosedItem = Element | Array | ComplexContent[MB1] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedItem ] FinalUnusedRegion[MB2] [MB1]Refactored to share the EnclosedItem concept to Choices also. Should perhaps be named ChildItem. This is useful when discussing how parsing, defaulting, etc. work as well. <SMH>Seems fine, it is effectively just a renaming<SMH> --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Table 14 Implicit Alignment in bits Note: Specifying the implicit alignment in bits does not imply that dfdl:lengthUnits 'bits' can be specified for all simple types.[MB1] [MB1]I do not understand this comment. What exactly is the restriction? <SMH>It is really saying that alignmentUnits and lengthUnits are independent and have their own rules for when they are applicable. <SMH> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- lengthKind Enum Controls how the representation length of the component is determined. Valid values are: 'explicit', 'delimited', 'prefixed', 'implicit', 'pattern', 'endOfParent' A full description of each enumeration is given in the later sections. 'explicit' means the length of the item is given by the dfdl:length property 'delimited' means the item is delimited by a terminator or separator[MB1] ‘prefixed’ means the length of the item is given by an immediately preceding prefix field specified using prefixLengthType. ‘implicit means the length is to be determined in terms of the type of the element and its schema-specified properties if any. ‘pattern’ means the length of the item is given by a regular expression specified using the dfdl:lengthPattern property. ‘endOfParent’ means that the item is terminated by the termination of the containing construct. Annotation: dfdl:element, dfdl:simpleType [MB1]To me this is a very strong statement. It means that an outside-in parse is allowed where for a sequence, we can scan and determine its end, and then parse the children. It requires that “scan” is a well-defined concept for the contents of the sequence. It means there can be nothing inside which requires the suspension of scanning. It means contained elements that have length explicit and representation binary, are simply not allowed. <SMH>Not entirely true, we allow some binary types to have delimited lengthKind (eg, BCD and Packed Decimals). There are formats out there that require this. <SMH> To me it also means you cannot change the character set encoding, or have a contained element that itself uses an overlapping set of delimiters with the enclosing group’s delimiters. – Is that going too far? – Delimited is like pattern. It restricts what is in the data stream substantially. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The rules for resolving ambiguity between delimiters are: 1. When two delimiters have a common prefix, the longest delimiter has precedence. 2. When two delimiters have exactly the same value, the innermost (most deeply nested) delimiter has precedence. 3. When the separator and terminator on a group have the same value, the separator has precedence.[MB1] [MB1]By precedence, this must mean it is tried first, but the parser might backtrack and assume it to be a terminator instead. This seems problematic to me. I’d like to either rule this out and say they can’t be the same, or see a use case where this is needed, a backtracking parser can parse it, and there’s no more reasonable way to structure the schema. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 12.3.5.1 Pattern-Based Lengths - Scanability Any element (complex, simple text, simple binary) may have a dfdl:lengthKind 'pattern' as long as the bytes in the content region of the element are legal in the stated encoding of that element. Where a complex element has children with binary representation in practice this means an 8-bit ASCII encoding. [MB1] [MB1]Not necessarily ASCII. An 8-bit encoding such that any byte value is valid is the real requirement. The point is no single byte value is invalid, and no combinations of adjacent byte values are invalid so that any binary data won’t trip up a character conversion and subsequent scan. Hmmm. The 8-bit character set used must have a transformation into Unicode which is bijective and information preserving. I.e., a unique Unicode character for each code point, and no “invalid” chracters which have no corresponding Unicode value. However, Ascii-based sets like 8859-1 are not strictly speaking, required. ----------------------------------------------------------------------------------------------------------------------------------------------- 12.3.7.1.3 Byte Order Mark If a byte-order mark codepoint appears at the start of a UTF-8, [MB1] UTF-16 or UTF-32 encoded string then the byte-order mark will be included as part of the string payload[1]. That is, for the UTF-8, UTF-16 and UTF-32 character encodings, a byte-order-mark codepoint is treated as a character of the string in DFDL and contributes to the length. A way of eliminating the byte-order mark so that it does not end up in the infoset is that the byte-order mark can be modeled as a separate element before the string. This BOM element can be either required or optional depending on whether one is expected or optional at the beginning of the string. [1] Byte-order marks are explicitly stated to be “not characters” in the Unicode standard. [MB1]No such thing as a BOM codepoint in a UTF-8 string. A UTF-8 byte sequence might encode the character code for a BOM, but this would be a meaningless inclusion of a BOM character code in a context where it will never be interpreted. I suggest that we drop the term UTF-8 here, and BOM’s that get encoded when they are interpreted as character codes, and translated by the UTF-8 encoding algorithm into a multi-byte UTF-8 byte sequence, is handled the same way as other non-characters, i.e., what do we do when a high or low surrogate codepoint is present and we’re to encode as UTF-8. I think the answer is we run the UTF-8 encode/decode algorithm, and whatever Unicode character code it creates is what it creates, and if that happens to come out as any of the non-characters (BOMs, surrogates, others perhaps), so be it. The topic is about Unicode non-characters, not specifically BOMs. The general topic is encoding/decoding our infoset Unicode character codes which have no real representation in the specified encoding. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2011-06-28
by Steve Hanson 27 Jun '11

27 Jun '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0