December 2011 - dfdl-wg - lists.ogf.org

documentFinalTerminatorCanBeMissing (resending due to delivery problems)
by Steve Hanson 16 Dec '11

16 Dec '11

I came across a format today which allows fields to be fixed length or delimited on a per element basis. The delimiter is allowed to vary on a per element basis. This indicates that the delimiter would be naturally modelled as a terminator in DFDL, as opposed to a separator. However, the final terminator in the document must never appear. Not can be missing, but must be missing. This suggests that dfdl:documentFinalTerminatorCanBeMissing is not flexible enough, and needs to apply on unparsing, and maybe needs to be a 3 way property... - 'yes' -> can be omitted when parsing, output when unparsing - 'no' -> must be present when parsing, output when unparsing -, 'always'. -> must not be present when parsing, must not be output when unparsing Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 155: Validation on output (resending due to delivery problems)
by Steve Hanson 16 Dec '11

16 Dec '11

We closed this in the call this week. However my write up was missing an important word - 'augmented' - see below. Any issues with this let me know, otherwise I will update the minutes and re-post on grid forge. 155 Errata: Timing of validation on unparsing (All) 8/11. Next step is to talk to XML Schema WG. 15/11: Steve was a call with Sandy Gao on Fri 25th. 22/11: Call on Nov 25th delayed until Dec 2nd 29/11: Call now delayed until Dec 9th. Steve has sent mail to Sandy 6/12: Call this Friday 13/12: Closed. Call held with Sandy Gao. XML does not have this problem to anywhere near the same extent, an XML document and its infoset should validate the same (a couple of minor exceptions are whitespace and namespace declarations). Decided that due to the inherent possibility of creating a DFDL schema that can unparse a stream that can not be reparsed, and for commonality with XML, that an errata should be taken on the spec to change the description of validation on unparse to be in terms of the augmented infoset. Noted that a user of an implementation always has the option to re-parse a stream to check that it will parse and validate. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 155: Validation on output
by Steve Hanson 15 Dec '11

15 Dec '11

We closed this in the call this week. However my write up was missing an important word - 'augmented' - see below. Any issues with this let me know, otherwise I will update the minutes and re-post on grid forge. 155 Errata: Timing of validation on unparsing (All) 8/11. Next step is to talk to XML Schema WG. 15/11: Steve was a call with Sandy Gao on Fri 25th. 22/11: Call on Nov 25th delayed until Dec 2nd 29/11: Call now delayed until Dec 9th. Steve has sent mail to Sandy 6/12: Call this Friday 13/12: Closed. Call held with Sandy Gao. XML does not have this problem to anywhere near the same extent, an XML document and its infoset should validate the same (a couple of minor exceptions are whitespace and namespace declarations). Decided that due to the inherent possibility of creating a DFDL schema that can unparse a stream that can not be reparsed, and for commonality with XML, that an errata should be taken on the spec to change the description of validation on unparse to be in terms of the augmented infoset. Noted that a user of an implementation always has the option to re-parse a stream to check that it will parse and validate. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Inconsistent naming for text bidi properties
by Steve Hanson 15 Dec '11

15 Dec '11

I'd be interested in folks thoughts about this one. If textBidi is 'yes' then the following properties come in to play: - textBidiTextOrdering - textBidiOrientation - textBidiSymmetric - textBidiTextShaped - textBidiNumeralShape Two of the properties have 'text' appearing twice in the name. This seems superfluous to me. Any reason why these two properties have an extra 'text' ? I only noticed this because the IBM DFDL model has coded 'textBidiTextOrientation' which does not match the spec. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

documentFinalTerminatorCanBeMissing
by Steve Hanson 15 Dec '11

15 Dec '11

I came across a format today which allows fields to be fixed length or delimited on a per element basis. The delimiter is allowed to vary on a per element basis. This indicates that the delimiter would be naturally modelled as a terminator in DFDL, as opposed to a separator. However, the final terminator in the document must never appear. Not can be missing, but must be missing. This suggests that dfdl:documentFinalTerminatorCanBeMissing is not flexible enough, and instead needs to be a 3 way property - 'yes', 'no', 'always'. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Issue 156 - ICU fallback mappings - character encoding/decoding errors (version 2 - modified per call 2011-12-06)
by Mike Beckerle 15 Dec '11

15 Dec '11

*Issue 156 - ICU fallback mappings - character encoding/decoding errors* (Modified per workgroup discussion on 2011-12-06 - removed rationale and discussion, simplified to just the minimum. Note couple of important TBDs in here. Topics we forgot to discuss.) *Summary* DFDL currently does not have adequate capability to handle encoding and decoding errors. Language in the spec is incorrect/infeasible to implement. ICU provides mechanisms giving degree of control over this issue, the question is whether and how to embrace those mechanisms, or provide some other alternative solution. *Discussion* This language in section 4.1.2 about character set decoding/encoding just doesn't work: This first part is unacceptable because it fails to specify what happens when the decoding fails because of errors. It specifies what to do when there is no mapping to Unicode (which is, frankly, a very unlikely situation today) meaning a character is legally decoded, but then has no mapping. *During parsing, characters whose value is unknown or unrepresentable in ISO 10646 are replaced by the Unicode Replacement Character U+FFFD. * This second part also fails to work: *During unparsing, characters that are unrepresentable in the target encoding will be replaced by the replacement character for that encoding.* Sounds symmetric and expedient, but the problem is that some character encodings have no reserved replacement character, and we expect that DFDL users will need a variety of different choices for how to deal with characters that cannot be encoded. *Suggested Resolution: Summary * - DFDL property dfdl:inputEncodingErrorPolicy with values 'skip', 'error', 'replace' - DFDL property dfdl:outputEncodingErrorPolicy with values 'skip', 'error', 'replace' - DFDL annotation element. Example: <dfdl:encodingReplacementCharacter encoding="ASCII" character="%#x7c;"/> *For Parsing/Decoding Errors* There are two errors that can occur when decoding characters into Unicode/ISO 10646. 1. the data is broken - invalid byte sequences that don't match the definition of the encoding are encountered. 2. not enough bytes are found to make up the entire encoding of a character. That is, a fragment of a valid encoding is found. The behavior in these cases is controlled by dfdl:inputEncodingErrorPolicy. If 'replace', then the Unicode replacement character<http://en.wikipedia.org/wiki/Replacement_character>'�' (U+FFFD) is substituted for the offending bytes, one replacement character for each invalid byte, one replacement character for any fragment of an encoding. *(TBD: Assumptions to validate: I am assuming here that if there are 6 invalid bytes, none of which can validly be unit 1 of the encoding of any character, that ICU will call the error hook either (a) 6 times, or (b) once but notifying about all 6 bad units - but providing a way for the hook-writer to say they want to substitute 6 characters for the 6 units.* *I am also assuming in the end-of-data fragment case that the ICU hook gets called once for the fragment, not once per byte of the fragment.)* *(TBD: We did not discuss on the call on Dec 6th, the issue of errors in unicode encodings. While there are no encodings where a properly encoded character is unmapped to unicode, the unicode UTF encodings themselves can contains things that are errors**. Here's a short list of some things that can happen:* - *utf-16 and unpaired surrogate code-point* - *utf-16 and out-of-order surrogate code-point pair* - *utf-8 parsing and 3-byte encoding of a surrogate code-point is found* - *utf-8 unparsing and code-point of an isolated surrogate is to be encoded.* - *utf-8 decoding, and if you assemble the bits the usual way, you get a code point out of range (higher than 0x10FFFF)* - *utf-8 encoding, and code-point to encode is higher than 0x10FFFF. * - *utf-16 encoding utf16Width='fixed' and a surrogate code point is encountered* - *utf-16 byte-order-marks found not at the beginning of the data * *We have an option here to be 'tolerant' of unicode-encoding foibles. We can preserve isolated surrogates in a natural way if we wish. I believe many Unicode and UTF implementations tolerate these situations. For example the standard Java utf-8 decoder/encoder InputStreamReader and OutputStreamWriter, is tolerant of incorrectly paired and isolated surrogate code points in the Java string data. * *I do not know what ICU does in these cases, i.e., if it provides us enough flexibility to do whatever we want, or if it doesn't even detect some of these things as errors.**)* If 'skip' then the invalid byte sequences are dropped/ignored. No corresponding characters are created in the DFDL infoset. If 'error' then a processing error occurs. It is suggested that if a DFDL user wants to preserve information containing data where the encodings have these kinds of errors, that they model such data as xs:hexBinary, or as a xs:string, but using an encoding such as iso-8859-1 which preserves all bytes. Note for errata: The language in section 4.1.2 Item 5 about decoding data into infoset Unicode has to change of course as well. *Suggested Resolution - Unparsing/Encoding Errors* The following are kinds of errors when encoding characters: 1. no mapping provided by the encoding specification. 2. not enough room to output the entire encoding of the character (e.g., need 2 bytes for a DBCS, but only 1 byte remains in the available length. The behavior in these cases is controlled by dfdl:outputEncodingErrorPolicy. If the policy is 'error' then a processing error occurs (both case 1 and case 2) If the policy is 'skip' then the character is skipped. No character is encoded to be output for case 1, and no partial character is attempted in case 2. If the policy is 'replace' then the behavior is determined by the encoding specification, and by the dfdl:encodingReplacementCharacter annotation element. If provided, the dfdl:encodingReplacementCharacter annotation can appear anywhere that DFDL annotations are allowed. It has two attributes, which are 'encoding', and 'character'. The 'encoding' attribute specifies the encoding for which a replacement character is being specified. This takes the same values as the dfdl:encoding format property. The 'character' attribute specifies a DFDL literal string specifying exactly one character. This character is used as the replacement character for the specified encoding whenever that encoding is in use, and the dfdl:encodingReplacementCharacter annotation is in scope according to the usual scoping rules. There are these cases to consider when policy is 'replace' 1. there is no standard replacement character defined as part of the encoding specification, and there is no dfdl:encodingReplacementCharacter annotation element. 2. there is a standard replacement character defined as part of the encoding, and there is no dfdl:encodingReplacementCharacter annotation element. 3. there is a dfdl:encodingReplacementCharacter annotation element In case 1, since no replacement is possible, a processing error occurs. In case 2, the standard replacement character is used to replace the unmapped or error data. In case 3, the character specified by the dfdl:encodingReplacementCharacter annotation is used to replace the unmapped or error data. Note specifically, if the character set has a standard replacement character, the dfdl:encodingReplacementCharacter annotation can be used to override use of the standard replacement character. In these cases 2, and 3, it is still possible to be unable to output the replacement character if there is not enough room for its encoding. This situation is always a processing error. Note for errata: The language in section 4.1.2 about encoding data from infoset Unicode has to change as well.

3 8

Re: [DFDL-WG] Issue 156 - ICU fallback mappings - character encoding/decoding errors (version 3 - modified per call 2011-12-13)
by Mike Beckerle 14 Dec '11

14 Dec '11

Revised to include better characterization of ICU behavior based on some experiments I ran. Basically, ICU tolerates isolated surrogates in 16-bit strings, but the UTF-8 conversion requires surrogates to be properly paired. This is "standard" behavior, i.e., when encoding into UTF-8, any non-iso10646 character code or surrogate pairing is an error. Similarly when decoding. --------------------------------------------------------------------------- *Issue 156 - ICU fallback mappings - character encoding/decoding errors* (modified per email thread on standardized ICU substitution/replacement characters) (Modified per workgroup discussion on 2011-12-06 - removed rationale and discussion, simplified to just the minimum. Note couple of important TBDs in here. Topics we forgot to discuss.) (Modified per workgropu discussion on 2011-12-13 - did experiments to answer the TBDs in here.) * Summary* DFDL currently does not have adequate capability to handle encoding and decoding errors. Language in the spec is incorrect/infeasible to implement. ICU provides mechanisms giving degree of control over this issue, the question is whether and how to embrace those mechanisms, or provide some other alternative solution. * Discussion* This language in section 4.1.2 about character set decoding/encoding just doesn't work: This first part is unacceptable because it fails to specify what happens when the decoding fails because of data errors. *During parsing, characters whose value is unknown or unrepresentable in ISO 10646 are replaced by the Unicode Replacement Character U+FFFD. * This second part also is inadequate: *During unparsing, characters that are unrepresentable in the target encoding will be replaced by the replacement character for that encoding.* This needs a citation for where these replacement characters are specified. It also needs to specify what happens in certain error situations. *Suggested Resolution: Summary* - DFDL property dfdl:encodingErrorPolicy with values 'skip', 'error', 'replace' - Clarify that DFDL Infoset allows any 16-bit codepoint, not just those allowed by ISO 10646 *For Parsing/Decoding Errors* There are two errors that can occur when decoding characters into Unicode/ISO 10646. 1. the data is broken - invalid byte sequences that don't match the definition of the encoding are encountered. 2. not enough bytes are found to make up the entire encoding of a character. That is, a fragment of a valid encoding is found. The behavior in these cases is controlled by dfdl:inputEncodingErrorPolicy. If 'replace', then the Unicode *replacement character*<http://en.wikipedia.org/wiki/Replacement_character>'�' (U+FFFD) is substituted for the offending errors, one replacement character for each invalid encoding error. This can be one per byte if there is a series of all-illegal bytes, or it can be fewer replacement characters if a multi-byte sequence encoding a character has an error in the later bytes. For example, in UTF-8, if a 4-byte character has an error in the last byte of the 4, then a single replacement character is created. Conversely, if the first byte of a 4-byte character encoding has been corrupted, then one might get as many as 4 replacement characters. ** If 'skip' then the invalid byte sequences are dropped/ignored. No corresponding characters are created in the DFDL infoset. If 'error' then a processing error occurs. It is suggested that if a DFDL user wants to preserve information containing data where the encodings have these kinds of errors, that they model such data as xs:hexBinary, or as a xs:string, but using an encoding such as iso-8859-1 which preserves all bytes. *Suggested Resolution - Unparsing/Encoding Errors* The following are kinds of errors when encoding characters: 1. no mapping provided by the encoding specification. 2. not enough room to output the entire encoding of the character (e.g., need 2 bytes for a DBCS, but only 1 byte remains in the available length. The behavior in these cases is controlled by dfdl:encodingErrorPolicy. If the policy is 'error' then a processing error occurs. If the policy is 'skip' then the character is skipped. No character is encoded to be output for case 1, and no partial character is attempted in case 2. If the policy is 'replace' then the behavior is determined by the encoding specification. Each encoding has a replacement/substitution character specified by the ICU. These can be found conveniently in the ICU Converter Explorer.<http://demo.icu-project.org/icu-bin/convexp> This character is substituted for the unmapped character or the character that has too large an encoding (errors 1, and 2 above). It is a processing error if it is not possible to output the replacement character because there is not enough room for its representation. For example, for UTF-8 encoding, the standard substitution character is represented by 3 bytes. If there is no room for 3 bytes, then it is a processing error. It is a processing error if a character encoding does not provide a substitution/replacement character definition and one is needed because of dfdl:encodingErrorPolicy='replace'. (This would be rare, but could occur if a DFDL implementation allows many encodings beyond the minimum set.)

1 0

OGF DFDL WG Call Minutes 2011-12-13
by Steve Hanson 13 Dec '11

13 Dec '11

Please find minutes from the above call on GridForge at http://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman.… . Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2011-12-13
by Steve Hanson 13 Dec '11

13 Dec '11

Please find agenda attached for the above call on GridForge at http://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman.… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Invitation: OGF DFDL Working Group weekly call (10 Jan 15:00 GMT in IBM Hursley DE2K09)
by Steve Hanson 12 Dec '11

12 Dec '11

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0