- dfdl-wg - lists.ogf.org

Action 242: units 'characters' and encodingErrorPolicy 'error' - valueLength and contentLength functions
by Mike Beckerle 19 Jul '16

19 Jul '16

In analysis of the valueLength and contentLength email discussion threads, we're not converged on whether these functions when measured in units 'characters', are allowed to compute the length without checking for decode errors when (a) the encoding is fixed width - so a character unit is just an alias for some number of bytes (b) encodingErrorPolicy='error'. I think we need to clarify that just because encodingErrorPolicy is 'error', doesn't mean all data in scope of that will be scanned to be sure there are no decode errors. Other features with similar issues are regex pattern asserts. In this case the regex is matching against text, and that match might or might not encounter a decode error, but the entire scope of data it's talking about is NOT going to get converted just to insure no chance of a decode error. Can we go so far as to say DFDL implementations can (or maybe even must) optimize performance by avoiding character decoding when possible? This means that some character decode errors may not be detected even though dfdl:encodingErrorPolicy is 'error'. I would suggest the language should say that only character decoding that results in a character being placed into the DFDL Infoset is guaranteed to cause an error should that character not be decodable. (Similarly for unparsing, it is only if we actually unparse an unmapped character from the infoset to the output stream, that a encoding error is guaranteed to occur. I might not have worded it well, but I think the above is what we're trying to allow - implementations are free to exploit fixed character width, and just jumping around the bytes and not decoding/encoding anything - whenever they can, because we all expect, and think our users will expect, this level of performance. If you use UTF-8, certainly a common thing,( or any other variable-width encoding) then you are likely to get some cases where implementations say "can't do that with utf-8" because it's just a limitation of the implementation. One may also get cases where switching from utf-16 to utf-8 for your data changes the behavior of the processor because utf-16 wouldn't detect some decoding errors because of fixed width, whereas when using utf-8 will have to measure length in characters by decoding and so will detect the error. Basically, we've tried to make a consistent position around a messy area: The schema contains a complex type element which is a mixture of character encodings, binary stuff, and may have decode errors in the corresponding data stream (or characters in the infoset that have no mapping into the representation of the encoding - for unparsing). Given this mess, there are ways that a schema can look at it, and foolishly-perhaps, treat it as characters. Asserts with test patterns are one. Specified length with units of 'characters' is another, and the dfdl:contentLength and dfdl:valueLength functions is yet another, since one can specify the units 'characters' as that 2nd argument. The only consistent position is that dfdl:contentLength or dfdl:valueLength of an element, with units 'character' does NOT necessarily imply those characters will be decoded/encoded. Anyway, it's moot in the scenario where an earlier OVC wants the dfdl:contentLength of something later,.... if when we unparse the thing later, we get the decode error at that point. We're just getting what is arguably an incorrect OVC computation, but followed by a later decode error. .... Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Action 242 - complexType with interior alignment - valueLength and contentLength
by Mike Beckerle 19 Jul '16

19 Jul '16

I have reexamined this material from prior discussions in this group, and considering the experience of implementing this in Daffodil. We're not yet converged on whether the dfdl:contentLength of a complex element, is allowed to be a function of its starting position due to variable-sized alignment regions interior to the complex representation. This creates a situation where a element with dfdl:outputValueCalc can have variable-length itself, and hence a complex element after it, that it forward references, can't compute its content's length, because it can't know the start position due to the variable length OVC element preceding it. (Got that?) All I can say is ..... gaaaak. This scenario is possible due to the way that features compose in DFDL, which is trying to be fairly orthogonal to each other. Alignment causes representation lengths to break this orthogonality because length now depends on the length of previous things. Our choices: 1) yup must detect this crazy deadlock. Runtime SDE. 2) rule it out by disallowing OVC elements that call dfdl:contentLength which reference complex elements that have interior alignment, to be variable length themselves. It's an SDE. (Gaaaaak) 3) rule it out by saying OVC elements can never be variable length (... .but I think I've seen prefix lengths in textual data before, and those are naturally variable length... and might not have been adjacent, so not lengthKind 'prefixed', but needs OVC..... double Gaaaaak) 4) punt it down the road by saying it's 'implementation dependent' behavior. Still have to say exactly what it is that is implementation behavior, but this could be a paragraph/subsection in the description of dfdl:contentLength, dfdl:valueLength, and a reference from the dfdl:outputValueCalc section. Better options? Ideas? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 06 Jul '16

06 Jul '16

steve away

1 0

Re: [DFDL-WG] Fw: clarification or maybe restriction needed: delimited hexBinary question
by Steve Hanson 05 Jul '16

05 Jul '16

WG agreed that an implementation note be added to the relevant section to explain the implications for the delimiter scanner if an implementation supports the 'delimited binary' or 'raw byte entity in delimiters' optional features. https://redmine.ogf.org/issues/315 Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: mbeckerle.dfdl(a)gmail.com Cc: DFDL-WG <dfdl-wg(a)ogf.org>, Andrew Edwards/UK/IBM@IBMGB Date: 16/06/2016 16:02 Subject: Re: Fw: [DFDL-WG] clarification or maybe restriction needed: delimited hexBinary question Mike Your example below should use dfdl:terminator not dfdl:separator but that does not materially change anything. I think it means that the data is raw bytes, no character decoding is happening at runtime. The delimiters are converted to bytes, and the scan is to find those bytes. Correct. However, that leaves off the ability to use delimiters with character class entities, like separator="年%WSP+;" since WSP+ one or more repeats of any of several whitespace characters. It can't just be converted to a sequence of bytes. The IBM DFDL scanner converts everything to bytes, including WSP+ and WSP* delimiters, even when matching characters, so the scan is purely byte-based. Originally this was so a scan handled raw byte entities and parent delimiters with different encodings. This was extended to handle the delimited binary elements erratum - the only difference is that bytes read from the input stream are not converted from an encoding. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 ----- Forwarded by Steve Hanson/UK/IBM on 16/06/2016 09:34 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 15/06/2016 21:41 Subject: [DFDL-WG] clarification or maybe restriction needed: delimited hexBinary question Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> In this example: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="年" dfdl:encoding="UTF-8" /> Note that 年 is 年 which requires 3 bytes in UTF-8: E5 B9 B4 The spec doesn't really say what happens here. It talks about scanning bcd and packed data for delimiters (I think the use case is TLOG format?), but doesn't really talk about what that means. I think it means that the data is raw bytes, no character decoding is happening at runtime. The delimiters are converted to bytes, and the scan is to find those bytes. However, that leaves off the ability to use delimiters with character class entities, like separator="年%WSP+;" since WSP+ one or more repeats of any of several whitespace characters. It can't just be converted to a sequence of bytes. I did not find a restriction in the DFDL spec on what the delimiters can contain when used for delimited binary data. E.g., only raw bytes or no char class entities. Perhaps there is such a restriction and I just didn't find it? If not, perhaps we need such a restriction, just to simplify implementor's lives, and avoid features nobody needs. Consider this: Easy implementation tricks like this don't generalize: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="å¹´" dfdl:encoding="iso-8859-1" /> The separator now contains the 3 bytes of the UTF-8 character, but as individual characters in iso-8859-1 where byte values and unicode codepoints are the same. It doesn't work because char class entities like WSP+ remain problematic. As a UTF-8 WSP+ allows repeats of any of the byte sequences corresponding to these unicode characters: U+0009-U+000D (Control characters) U+0020 SPACE U+0085 NEL U+00A0 NBSP U+1680 OGHAM SPACE MARK U+180E MONGOLIAN VOWEL SEPARATOR U+2000-U+200A (different sorts of spaces) U+2028 LSP U+2029 PSP U+202F NARROW NBSP U+205F MEDIUM MATHEMATICAL SPACE U+3000 IDEOGRAPHIC SPACE I can't express with separator, a repeating disjunction of the byte sequences corresponding to the above. Now, I think all this complexity adds no value for anyone. To avoid all this, I would propose these restrictions on delimited binary data 1) can only use SBCS encodings 2) no repeating char-class entities (WSP*, WSP+) are allowed. With those restrictions I believe the "trick" above of using an iso-8859-1 "decoder", and converting the delimiters into iso-8859-1 character sequences, can be made to work. Comments? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2016-07-05
by Steve Hanson 05 Jul '16

05 Jul '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13537?download= @Michele - please can you send to the WG a schema that shows your use of a regex in the path step of a DFDL expression ? Next call Aug 2nd Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2016-07-05
by Steve Hanson 04 Jul '16

04 Jul '16

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13537?download= Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 17 Jun '16

17 Jun '16

Mike unable to make call

1 0

Re: [DFDL-WG] Fw: clarification or maybe restriction needed: delimited hexBinary question
by Steve Hanson 16 Jun '16

16 Jun '16

Mike Your example below should use dfdl:terminator not dfdl:separator but that does not materially change anything. I think it means that the data is raw bytes, no character decoding is happening at runtime. The delimiters are converted to bytes, and the scan is to find those bytes. Correct. However, that leaves off the ability to use delimiters with character class entities, like separator="年%WSP+;" since WSP+ one or more repeats of any of several whitespace characters. It can't just be converted to a sequence of bytes. The IBM DFDL scanner converts everything to bytes, including WSP+ and WSP* delimiters, even when matching characters, so the scan is purely byte-based. Originally this was so a scan handled raw byte entities and parent delimiters with different encodings. This was extended to handle the delimited binary elements erratum - the only difference is that bytes read from the input stream are not converted from an encoding. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 ----- Forwarded by Steve Hanson/UK/IBM on 16/06/2016 09:34 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 15/06/2016 21:41 Subject: [DFDL-WG] clarification or maybe restriction needed: delimited hexBinary question Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> In this example: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="年" dfdl:encoding="UTF-8" /> Note that 年 is 年 which requires 3 bytes in UTF-8: E5 B9 B4 The spec doesn't really say what happens here. It talks about scanning bcd and packed data for delimiters (I think the use case is TLOG format?), but doesn't really talk about what that means. I think it means that the data is raw bytes, no character decoding is happening at runtime. The delimiters are converted to bytes, and the scan is to find those bytes. However, that leaves off the ability to use delimiters with character class entities, like separator="年%WSP+;" since WSP+ one or more repeats of any of several whitespace characters. It can't just be converted to a sequence of bytes. I did not find a restriction in the DFDL spec on what the delimiters can contain when used for delimited binary data. E.g., only raw bytes or no char class entities. Perhaps there is such a restriction and I just didn't find it? If not, perhaps we need such a restriction, just to simplify implementor's lives, and avoid features nobody needs. Consider this: Easy implementation tricks like this don't generalize: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="å¹´" dfdl:encoding="iso-8859-1" /> The separator now contains the 3 bytes of the UTF-8 character, but as individual characters in iso-8859-1 where byte values and unicode codepoints are the same. It doesn't work because char class entities like WSP+ remain problematic. As a UTF-8 WSP+ allows repeats of any of the byte sequences corresponding to these unicode characters: U+0009-U+000D (Control characters) U+0020 SPACE U+0085 NEL U+00A0 NBSP U+1680 OGHAM SPACE MARK U+180E MONGOLIAN VOWEL SEPARATOR U+2000-U+200A (different sorts of spaces) U+2028 LSP U+2029 PSP U+202F NARROW NBSP U+205F MEDIUM MATHEMATICAL SPACE U+3000 IDEOGRAPHIC SPACE I can't express with separator, a repeating disjunction of the byte sequences corresponding to the above. Now, I think all this complexity adds no value for anyone. To avoid all this, I would propose these restrictions on delimited binary data 1) can only use SBCS encodings 2) no repeating char-class entities (WSP*, WSP+) are allowed. With those restrictions I believe the "trick" above of using an iso-8859-1 "decoder", and converting the delimiters into iso-8859-1 character sequences, can be made to work. Comments? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

clarification or maybe restriction needed: delimited hexBinary question
by Mike Beckerle 15 Jun '16

15 Jun '16

In this example: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="年" dfdl:encoding="UTF-8" /> Note that 年 is 年 which requires 3 bytes in UTF-8: E5 B9 B4 The spec doesn't really say what happens here. It talks about scanning bcd and packed data for delimiters (I think the use case is TLOG format?), but doesn't really talk about what that means. I think it means that the data is raw bytes, no character decoding is happening at runtime. The delimiters are converted to bytes, and the scan is to find those bytes. However, that leaves off the ability to use delimiters with character class entities, like separator="年%WSP+;" since WSP+ one or more repeats of any of several whitespace characters. It can't just be converted to a sequence of bytes. I did not find a restriction in the DFDL spec on what the delimiters can contain when used for delimited binary data. E.g., only raw bytes or no char class entities. Perhaps there is such a restriction and I just didn't find it? If not, perhaps we need such a restriction, just to simplify implementor's lives, and avoid features nobody needs. Consider this: Easy implementation tricks like this don't generalize: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="å¹´" dfdl:encoding="iso-8859-1" /> The separator now contains the 3 bytes of the UTF-8 character, but as individual characters in iso-8859-1 where byte values and unicode codepoints are the same. It doesn't work because char class entities like WSP+ remain problematic. As a UTF-8 WSP+ allows repeats of any of the byte sequences corresponding to these unicode characters: U+0009-U+000D (Control characters) U+0020 SPACE U+0085 NEL U+00A0 NBSP U+1680 OGHAM SPACE MARK U+180E MONGOLIAN VOWEL SEPARATOR U+2000-U+200A (different sorts of spaces) U+2028 LSP U+2029 PSP U+202F NARROW NBSP U+205F MEDIUM MATHEMATICAL SPACE U+3000 IDEOGRAPHIC SPACE I can't express with separator, a repeating disjunction of the byte sequences corresponding to the above. Now, I think all this complexity adds no value for anyone. To avoid all this, I would propose these restrictions on delimited binary data 1) can only use SBCS encodings 2) no repeating char-class entities (WSP*, WSP+) are allowed. With those restrictions I believe the "trick" above of using an iso-8859-1 "decoder", and converting the delimiters into iso-8859-1 character sequences, can be made to work. Comments? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

[DFDL]: Apply daffodil on S2G mission schemas
by Michele Zundo 07 Jun '16

07 Jun '16

Dear Steve/Mike, we have run some of our DFDL schema through the daffodil parser and identified 3 issues described in the attached document. The second is expected, but could you please review/comment the first and the last (in red in the text ? ) Thanks This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Michele

3 3