- dfdl-wg - lists.ogf.org

Cancelled: OGF DFDL Working Group call
by Steve Hanson 17 Jun '16

17 Jun '16

Mike unable to make call

1 0

Re: [DFDL-WG] Fw: clarification or maybe restriction needed: delimited hexBinary question
by Steve Hanson 16 Jun '16

16 Jun '16

Mike Your example below should use dfdl:terminator not dfdl:separator but that does not materially change anything. I think it means that the data is raw bytes, no character decoding is happening at runtime. The delimiters are converted to bytes, and the scan is to find those bytes. Correct. However, that leaves off the ability to use delimiters with character class entities, like separator="年%WSP+;" since WSP+ one or more repeats of any of several whitespace characters. It can't just be converted to a sequence of bytes. The IBM DFDL scanner converts everything to bytes, including WSP+ and WSP* delimiters, even when matching characters, so the scan is purely byte-based. Originally this was so a scan handled raw byte entities and parent delimiters with different encodings. This was extended to handle the delimited binary elements erratum - the only difference is that bytes read from the input stream are not converted from an encoding. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 ----- Forwarded by Steve Hanson/UK/IBM on 16/06/2016 09:34 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 15/06/2016 21:41 Subject: [DFDL-WG] clarification or maybe restriction needed: delimited hexBinary question Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> In this example: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="年" dfdl:encoding="UTF-8" /> Note that 年 is 年 which requires 3 bytes in UTF-8: E5 B9 B4 The spec doesn't really say what happens here. It talks about scanning bcd and packed data for delimiters (I think the use case is TLOG format?), but doesn't really talk about what that means. I think it means that the data is raw bytes, no character decoding is happening at runtime. The delimiters are converted to bytes, and the scan is to find those bytes. However, that leaves off the ability to use delimiters with character class entities, like separator="年%WSP+;" since WSP+ one or more repeats of any of several whitespace characters. It can't just be converted to a sequence of bytes. I did not find a restriction in the DFDL spec on what the delimiters can contain when used for delimited binary data. E.g., only raw bytes or no char class entities. Perhaps there is such a restriction and I just didn't find it? If not, perhaps we need such a restriction, just to simplify implementor's lives, and avoid features nobody needs. Consider this: Easy implementation tricks like this don't generalize: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="å¹´" dfdl:encoding="iso-8859-1" /> The separator now contains the 3 bytes of the UTF-8 character, but as individual characters in iso-8859-1 where byte values and unicode codepoints are the same. It doesn't work because char class entities like WSP+ remain problematic. As a UTF-8 WSP+ allows repeats of any of the byte sequences corresponding to these unicode characters: U+0009-U+000D (Control characters) U+0020 SPACE U+0085 NEL U+00A0 NBSP U+1680 OGHAM SPACE MARK U+180E MONGOLIAN VOWEL SEPARATOR U+2000-U+200A (different sorts of spaces) U+2028 LSP U+2029 PSP U+202F NARROW NBSP U+205F MEDIUM MATHEMATICAL SPACE U+3000 IDEOGRAPHIC SPACE I can't express with separator, a repeating disjunction of the byte sequences corresponding to the above. Now, I think all this complexity adds no value for anyone. To avoid all this, I would propose these restrictions on delimited binary data 1) can only use SBCS encodings 2) no repeating char-class entities (WSP*, WSP+) are allowed. With those restrictions I believe the "trick" above of using an iso-8859-1 "decoder", and converting the delimiters into iso-8859-1 character sequences, can be made to work. Comments? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

clarification or maybe restriction needed: delimited hexBinary question
by Mike Beckerle 15 Jun '16

15 Jun '16

In this example: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="年" dfdl:encoding="UTF-8" /> Note that 年 is 年 which requires 3 bytes in UTF-8: E5 B9 B4 The spec doesn't really say what happens here. It talks about scanning bcd and packed data for delimiters (I think the use case is TLOG format?), but doesn't really talk about what that means. I think it means that the data is raw bytes, no character decoding is happening at runtime. The delimiters are converted to bytes, and the scan is to find those bytes. However, that leaves off the ability to use delimiters with character class entities, like separator="年%WSP+;" since WSP+ one or more repeats of any of several whitespace characters. It can't just be converted to a sequence of bytes. I did not find a restriction in the DFDL spec on what the delimiters can contain when used for delimited binary data. E.g., only raw bytes or no char class entities. Perhaps there is such a restriction and I just didn't find it? If not, perhaps we need such a restriction, just to simplify implementor's lives, and avoid features nobody needs. Consider this: Easy implementation tricks like this don't generalize: <xs:element name="foo" type="xs:hexBinary" dfdl:lengthKind="delimited" dfdl:separator="å¹´" dfdl:encoding="iso-8859-1" /> The separator now contains the 3 bytes of the UTF-8 character, but as individual characters in iso-8859-1 where byte values and unicode codepoints are the same. It doesn't work because char class entities like WSP+ remain problematic. As a UTF-8 WSP+ allows repeats of any of the byte sequences corresponding to these unicode characters: U+0009-U+000D (Control characters) U+0020 SPACE U+0085 NEL U+00A0 NBSP U+1680 OGHAM SPACE MARK U+180E MONGOLIAN VOWEL SEPARATOR U+2000-U+200A (different sorts of spaces) U+2028 LSP U+2029 PSP U+202F NARROW NBSP U+205F MEDIUM MATHEMATICAL SPACE U+3000 IDEOGRAPHIC SPACE I can't express with separator, a repeating disjunction of the byte sequences corresponding to the above. Now, I think all this complexity adds no value for anyone. To avoid all this, I would propose these restrictions on delimited binary data 1) can only use SBCS encodings 2) no repeating char-class entities (WSP*, WSP+) are allowed. With those restrictions I believe the "trick" above of using an iso-8859-1 "decoder", and converting the delimiters into iso-8859-1 character sequences, can be made to work. Comments? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

[DFDL]: Apply daffodil on S2G mission schemas
by Michele Zundo 07 Jun '16

07 Jun '16

Dear Steve/Mike, we have run some of our DFDL schema through the daffodil parser and identified 3 issues described in the attached document. The second is expected, but could you please review/comment the first and the last (in red in the text ? ) Thanks This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Michele

3 3

OGF DFDL WG Call Minutes 2016-06-07
by Steve Hanson 07 Jun '16

07 Jun '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13534?download= @Michele - please can you send to the WG a schema that shows your use of a regex in the path step of a DFDL expression ? Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2016-05-24
by Steve Hanson 07 Jun '16

07 Jun '16

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13534?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

unparser IS TOO a word
by Mike Beckerle 06 Jun '16

06 Jun '16

If wikipedia says it is, then it is. https://en.wikipedia.org/wiki/Unparser Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2016-05-24
by Steve Hanson 25 May '16

25 May '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13532?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot be empty string? textStandardNaNRep and friends cannot be empty string.
by Mike Beckerle 25 May '16

25 May '16

I believe the analogy is not quite right. For zero, having textStandardZeroRep "" means that the "ordinary" string for zero i.e., "0" means zero. There's an implied string conceptually. But for NaN and Inf there is no "ordinary" string to fall back on. They would have to cause an error. I think just prohibiting these from being "" is a better choice. ....mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, May 24, 2016 at 11:04 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike > > I came to write up the issue tracker for this and had a thought. Property > textStandardZeroRep says: > > *The empty string means that there is no special literal string for zero. > * > > I wondered if the equivalent is a better interpretation of empty string > for textStandardNanRep and textStandardInfinityRep ? > > Regards > > Steve Hanson > *IBM Integration Bus* > <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, > Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > mob:+44-7717-378890 > > > > From: Steve Hanson/UK/IBM > To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> > Date: 19/05/2016 21:53 > Subject: Re: [DFDL-WG] clarification: textBooleanTrue/FalseRep > cannot be empty string? textStandardNaNRep and friends cannot be empty > string. > ------------------------------ > > > Agree that textBooleanxxxRep properties should state empty string not > allowed. > > The textStandardNanRep and textStandardInfinityRep properties should state > empty string not allowed. The others you list already mention empty string. > > Regards > > Steve Hanson > *IBM Integration Bus* > <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, > Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > mob:+44-7717-378890 > > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> > Date: 19/05/2016 17:38 > Subject: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot > be empty string? textStandardNaNRep and friends cannot be empty string. > Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> > ------------------------------ > > > > > We don't stipulate that textBooleanTrueRep nor textBooleanFalseRep cannot > be empty string. > > We do stipulate that entity ES is not allowed. So I expect these > properties cannot be "" meaning the same thing that "%ES;" would mean. > Since we disallow the latter, we should disallow just "". > > Similarly, for textStandardGroupingSeparator, textStandardExponentRep, > textStandardNaNRep, and textStandardInfinityRep we specify that the string > literal value behaves as specified in textStandardDecimalSeparator, but we > don't specify that these cannot be empty string (which is specified in > textStandardDecimalSeparator, but not in the "Text Number Character > Restrictions" part of that property description that is referenced from > these other properties.) > > ES and WSP* are not allowed, so almost certainly empty string should not > be allowed. > > > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > *www.tresys.com* <http://www.tresys.com/> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the *OGF Intellectual Property Policy* > <http://www.ogf.org/About/abt_policies.php> > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >

1 0

Re: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot be empty string? textStandardNaNRep and friends cannot be empty string.
by Steve Hanson 24 May '16

24 May '16

Mike I came to write up the issue tracker for this and had a thought. Property textStandardZeroRep says: The empty string means that there is no special literal string for zero. I wondered if the equivalent is a better interpretation of empty string for textStandardNanRep and textStandardInfinityRep ? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 19/05/2016 21:53 Subject: Re: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot be empty string? textStandardNaNRep and friends cannot be empty string. Agree that textBooleanxxxRep properties should state empty string not allowed. The textStandardNanRep and textStandardInfinityRep properties should state empty string not allowed. The others you list already mention empty string. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 19/05/2016 17:38 Subject: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot be empty string? textStandardNaNRep and friends cannot be empty string. Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> We don't stipulate that textBooleanTrueRep nor textBooleanFalseRep cannot be empty string. We do stipulate that entity ES is not allowed. So I expect these properties cannot be "" meaning the same thing that "%ES;" would mean. Since we disallow the latter, we should disallow just "". Similarly, for textStandardGroupingSeparator, textStandardExponentRep, textStandardNaNRep, and textStandardInfinityRep we specify that the string literal value behaves as specified in textStandardDecimalSeparator, but we don't specify that these cannot be empty string (which is specified in textStandardDecimalSeparator, but not in the "Text Number Character Restrictions" part of that property description that is referenced from these other properties.) ES and WSP* are not allowed, so almost certainly empty string should not be allowed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0