May 2021 - dfdl-wg - lists.ogf.org

example for extraEscapedCharacters
by Mike Beckerle 25 May '21

25 May '21

I've been searching our email history for the motivating example for extraEscapedCharacters property. I can't find it. Does anyone recall this? I vaguely recall a format where we had CSV-like data that already contained escaped characters like "foo\nbar\,baz,next,next" Where the \n is just part of the data, regardless of the fact that \ is also the escape character. Normal rules would remove the "\" since it is the escape character, and that would result in corrupting such data. But it makes no sense to have extraEscapedCharacter="n" since n is just a letter and could appear without the escape. It would be wrong to escape every letter n. So this example isn't sufficient. I see no way to cope with this particular case other than lengthKind='pattern' which is the usual way to cope with complexities like this. But I know there was a motivating example for extraEscapedCharacters property. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Fw: Spec question about extraEscapedCharacters
by Steve Hanson 25 May '21

25 May '21

Mike Fyi, from 2009. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday ----- Forwarded by Steve Hanson/UK/IBM on 25/05/2021 08:56 ----- From: Steve Hanson/UK/IBM To: Alan Powell/UK/IBM@IBMGB Cc: Dragan Besevic/Boca Raton/IBM@IBMUS, Stephanie Fetzer/Charlotte/IBM@IBMUS Date: 04/11/2009 11:14 Subject: Re: Spec question about extraEscapedCharacters Alan On parsing, MRM just looks for the escape character (like DFDL). On writing, MRM needs to know what to escape and the reserved characters list is used for that (markup is not automatically included, as you say). So why does DFDL need extraEscapedCharacters property? Let's say I serialise an infoset corresponding to a structure. That will escape any in-scope markup characters in the data. But if I then include that serialised data as a BLOB is some envelope structure that also has terminating markup, I won't have escaped any of the envelope's markup characters in my BLOB. To do that I need extraEscapedCharacters property. I think the spec needs clarifying to say that extraEscapedCharacters is an output only control. Regards Steve Hanson Programming Model Architect, WebSphere Message Brokers, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 From: Alan Powell/UK/IBM To: Stephanie Fetzer/Charlotte/IBM@IBMUS, Steve Hanson/UK/IBM@IBMGB Cc: Dragan Besevic/Boca Raton/IBM@IBMUS Date: 03/11/2009 10:48 Subject: Re: Spec question about extraEscapedCharacters Stephanie extraEscapedCharacters comes from the WMB Reserved Characters property. It isn't exactly the same as WMB doesn't automatically look for markup. Also the documentation says that Reserved Characters aren't used on parsing which doesn't seem consistent. Steve do you know why? As currently specified your example wouldn't be a parsing error but may cause the data to be interpreted incorrectly if ^ is in fact markup. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Stephanie Fetzer/Charlotte/IBM@IBMUS To: Alan Powell/UK/IBM@IBMGB Cc: Dragan Besevic/Boca Raton/IBM@IBMUS Date: 02/11/2009 21:05 Subject: Spec question about extraEscapedCharacters Alan: Real quick question about extraEscapedCharacters. The current definition of the term is: String A space separated list of single characters that must be escaped in addition to markup. Annotation: dfdl: escapeScheme So - my interpretation is that is someone wants to enforce...even though the ^ is not an initiator, terminator, delimiter, or any other types or markup in this document - I want to force the sender of the data to always escape it. So extraEscapedCharacters="^" and the data to be parsed looks like: "NAME#ADDRESS#PHONENUMB^ER" That would be an error (unless the B is an escape char). We would expect the ^ to always be escaped in the data. So even though we don't need it to be escaped from a parsing standpoint we would expect it to be a parsing error? Or is the property merely a suggestion on parsing? Any hints that you can provide on the reasoning for this property would be appreciated. Cheers, -Steph WebSphere Transformation Extender Industry Packs - Software Engineer Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 11 May '21

11 May '21

1 0

Clarification: priority of delimiters vs. escape chars
by Mike Beckerle 04 May '21

04 May '21

Consider two elements in a sequence, dfdl:separator="/ // ///" with escapeCharacter="/" and escapeEscapeCharacter="/" I did not spot language in the spec that makes it clear what gets priority, interpreting a character as an escape char or escape-escape char, or interpreting it as a delimiter. Consider data "foo///bar". 1. I could interpret that as escapeEscape, escape, and minimum length separator "/" 2. Or I could interpret that as "///" maximum length separator, with no escaping. 3. Or it could be an SDE. To me, we'd be best off if the escapeCharacter was not allowed to be (SDE) the same as the first character of any in-scope terminating delimiter. We're not doing anyone any favors by allowing this. Likely a similar restriction would be needed for escapeBlockEnd, that the value of this property could not be a prefix of any in-scope-terminating delimiter, and escapeEscapeCharacter could not be the same as the first character of the escapeBlockEnd. E.g., dfdl:escapeBlockStart="/" escapeBlockEnd="/" dfdl:separator="/ // ///" With data "/foo///bar" Is that 1. escapeBlockStart, foo, escapeBlockEnd, separator "//" bar ? 2. Or escapeBlockStart, foo/, separator "/" bar ? 3. Or SDE? Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2