- dfdl-wg - lists.ogf.org

OGF DFDL Working Group Call
by Steve Hanson 08 Jun '21

08 Jun '21

1 0

example for extraEscapedCharacters
by Mike Beckerle 25 May '21

25 May '21

I've been searching our email history for the motivating example for extraEscapedCharacters property. I can't find it. Does anyone recall this? I vaguely recall a format where we had CSV-like data that already contained escaped characters like "foo\nbar\,baz,next,next" Where the \n is just part of the data, regardless of the fact that \ is also the escape character. Normal rules would remove the "\" since it is the escape character, and that would result in corrupting such data. But it makes no sense to have extraEscapedCharacter="n" since n is just a letter and could appear without the escape. It would be wrong to escape every letter n. So this example isn't sufficient. I see no way to cope with this particular case other than lengthKind='pattern' which is the usual way to cope with complexities like this. But I know there was a motivating example for extraEscapedCharacters property. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Fw: Spec question about extraEscapedCharacters
by Steve Hanson 25 May '21

25 May '21

Mike Fyi, from 2009. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday ----- Forwarded by Steve Hanson/UK/IBM on 25/05/2021 08:56 ----- From: Steve Hanson/UK/IBM To: Alan Powell/UK/IBM@IBMGB Cc: Dragan Besevic/Boca Raton/IBM@IBMUS, Stephanie Fetzer/Charlotte/IBM@IBMUS Date: 04/11/2009 11:14 Subject: Re: Spec question about extraEscapedCharacters Alan On parsing, MRM just looks for the escape character (like DFDL). On writing, MRM needs to know what to escape and the reserved characters list is used for that (markup is not automatically included, as you say). So why does DFDL need extraEscapedCharacters property? Let's say I serialise an infoset corresponding to a structure. That will escape any in-scope markup characters in the data. But if I then include that serialised data as a BLOB is some envelope structure that also has terminating markup, I won't have escaped any of the envelope's markup characters in my BLOB. To do that I need extraEscapedCharacters property. I think the spec needs clarifying to say that extraEscapedCharacters is an output only control. Regards Steve Hanson Programming Model Architect, WebSphere Message Brokers, OGF DFDL WG Co-Chair, Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 From: Alan Powell/UK/IBM To: Stephanie Fetzer/Charlotte/IBM@IBMUS, Steve Hanson/UK/IBM@IBMGB Cc: Dragan Besevic/Boca Raton/IBM@IBMUS Date: 03/11/2009 10:48 Subject: Re: Spec question about extraEscapedCharacters Stephanie extraEscapedCharacters comes from the WMB Reserved Characters property. It isn't exactly the same as WMB doesn't automatically look for markup. Also the documentation says that Reserved Characters aren't used on parsing which doesn't seem consistent. Steve do you know why? As currently specified your example wouldn't be a parsing error but may cause the data to be interpreted incorrectly if ^ is in fact markup. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Stephanie Fetzer/Charlotte/IBM@IBMUS To: Alan Powell/UK/IBM@IBMGB Cc: Dragan Besevic/Boca Raton/IBM@IBMUS Date: 02/11/2009 21:05 Subject: Spec question about extraEscapedCharacters Alan: Real quick question about extraEscapedCharacters. The current definition of the term is: String A space separated list of single characters that must be escaped in addition to markup. Annotation: dfdl: escapeScheme So - my interpretation is that is someone wants to enforce...even though the ^ is not an initiator, terminator, delimiter, or any other types or markup in this document - I want to force the sender of the data to always escape it. So extraEscapedCharacters="^" and the data to be parsed looks like: "NAME#ADDRESS#PHONENUMB^ER" That would be an error (unless the B is an escape char). We would expect the ^ to always be escaped in the data. So even though we don't need it to be escaped from a parsing standpoint we would expect it to be a parsing error? Or is the property merely a suggestion on parsing? Any hints that you can provide on the reasoning for this property would be appreciated. Cheers, -Steph WebSphere Transformation Extender Industry Packs - Software Engineer Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 11 May '21

11 May '21

1 0

Clarification: priority of delimiters vs. escape chars
by Mike Beckerle 05 May '21

05 May '21

Consider two elements in a sequence, dfdl:separator="/ // ///" with escapeCharacter="/" and escapeEscapeCharacter="/" I did not spot language in the spec that makes it clear what gets priority, interpreting a character as an escape char or escape-escape char, or interpreting it as a delimiter. Consider data "foo///bar". 1. I could interpret that as escapeEscape, escape, and minimum length separator "/" 2. Or I could interpret that as "///" maximum length separator, with no escaping. 3. Or it could be an SDE. To me, we'd be best off if the escapeCharacter was not allowed to be (SDE) the same as the first character of any in-scope terminating delimiter. We're not doing anyone any favors by allowing this. Likely a similar restriction would be needed for escapeBlockEnd, that the value of this property could not be a prefix of any in-scope-terminating delimiter, and escapeEscapeCharacter could not be the same as the first character of the escapeBlockEnd. E.g., dfdl:escapeBlockStart="/" escapeBlockEnd="/" dfdl:separator="/ // ///" With data "/foo///bar" Is that 1. escapeBlockStart, foo, escapeBlockEnd, separator "//" bar ? 2. Or escapeBlockStart, foo/, separator "/" bar ? 3. Or SDE? Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Confirming expected behavior of separatorSuppressionPolicy 'never'
by Mike Beckerle 22 Apr '21

22 Apr '21

An earlier email I sent had this schema: <xs:element name="file1"> <xs:complexType> <xs:sequence dfdl:separator="," dfdl:separatorPosition="infix" dfdl:separatorSuppressionPolicy="never"> <xs:element name="given-name" type="xs:string" minOccurs="0" maxOccurs="3" /> <xs:element name="surname" type="xs:string" minOccurs="0"/> <xs:element name="phone" type="xs:string" minOccurs="0" maxOccurs="6" /> </xs:sequence> </xs:complexType> </xs:element> Given that, and a default dfd:format with occursCountKind='implicit' I expect this data: "madonna,,,,,,,,," To produce this DFDL Infoset (cast as XML): <file1> <given-name>madonna</given-name> </file1> I don't expect to see any empty elements like <given-name></given-name> because all elements are optional, so a zero-length representation for a xs:string is the Empty representation, and optional empty is never added to the infoset. Furthermore, this should "round trip" i.e., unparse back to the original input data. Am I correct here? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 1

example/use-case wanted - separatorSuppressionPolicy 'trailingEmptyStrict' and 'never'
by Mike Beckerle 21 Apr '21

21 Apr '21

Users of Apache Daffodil have asked what these property values are for, i.e., what are idiomatic uses of this property for actual formats, or in other words, what kind of data are these properties needed for? I'd like to provide them examples of this if anyone can recall the kinds of specific use cases for these property values. I've been searching through the example DFDL schemas I can find, and none use trailingEmptyStrict or never for this property. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Latest OGF DFDL WG Call Minutes
by Steve Hanson 16 Apr '21

16 Apr '21

Please find minutes from the latest call at https://github.com/OpenGridForum/DFDL/wiki/DFDL-Workgroup-News Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 15 Apr '21

15 Apr '21

Please find agenda for next call on GitHub at https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-04-15_DFDL-WG-… Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

newVariableInstance direction property discussion - PCAP example
by Mike Beckerle 13 Apr '21

13 Apr '21

For our next conference call, I said I would walk through the Apache Daffodil newVariableInstance feature, and how for unparsing we had to add a direction property to variables so they can be evaluated with forward referencing for unparsing. The code I'd like to discuss is here: https://github.com/DFDLSchemas/PCAP/blob/cb9ee66aa565d982f01130918bcd785def… The defineVariable that is the best motivating example is defined at line 104, and is used at line 213. This is a simple common-subexpression elimination, but it is used to avoid redundant calls to dfdl:valueLength in the dfdl:outputValueCalc of the stored length field named InclLen. There is a second example of using newVariableInstance at unparse time in this schema also, with the variable definitions starting at line 115, and usage in the group defined starting at line 320. This usage is far more complex, and in hindsight I think a DFDL parse of PCAP data should not be creating IP addresses in the Infoset that look like "1.2.3.4", because it requires all this complexity to unparse from that back to the binary representation. However, it is a good exercise in how the newVariableInstance feature works. -mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 3