November 2017 - dfdl-wg - lists.ogf.org

clarification needed: unparsing and hidden group choices and choices with dfdl:choiceDispatchKey
by Mike Beckerle 13 Dec '19

13 Dec '19

So interesting issue. Comes up as part of our experience implementing the advanced expression language features: hidden groups, and computed elements. We have hidden groups in DFDL. If a xs:choice is inside a hidden group, then by definition, when unparsing, elements corresponding to some branch of the choice are not in the Infoset. DFDL spec clearly states that dfdl:outputValueCalc is not allowed on the head of a choice branch. But elements inside hidden groups must have dfdl:outputValueCalc, or a default value, otherwise there's no way for them to be filled in as part of the augmented infoset. So the only way a choice can be inside a hidden group is if the head of the choice branches have default values. This is not sufficient. I do not think it was our intention to make hidden groups incapable of holding things complex enough to contain a choice. Rather, the point is to be able to hide things that have excess complexity needed only for parsing/unparsing, so that they need not appear in the final Infoset. So if a choice is allowed inside a hidden group, then a mechanism is needed by which when unparsing a choice branch is selected based on values of other parts of the Infoset, and then the elements of that branch are computed and added to the augmented infoset. Note that without hidden groups, this problem never arises, as the Infoset always has to take a position on choice branches by having an unambiguous element in it. There are 3 cases: 1. unparsing an xs:choice where for parsing, each branch has a discriminator, or is resolved by the simple "first successful parse" rule. 2. unparsing an xs:choice with dfdl:choiceDispatchKey 3. unparsing xs:choice with dfdl:initiatedContent='yes' For all cases, I suggest we must remove the restriction on dfdl:outputValueCalc being disallowed on choice branch head. For (1). First branch where the branch can be constructed successfully is added to the augmented infoset. This is a case where an unparse error is non-fatal, and causes a form of unparser backtracking through alternatives. Branches would be attempted in schema order. For (2) I suggest dfdl:unparseChoiceDispatchKey, evaluated at unparse time only. Produces a choice branch key. That choice branch is then constructed, and default values or dfdl:outputValueCalcs are used to fill in values. (BTW: this is the driving use case we have in Daffodil for this issue.) For (3).... I have no specific solution. I think it may not actually be a separate case and has to be treated as (1) or (2). Comments/Discussion?

2 1

Re: [DFDL-WG] how to trim inside of escape block?
by Steve Hanson 22 Nov '17

22 Nov '17

Mike, your 2nd solution has a problem with the simple case when 3rd field is a single unquoted item - the List element will consume C and D - unless you can put a discriminator on C to stop it being parsed as a List - then it would work. This even simpler solution suffers from the same problem but can also be cured with a discriminator on C. <xs:sequence dfdl:separator=', ,%WSP*;" "%WSP*;,'> <xs:element name="A" type="xs:string"/> <xs:element name="B" type="xs:string"/> <xs:sequence dfdl:separator=","> <xs:element name="List" type="xs:string" maxOccurs="unbounded"/> </xs:sequence> <xs:element name="C" type="xs:string"/> <xs:element name="D" type="xs:string"/> </xs:sequence> An alternative would be to parse the 3rd field as a string with the " as escape block start/end, but make it hidden, then use an inputValueCalc expression to split the string and assign each chunk to an occurrence of List. But that would mean relaxing the restriction that inputValueCalc is not allowed on arrays. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 22/11/2017 17:41 Subject: Re: [DFDL-WG] how to trim inside of escape block? Another related problem: a, b, notList, c, d a, b, "list1, list2, list3",c,d Here the 3rd field is a list, comma separated. Quoted if there is more than one list item. I think to parse this I have to treat the quotation marks as initiator/terminator, and set dfdl:separator="", but since the quotes are optional for the single-list-item case, I'm going to need a choice. I think the best I can do is <ignore:ListOf1__XMLSchemaMakesMeHaveThisForUPA/> <List>notList</List> and <ignore:ListOfN__XMLSchemaMakesMeHaveThisForUPA/> <List>list1</List><List>list2</List><List>list3</List> as the XML representations. Are there any better/cleaner solutions? I did think of this way: (note: I've omitted xs:annotation and xs:appinfo for brevity), but it isn't exactly "clean". This is what I call "modeling syntax as data".... <dfdl:defineVariable name="foundOpenQuote" type="xs:boolean"/> <xs:group name="optionalOpenQuote"> <choice> <xs:sequence dfdl:initiiator='"'> <dfdl:setVariable ref="foundOpenQuote" value="{ fn:true() }"/> </xs:sequence> <xs:sequence dfdl:initiator=""/> </choice> </xs:group> <xs:group name="matchingCloseQuote"> <choice> <xs:sequence dfdl:terminator='"'> <dfdl:discriminator>{ $foundOpenQuote eq fn:true() }</dfdl:assert> </xs:sequence> <xs:sequence /> </choice> </xs:group> // The main sequence for the data would then have this as the list element: <xs:sequence> <dfdl:newVariableInstance ref="foundOpenQuote" defaultValue="false"/> <xs:sequence dfdl:hiddenGroupRef="optionalOpenQuote"/> <xs:sequence dfdl:separator=","> <xs:element name="List" type="xs:string" maxOccurs="unbounded"/> </xs:sequence> <xs:sequence dfdl:hiddenGroupRef="matchingCloseQuote"/> </xs:sequence> I'd try this out, except that we haven't got dfdl:newVariableInstance yet. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Nov 22, 2017 at 4:11 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I don't think there is a way to achieve what you want. As you say, trimming pad chars takes precedence over applying escape scheme. I wondered if you could define the escapeBlockStart and End as "%WSP*; and %WSP*;" respectively but the white space entities are not allowed as escape character or in escape block start/end. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 22/11/2017 01:28 Subject: [DFDL-WG] how to trim inside of escape block? Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> I have a CSV file Some lines look like this a,b," started with spaces, appearing right after the escape block start ",c,d,e I reviewed the spec, and I see that pad characters appear outside of the quotation marks (escape block start/end). What I'm trying to do is remove the whitespace after the escape block start, and before the escape block end. This is just spurious whitespace, appears because some of these CSV files were edited by people. In my data the quoting characters are not always present. They are only there if a comma appears in the data string. Is there a technique for getting rid of the leading/trailing whitespace inside the escape block start/end that I have forgotten? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] how to trim inside of escape block?
by Steve Hanson 22 Nov '17

22 Nov '17

I don't think there is a way to achieve what you want. As you say, trimming pad chars takes precedence over applying escape scheme. I wondered if you could define the escapeBlockStart and End as "%WSP*; and %WSP*;" respectively but the white space entities are not allowed as escape character or in escape block start/end. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 22/11/2017 01:28 Subject: [DFDL-WG] how to trim inside of escape block? Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> I have a CSV file Some lines look like this a,b," started with spaces, appearing right after the escape block start ",c,d,e I reviewed the spec, and I see that pad characters appear outside of the quotation marks (escape block start/end). What I'm trying to do is remove the whitespace after the escape block start, and before the escape block end. This is just spurious whitespace, appears because some of these CSV files were edited by people. In my data the quoting characters are not always present. They are only there if a comma appears in the data string. Is there a technique for getting rid of the leading/trailing whitespace inside the escape block start/end that I have forgotten? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

how to trim inside of escape block?
by Mike Beckerle 22 Nov '17

22 Nov '17

I have a CSV file Some lines look like this a,b," started with spaces, appearing right after the escape block start ",c,d,e I reviewed the spec, and I see that pad characters appear outside of the quotation marks (escape block start/end). What I'm trying to do is remove the whitespace after the escape block start, and before the escape block end. This is just spurious whitespace, appears because some of these CSV files were edited by people. In my data the quoting characters are not always present. They are only there if a comma appears in the data string. Is there a technique for getting rid of the leading/trailing whitespace inside the escape block start/end that I have forgotten? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Rescheduled: OGF DFDL Working Group call (21 Nov 15:00 GMT in IBM: Hursley DE3S09)
by Steve Hanson 17 Nov '17

17 Nov '17

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 287: Find a way to handle a variable path step in DFDL expression - Re: DFDL4S use of wildcard/regex in length path expression
by Mike Beckerle 13 Nov '17

13 Nov '17

I have analyzed this use of (.*) notation in the Sentinel2X-bandTMISPData.xsd file. Below is my discussion of what this is, why it is used, the alternative to it, and why it is more problematic than it seems at first, as an addition to DFDL. There are two instances of (.*) in the Sentinel schema files. What it expresses is a partial step-name wildcard. This is not a variation on the XPath * notation, as that matches any step, and can't match a partial name. In the usage in the Sentinel schema, this wild card is a regex that matches against a step name, and it matches exactly a single name. It is not used in a way where it can result in a node-set instead of a single node. The question really is why is this (.*) needed - that is what is it trying to achieve, and whether there is an acceptable alternative for what it is doing. I find that this wildcard (.*) is used to achieve a parameterization of the types TypeISPData and TypeISPData_HKTM. These types are polymorphic, in that their exact behavior depends on aspects of their surrounding context. Each of these types incorporates a length of some content that is outside of its own definition. As the types are used now, the length comes from a thing outside of them in the schema that happens to have a particular suffix on its name, which is "Packet_Secondary_Header" or just "_Secondary_Header". So these names, while outside of the type, are in some sense being hard-coded in these types. Even though they are outside of these types, you cannot change the names of these elements without breaking the ability of the type to find them via this (.*) notation and a name suffix. These types, TypeISPData and TypeISPData_HKTM are placed inside other types, and those are then placed in context with various packet-header structures. Those structures have various distinguishing prefixes of sub-elemetnts such as MSI_Packet_Secondary_Header. There are a variety of different things instead of "MSI_", but there's always something with suffix "Packet_Secondary_Header" or "Secondary_Header" in it. The use of this (.*) name wildcard seems convenient, but doesn't offer anything that isn't better captured by a true parameterization mechanism which decouples the names used outside the type from those used inside it. DFDL provides a general mechanism for this sort of parameterization using variables and dfdl:newVariableInstance. Let's look at just one of the two instances, TypeISPData. To use variables, a variable is created which represents this parameter to the TypeISPData. It is declared in the schema file where TypeISPData is defined. <dfdl:defineVariable name="Additional_Content_Length" type="xs:int"/> The expression within TypeISPData that currently contains "contentLength(/Packet_Data_Field/(.*)Packet_Secondary_Header, 'bytes')" That part of the expression instead references the parameter variable $Additional_Content_Length At the point of use, where this packet secondary header element is combined with the element that contains the TypeISPData, at that location, a dfdl:newVariableInstance is created and bound like so: ... <xs:element name="MSI_Packet_Secondary_Header" type="TypePacketData_MSI"/> <xs:sequence> <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl"> <dfdl:newVariableInstance ref="Additional_Content_Length" value="{ dfdl:contentLength(../MSI_Packet_Secondary_Header, 'bytes') }"/> </xs:appinfo></xs:annotation> <xs:element name="MSI_User_Data_Field" type="TypeUserData"/> </xs:sequence> ... This binds the parameter to the needed length around the point of use of the type that needs it (which is down inside the TypeUserData) This seems bulky due to the XML and XSD-annotation notational overheads, but it is just parameter-binding as would occur in an ordinary programming language when passing an argument. Using variables decouples the names of the two parts of the schema entirely. For example, if MSI_Packet_Secondary_Header were to be renamed to MSI_PSH, it would change only the point of use where the variable binding occurs, and would not affect the type definition. This also facilitates testing the types in isolation. It eliiminates the need to have the complete data structure surrounding them. Use of (*.) name matching might seem convenient, but really it depends on an unstated invariant about the way names are chosen in the Sentinel schema. If the names weren't constructed so uniformly, this notational trick would be unable to make the necessary distinctions, and you'd have to fall back on using variables. So, the above explains an alternative, already in DFDL, that can achieve the parameterization of types that is needed. There is an additional issue that makes this (.*) notation problematic. The issue is thes semantics relative to QNames and ordinary XML namespace management for name conflict control, The Sentinel schema is all a "no namespace" schema. In that world a path step is just an identifier with no structure. However, more generally in DFDL, a path step might be a QName, and those have a namespace prefix part, and a name part. Name collisions and conflicts can be managed by use of different target namespaces and namespace prefixes. Using (.*) prefix regex-matching against QNames is problematic, as it subverts the ability to use namespace prefixes to eliminate name conflicts. E.g., does (.*)_Secondary_Header match "foo:My_Secondary_Header" given that it is in the foo namespace? Matching against things without taking namespace prefixes into consideration breaks XSD's ability to use prefixes to manage name collsions. So the answer of does "(.*)_Secondary_Header" match "foo:My_Secondary_Header" would have to be that it does not match, and that if one wanted to match that one would have to write "bar:(.*)_Secondary_Header" where bar prefix is bound to the same namespace as foo in the schema file where My_Secondary_Header is defined. This mixture of some plain-textual regex matching, and some name-prefix-qualified namespace-sensitive matching is not very attractive. There is certainly nothing like this in XPath. Last issue is implementation complexity: So, if one wants to do without the (.*) notation, in terms of making DFDL4S more complex to implement, you would need to add variables. Fortunately variables are really one of the simpler things in DFDL to implement at least for parsing, and is very simple if the dfdl:setVariable annotation is not implemented since it is not needed in this case. Each variable has a stack. the dfdl:newVariableInstance pushes a new entry onto the stack, and in this case, populates it immediately with a value. Reference to the variable from an expression always takes the value of the top-of-stack location. Exiting the scope of the annotation where the dfdl:newVariableInstance appeared pops the stack. In summary, given (a) the availability of a robust parameterization mechanism using parameter variables and newVariableInstance to bind them (b) the ease of implementing variables, and (c) the complexity of working out interactions with XML/XSD namespaces/prefixes and name management,..... given those reasons, I would be disinclined to advocate something like this (.*) notation to DFDL unless it was first added to XPath, where all necessary details were worked out for us. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Sep 13, 2016 at 10:18 AM, Michele Zundo <michele.zundo(a)esa.int> wrote: > Dear Steve, > > sorry for the delay due to Summer break and other projects. > > Here is the .zip. > > Please note that we realised that one of our previous replies date 25 July > 2016 at 17:45:05 GMT+2 > was a misunderstanding from our part and not applicable. > > Regarding your questions please find answers interleaved below > > Michele > > This message and any attachments are intended for the use of the addressee or addressees only. > The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its > content is not permitted. > If you received this message in error, please notify the sender and delete it from your system. > Emails can be altered and their integrity cannot be guaranteed by the sender. > > Please consider the environment before printing this email. > > > > > Regards > > > > > > > *From:* Steve Hanson [*smh(a)uk.ibm.com* <smh(a)uk.ibm.com>] > *Sent:* Tuesday, July 26, 2016 3:20 AM > *To:* Michele Zundo > *Cc:* Mike Beckerle; *rui.mestre(a)deimos.com.pt* <rui.mestre(a)deimos.com.pt> > *Subject:* Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05 > > > ….snip > > > dfdl:length="{/Packet_Primary_Header/Packet_Data_Length + 1 - > contentLength(/Packet_Data_Fi > eld/(.*)Packet_Secondary_Header, 'bytes') - 2}" > > Firstly, contentLength is a DFDL function so it needs to be in the DFDL > namespace, eg, dfdl:contentLength(). > > > Yes agree with you. We will add the* dfdl:* it in future releases and > modify the applications accordingly. > > > Secondly, the first argument to dfdl:contentLength() is a path, so you are > effectively still using regular expressions in path steps. > > > Yes. For now we are using it and expect this to become part of the > standard. > > > Regards > > Steve Hanson > *IBM Integration Bus* > <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, > Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:*+44-1962-815848* <%2B44-1962-815848> > mob:*+44-7717-378890* <%2B44-7717-378890> > > > > From: Michele Zundo <*michele.zundo(a)esa.int* > <michele.zundo(a)esa.int>> > To: Steve Hanson/UK/IBM@IBMGB > Cc: Mike Beckerle <*mbeckerle(a)tresys.com* <mbeckerle(a)tresys.com>> > Date: 25/07/2016 17:10 > Subject: Fwd: OGF DFDL WG Call Minutes 2016-07-05 > ------------------------------ > > > > > Dear Steve, > > Please find below the answer from our developers and example. > > Note that we have updated our implementation of DFDL to be as compliant as > we > can at this point in time with the exception noted below. > > Michele > > Begin forwarded message: > > *From: *"Rui Mestre (DME)" <*rui.mestre(a)deimos.com.pt* > <rui.mestre(a)deimos.com.pt>> > *Subject: Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05* > *Date: *25 July 2016 at 17:45:05 GMT+2 > > Dear Michele, > > I believe that after our DFDL compliance effort the mentioned "use of a > regex in the path step of a DFDL expression" is no longer in place. > > Currently the only extension implemented in DFDL4S regarding the use of > regular expressions is that implementation of dfdl:contentLength is > extended to support also regular expressions when specifying the node. > > Please find attached a schema file example containing such extension in > the use of dfdl:contentLength. > > Best regards, > Rui > > > Begin forwarded message: > > *From: *Steve Hanson < <smh(a)uk.ibm.com>*smh(a)uk.ibm.com* <smh(a)uk.ibm.com>> > *Subject: OGF DFDL WG Call Minutes 2016-07-05* > *Date: *5 July 2016 at 17:49:13 GMT+2 > *To: * <dfdl-wg(a)ogf.org>*dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> > *Cc: *"Mike Beckerle" < <mbeckerle(a)tresys.com>*mbeckerle(a)tresys.com* > <mbeckerle(a)tresys.com>>, "Michele Zundo" <*michele.zundo(a)esa.int* > <michele.zundo(a)esa.int>> > > Please find minutes from the above call at > <https://redmine.ogf.org/dmsf_files/13537?download=> > *https://redmine.ogf.org/dmsf_files/13537?download=* > <https://redmine.ogf.org/dmsf_files/13537?download=> > > *@Michele - please can you send to the WG a schema that shows your use of > a regex in the path step of a DFDL expression ?* > > Next call *Aug 2nd* > > Regards > > Steve Hanson > Architect, IBM DFDL, > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > *tel:+44-1962-815848* <+44-1962-815848> > > ----------------------------------------- > Michele Zundo > > Head of Ground System Definition and Verification Office > EOP-PEP > European Space Agency, ESTEC > e-mail: <michele.zundo(a)esa.int>*michele.zundo(a)esa.int* > <michele.zundo(a)esa.int> > > > This message and any attachments are intended for the use of the addressee > or addressees only. > The unauthorised disclosure, use, dissemination or copying (either in > whole or in part) of its > content is not permitted. > If you received this message in error, please notify the sender and delete > it from your system. > Emails can be altered and their integrity cannot be guaranteed by the > sender. > > Please consider the environment before printing this email. > > ----------------------------------------- > Michele Zundo > > Head of Ground System Definition and Verification Office > EOP-PEP > European Space Agency, ESTEC > e-mail: *michele.zundo(a)esa.int* <michele.zundo(a)esa.int> > > > > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > #### Sentinel2X-bandTMISPData.xsd moved to MyAttachments Repository V3.8 ( > Link > <notes:///802575AF0030E827/5DE5236E5AD1645685256EE0001BBADF/ABC83CA7B3700B0380258018002AE581>) > on 23 August 2016 by Steve Hanson. > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > ----------------------------------------- > Michele Zundo > > Head of Ground System Definition and Verification Office > EOP-PEP > European Space Agency, ESTEC > e-mail: michele.zundo(a)esa.int > > > > > > > > > >

3 5

Agenda for next OGF DFDL WG Call
by Steve Hanson 13 Nov '17

13 Nov '17

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13568?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 13 Nov '17

13 Nov '17

1 0

Re: [DFDL-WG] delimited binary data - clarifications
by Steve Hanson 08 Nov '17

08 Nov '17

1. Yes, all delimiters allowed 2. No restriction on the encoding of such delimiters 3. Escape schemes only apply to text representation The dfdl:escapeSchemeRef property only appears in section 13.2 Properties Common to All Simple Types with Text representation. When creating a DFDL schema that involves delimited binary data, you have to be careful that your data can't contain any bytes that match any in-scope delimiter. I believe that IBM DFDL's byte scanner converts in-scope delimiters into the equivalent bytes using the dfdl:encoding of the object, then matches the bytes. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Josh Adams <jadams(a)tresys.com> Date: 08/11/2017 13:49 Subject: [DFDL-WG] delimited binary data - clarifications Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Daffodil project is implementing various packed formats, and looking at the TLOG schema on the DFDL Schemas site. The DFDL spec is clear that lengthKind delimited is allowed for packed formats (all variants of packed) and hexBinary. My question is whether there is any restriction on the generality of this that was intended, but not stated in the spec, where we should be issuing a clarification. E.g., 1. Can binary data have all of initiators, terminators, and separators? 2. Is there a restriction on the charset encoding used to specify these, e.g., SBCS? Or do the byte patterns being used to scan for these require conversion of the specified delimiter to bytes from any supported encoding? 3. Do escape schemes apply to delimited binary? If, in fact, all these things are allowed, then I believe we should add a one-liner to section 12.3.2.2 specifying that all aspects of delimited parsing including the above, are specifically allowed. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

delimited binary data - clarifications
by Mike Beckerle 08 Nov '17

08 Nov '17

Daffodil project is implementing various packed formats, and looking at the TLOG schema on the DFDL Schemas site. The DFDL spec is clear that lengthKind delimited is allowed for packed formats (all variants of packed) and hexBinary. My question is whether there is any restriction on the generality of this that was intended, but not stated in the spec, where we should be issuing a clarification. E.g., 1. Can binary data have all of initiators, terminators, and separators? 2. Is there a restriction on the charset encoding used to specify these, e.g., SBCS? Or do the byte patterns being used to scan for these require conversion of the specified delimiter to bytes from any supported encoding? 3. Do escape schemes apply to delimited binary? If, in fact, all these things are allowed, then I believe we should add a one-liner to section 12.3.2.2 specifying that all aspects of delimited parsing including the above, are specifically allowed.

1 0