November 2019 - dfdl-wg - lists.ogf.org

DFDL pointer & offset proposal
by Bradd Kadlecik 30 Apr '20

30 Apr '20

Here's what I've put together regarding the pointer & offset proposal for the next meeting's review. (See attached file: DFDL_Indirection.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 04/05/2019 12:42 PM Subject: Re: Latest OGF DFDL WG Call Minutes |------+------------------------------------------------------------------| | No |Action | |------+------------------------------------------------------------------| | 309 |Create example scenarios to illustrate offset & pointer | | |requirements (Bradd) | | |5/4/19: Daffodil have a draft proposal for offset support, TPF | | |have experimental implementation for pointer support. Need | | |examples to show the requirement, especially unparsing. | |------+------------------------------------------------------------------| Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 05/04/2019 10:29 Subject: Re: Latest OGF DFDL WG Call Minutes I can see a difference between offsets and pointers. If I follow an offset and parse an element x then I won't automatically jump back to where I was - the next element y I parse will continue from location offset + length x unless I use offset again to jump back to the original location. If I follow a pointer and parse an element x, then the next element y I parse will continue from original location + length(pointer). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 04/04/2019 09:42 Subject: Re: Latest OGF DFDL WG Call Minutes This is a proposal from the Daffodil team for offset support, needed for formats like TIFF and if we ever want to be able to handle zip files. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74687382 I think this proposal can implement your requirement - the dfdl:offset property can be an expression that refers to another element (your pointer element), which can be hidden so as not to appear in the infoset. I think what you are proposing is a more convenient way of handling offsets that are defined dynamically in the data, as opposed to defined statically with fixed values (though as I said below I need unparsing explained). But I may be mis-understanding your use cases. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 03/04/2019 16:50 Subject: Re: Latest OGF DFDL WG Call Minutes Ok, I'll work on putting a scenario together for various pointer setups (array, complex element, string) and show how it looks for both the JSON and the binary. I'm currently at a conference this week but will be returning late Thursday so expect to be available for the call Friday. I presented the TPF specific pointer implementation at the conference this week and there are some that will be trying to use it soon. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 04/03/2019 10:29 AM Subject: Re: Latest OGF DFDL WG Call Minutes Hi Bradd I have a few questions please ... inline below. I'm still going to need a real worked example, starting with some actual data and its schema, what it appears like after paring in the infoset, and how unparsing lays it back out again. Are you able to make the rescheduled call this Friday? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 27/02/2019 22:22 Subject: Re: Latest OGF DFDL WG Call Minutes Regarding proposal for offsets and pointers: The following are the properties to be defined: SMH: I was expecting to see these properties only on dfdl:element, especially as you say '...the element contents... ? indirectKind Enum Valid values 'pointer', 'offset' (there is also a thought of objectId or refId for handling BSON but not at this time) Specifies the type of indirection used to access the element contents in the data stream. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I am missing the distinction between offset and pointer. Is one relative to current position and the other relative to start of bitstream? SMH: In earlier DFDL proposals for offset support, we had used the term to refer to a property to be used to establish position of the current element instead of assuming the current element followed straight after the previous one. It would allow sparse modelling of fixed structures. The offset could be relative to start of bitstream or some other point. I don't think that's what you mean when you say 'offset' so I will refer to your new concept as 'pointer'. SMH: Assuming that indirectKind is a normal DFDL property, it can be in scope. It would therefore need to have an enum 'None' which would be the default used in most schemas. indirectLength Non-negative Integer or DFDL expression Specifies the length of the indirection in units according to the indirectUnits property. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group indirectUnits Enum Valid values 'bytes','bits' Specifies the units to be used for reading or writing the indirection according to indirectLength. The default value is 'bytes'. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I think a better approach is to provide a property dfdl:indirectType, instead of indirectLength/indirectUnits, which refers to a simple type (not element) that carries its own lengthKind, length & lengthUnits properties. Similar idea to dfdl:prefixLengthType. That allows a lot of flexibility on how the pointer can appear. offsetBase non-empty string containing an absolute or relative XPath expression for the base element. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group The proposal would be to have the contents of the indirection be after the LeadingAlignment and before the TrailingAlignment. This would mean the aligment and skip factors apply to the indirection values in the data stream instead of the the contents of the indirection. SMH: Agree. This also then means in an array element, each element has its own indirection value (pointer or offset) and the alignment,skip factors then apply to each of these indirection values. SMH: Do you mean '...each occurrence...' ? It would be thought that the indirection values apply only to the data stream and not the infoset. During parse when the infoset is populated from the data stream, the indirection values are replaced by the contents. During unparse, the indirection values don't exist in the infoset and are created during the writing to/creation of the data stream. SMH: I agree that the indirection should be a purely physical thing, but I am not clear how the value is filled in when unparsing. Where does the value come from? outputValueCalc? Or maybe it's not needed when unparsing, and the data is always contiguous? For pointers, a null pointer creates the scenario of either nil representation or empty representation depending on whether or not nillable is defined as true. Unless default values (or 0 occurrence) are defined for all underlying content, then this is a processing error. During unparse, the only scenario in which a null pointer would be created is for a nil representation. SMH: This needs more thought. The nil & default properties apply to the contents of the indirection, not to the pointer. If you want to give a nil semantic to the pointer value itself, then that would require a new enum for dfdl:nilKind. I don't see why a pointer value 0 can't be treated like any other indirection value. A missing pointer is an error - it must be present - there is no way to control optionality because minOccurs/maxOccurs apply to the contents. (Alternatively, if you want the concepts of nil, default, occurs to apply to the indirect value, then dfdl:indirectType could point at an element instead of a simple type - but that seems way too over engineered). Examples: The following is the definition for the address of a null-terminated string in which the string address may be NULL as indicated by a nillable value of true: <xs:element name="myString" type="xs:string" dfdl:lengthKind="delimited" dfdl:encoding="UTF-8" dfdl:terminator="%NUL;" dfdl:indirectKind="pointer" dfdl:indirectLength="8" dfdl:indirectUnits="bytes" nillable="true" /> The following is the definition for an array of three 4 byte addresses of a complex element defined by ns0:myStruct: <xs:element name="myArray" type="ns0:myStruct" dfdl:lengthKind="implicit" dfdl:indirectKind="pointer" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" minOccurs="3" maxOccurs="3" dfdl:occursCountKind="fixed" /> The following is the definition for a 4 byte offset to a 100 byte hexBinary value from the start of the parent element definition: <xs:element name="myData" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="100" dfdl:lengthUnits="bytes" dfdl:indirectKind="offset" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" dfdl:offsetBase=".." /> SMH: I don't see how unparsing works. What provides the value? The proposal would also allow for the following optional item but I don't currently see a need for this: dfdl:offsetKind with values "startToStart" or "endToStart" - indicates if the offset is from the start of the base element or the end of the base element. I tried getting this out before my vacation so it might take a little bit to respond for issues. Thank you for your time. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 02/07/2019 12:32 PM Subject: Latest OGF DFDL WG Call Minutes Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 3

clarification needed: unparsing and hidden group choices and choices with dfdl:choiceDispatchKey
by Mike Beckerle 13 Dec '19

13 Dec '19

So interesting issue. Comes up as part of our experience implementing the advanced expression language features: hidden groups, and computed elements. We have hidden groups in DFDL. If a xs:choice is inside a hidden group, then by definition, when unparsing, elements corresponding to some branch of the choice are not in the Infoset. DFDL spec clearly states that dfdl:outputValueCalc is not allowed on the head of a choice branch. But elements inside hidden groups must have dfdl:outputValueCalc, or a default value, otherwise there's no way for them to be filled in as part of the augmented infoset. So the only way a choice can be inside a hidden group is if the head of the choice branches have default values. This is not sufficient. I do not think it was our intention to make hidden groups incapable of holding things complex enough to contain a choice. Rather, the point is to be able to hide things that have excess complexity needed only for parsing/unparsing, so that they need not appear in the final Infoset. So if a choice is allowed inside a hidden group, then a mechanism is needed by which when unparsing a choice branch is selected based on values of other parts of the Infoset, and then the elements of that branch are computed and added to the augmented infoset. Note that without hidden groups, this problem never arises, as the Infoset always has to take a position on choice branches by having an unambiguous element in it. There are 3 cases: 1. unparsing an xs:choice where for parsing, each branch has a discriminator, or is resolved by the simple "first successful parse" rule. 2. unparsing an xs:choice with dfdl:choiceDispatchKey 3. unparsing xs:choice with dfdl:initiatedContent='yes' For all cases, I suggest we must remove the restriction on dfdl:outputValueCalc being disallowed on choice branch head. For (1). First branch where the branch can be constructed successfully is added to the augmented infoset. This is a case where an unparse error is non-fatal, and causes a form of unparser backtracking through alternatives. Branches would be attempted in schema order. For (2) I suggest dfdl:unparseChoiceDispatchKey, evaluated at unparse time only. Produces a choice branch key. That choice branch is then constructed, and default values or dfdl:outputValueCalcs are used to fill in values. (BTW: this is the driving use case we have in Daffodil for this issue.) For (3).... I have no specific solution. I think it may not actually be a separate case and has to be treated as (1) or (2). Comments/Discussion?

2 1

New Experimental Feature document: dfdl:emptyElementParsePolcy
by Mike Beckerle 02 Dec '19

02 Dec '19

Per one of our trackers I uploaded an "official" document to describe this experimental feature. gwde-dfdl-experience-7-emptyElementParsePolicy.docx This describes the feature as we have implemented it in Daffodil. There has been some discussion on this proposal already, particularly on whether the enum values for the property are appropriately chosen so that they clearly identify the different behaviors. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 6

Cancelled: OGF DFDL Working Group call
by Steve Hanson 28 Nov '19

28 Nov '19

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call on 2019-11-28 - cancelled
by Steve Hanson 28 Nov '19

28 Nov '19

Thanksgiving in USA. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Confirmed: OGF DFDL Working Group call (28 Nov 16:00 GMT in https:/ibm.webex.com/meet/smh)
by Steve Hanson 27 Nov '19

27 Nov '19

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 27 Nov '19

27 Nov '19

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13598?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Proposal: Adopt experimental emptyElementParsePolicy property in DFDL v1.0
by Mike Beckerle 27 Nov '19

27 Nov '19

A feature is described in https://redmine.ogf.org/dmsf_files/13596?download= . It describes a new property to control parser behavior which is already implemented as the standard behavior by IBM DFDL and as an optional experimental behavior by Daffodil. This is desirable functionality and it is needed for some interoperability between these implementations; hence, I propose it be included in the DFDL 1.0 spec., however, aspects that aren't implemented by all implementations of DFDL should be made optional features. We would have to create an erratum for it. I suggest the following are the changes that would be needed. * Section 9.4.2 is amended by inserting a new first paragraph "The property dfdl:emptyElementParsePolicy controls the behavior when empty representation is established while parsing. If dfdl:emptyElementParsePolicy is 'treatAsAbsent', then the empty representation is treated the same as the absent representation, and no default values are considered for addition to the infoset. The remainder of this section describes the behavior for dfdl:emptyElementParsePolicy='treatAsEmpty'." * Section 12.2 - Property description from experimental features document goes right after dfdl:emptyElementDelimiterPolicy. * Section 21 - The row for feature "Defaults" is modified. In the Detection column add "dfdl:emptyElementParsePolicy='treatAsEmpty'". The upshot of this change is that we have the pain in the neck of introducing a new property to implementations, but what we get is that IBM DFDL behavior and Daffodil behavior are then both conforming to the DFDL spec. This is way better than having to document non-conformance or deal with extension features (that not everyone has) to obtain portability. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Errata doc updated with draft errata 5.39 to 5.59 - from all Trackers
by Mike Beckerle 22 Nov '19

22 Nov '19

I have updated our document gwde-dfdl-experience-4-v0.9.docx (The Errata doc) with draft errata for all known issues that had trackers. This adds Errata 5.39 to 5.59. This is mostly just copying language that was in the trackers into the Errata document, but in some cases language had to be consolidated across the original tracker description and subsequent update comments. In some cases I had to compose descriptive language. 5.41 is not complete, as it involves review of all occurrences of MUST, MUST NOT, SHOULD, SHOULD NOT, etc. throughout the document. I plan to handle this after all other errata are incorporated into the spec, with a single sweep, then people can review by looking for those keywords. In reviewing these errata, I suggest they are bucketed into those that are not controversial, and are best reviewed by editing them into the specification, and reviewing there, versus those where there is more concern as to whether the described change is correct and complete. The list of these new errata is below. They are not controversial (to me) except where noted. 5.39 5.40 5.41 - one identified change not controvercial. The sweeping pass to examine all MUST/SHOULD, etc. is separate. 5.42 5.43 5.44 - needs careful review 5.45 - needs careful review 5.46 - new language - needs review. It is a lengthy discussion. Perhaps can be made shorter ? 5.47 - needs careful review 5.48 - do we need all 3 overloads of fn:error()? - also, all behavior here is very implementation-dependent. 5.49 5.50 5.51 5.52 5.53 5.54 5.55 5.56 5.57 5.58 5.59 - needs careful review - description may not match desired/actual behavior of implemetations Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 4

Re: [DFDL-WG] Action 313: Plus '+' sign and lax textNumberCheckPolicy
by Steve Hanson 15 Nov '19

15 Nov '19

https://unicode-org.atlassian.net/browse/ICU-20896 issue raised. I still think we need to pin DFDL 1.0 to a specific release(s). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, slawrence(a)apache.org Cc: DFDL-WG <dfdl-wg(a)ogf.org>, Liam O'Neill/UK/IBM@IBMGB Date: 30/08/2019 15:48 Subject: Re: [DFDL-WG] Action 313: Plus '+' sign and lax textNumberCheckPolicy ICU changing behaviour in an incompatible way is not good. IBM DFDL is way behind, and is still on ICU 51.2. We are limited in what we can do as we try to keep the same level as IBM Integration Bus & WTX as we have had C namespacing issues in the past. Looking at the links, there are other changes that have crept in when lenient. - The string must contain a complete prefix and suffix. For example, if the pattern is "{#};(#)", then "{123}" or "(123)" would match, but "{123", "123}", and "123" would all fail. (The latter strings would be accepted in lenient mode.) - Minus and plus signs can only appear if specified in the pattern. In lenient mode, a plus or minus sign can always precede a number. In typical ICU fashion, even this is not complete. It says nothing about what happens if the pattern has a sign and the data doesn't. I suggest you test all the combos with Daffodil and establish the truth. Then we need to decide what to do. If there is no way of controlling this (eg, parameter or env var) then the safest option is to backoff Daffodil to the latest ICU release that matches the DFDL 1.0 spec, and change the spec so that the link to ICU is specific rather than the generic link which is in the spec today ( http://www.icu-project.org/apiref/icu4c/classDecimalFormat.html#_details) and which floats to the latest release. We can't have a moving target. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 29/08/2019 19:49 Subject: [DFDL-WG] Action 313: Plus '+' sign and lax textNumberCheckPolicy Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Looks like ICU changed behavior.... From: Steve Lawrence <slawrence(a)apache.org> Sent: Thursday, August 29, 2019 1:30 PM To: users(a)daffodil.apache.org Subject: Re: Plus '+' sign and lax textNumberCheckPolicy - was: Re: How to model a fixed-length integer that may be padded with space on the left? I think this is a difference in ICU version? A little grepping through ICU source, I found a change [1] to their number parsing logic in Dec 2017: + if (!isStrict) { + parser.addMatcher(WhitespaceMatcher.getInstance()); + parser.addMatcher(new PlusSignMatcher()); + } That looks to me like a change to make it so plus signs are always matched in lax/lenient mode regardless of the pattern (Daffodils current behavior). A couple minor changes have been made to that section, but nothing that allows you to turn if off if lenient is on. It's hard to tell in the git history what release that was in, but it looks like around version 61, which is relatively new (Daffodil is on version 62). Also, the latest version of DecimalFormatProperties.java (looks to be an internal implementation, so no online javadocs), has javadocs that states that plus signs are always allowed in lenient/lax mode [2]. I think this is a change in ICU behavior in newer versions. - Steve [1] https://github.com/unicode-org/icu/commit/68340c8464bd988477d6c88f46f9dfe45… [2] https://github.com/unicode-org/icu/blob/master/icu4j/main/classes/core/src/… -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2