August 2019 - dfdl-wg - lists.ogf.org

clarification needed: unparsing and hidden group choices and choices with dfdl:choiceDispatchKey
by Mike Beckerle 13 Dec '19

13 Dec '19

So interesting issue. Comes up as part of our experience implementing the advanced expression language features: hidden groups, and computed elements. We have hidden groups in DFDL. If a xs:choice is inside a hidden group, then by definition, when unparsing, elements corresponding to some branch of the choice are not in the Infoset. DFDL spec clearly states that dfdl:outputValueCalc is not allowed on the head of a choice branch. But elements inside hidden groups must have dfdl:outputValueCalc, or a default value, otherwise there's no way for them to be filled in as part of the augmented infoset. So the only way a choice can be inside a hidden group is if the head of the choice branches have default values. This is not sufficient. I do not think it was our intention to make hidden groups incapable of holding things complex enough to contain a choice. Rather, the point is to be able to hide things that have excess complexity needed only for parsing/unparsing, so that they need not appear in the final Infoset. So if a choice is allowed inside a hidden group, then a mechanism is needed by which when unparsing a choice branch is selected based on values of other parts of the Infoset, and then the elements of that branch are computed and added to the augmented infoset. Note that without hidden groups, this problem never arises, as the Infoset always has to take a position on choice branches by having an unambiguous element in it. There are 3 cases: 1. unparsing an xs:choice where for parsing, each branch has a discriminator, or is resolved by the simple "first successful parse" rule. 2. unparsing an xs:choice with dfdl:choiceDispatchKey 3. unparsing xs:choice with dfdl:initiatedContent='yes' For all cases, I suggest we must remove the restriction on dfdl:outputValueCalc being disallowed on choice branch head. For (1). First branch where the branch can be constructed successfully is added to the augmented infoset. This is a case where an unparse error is non-fatal, and causes a form of unparser backtracking through alternatives. Branches would be attempted in schema order. For (2) I suggest dfdl:unparseChoiceDispatchKey, evaluated at unparse time only. Produces a choice branch key. That choice branch is then constructed, and default values or dfdl:outputValueCalcs are used to fill in values. (BTW: this is the driving use case we have in Daffodil for this issue.) For (3).... I have no specific solution. I think it may not actually be a separate case and has to be treated as (1) or (2). Comments/Discussion?

2 1

pointers and offsets - BDDL
by Mike Beckerle 30 Aug '19

30 Aug '19

This is a link to a format description language (proprietary) which mentions a pointers/offsets capability. http://www.binarydom.com/sdk/doc/bddl.shtml We may want to review this w.r.t. DFDL capabilities being considered in this area. There aren't too many of these to draw from in standardization. I am trying to get the Tresys one (which is not part of Daffodil) into form where we can describe it here also. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Action 313: Plus '+' sign and lax textNumberCheckPolicy
by Mike Beckerle 30 Aug '19

30 Aug '19

Looks like ICU changed behavior.... From: Steve Lawrence <slawrence(a)apache.org> Sent: Thursday, August 29, 2019 1:30 PM To: users(a)daffodil.apache.org Subject: Re: Plus '+' sign and lax textNumberCheckPolicy - was: Re: How to model a fixed-length integer that may be padded with space on the left? I think this is a difference in ICU version? A little grepping through ICU source, I found a change [1] to their number parsing logic in Dec 2017: + if (!isStrict) { + parser.addMatcher(WhitespaceMatcher.getInstance()); + parser.addMatcher(new PlusSignMatcher()); + } That looks to me like a change to make it so plus signs are always matched in lax/lenient mode regardless of the pattern (Daffodils current behavior). A couple minor changes have been made to that section, but nothing that allows you to turn if off if lenient is on. It's hard to tell in the git history what release that was in, but it looks like around version 61, which is relatively new (Daffodil is on version 62). Also, the latest version of DecimalFormatProperties.java (looks to be an internal implementation, so no online javadocs), has javadocs that states that plus signs are always allowed in lenient/lax mode [2]. I think this is a change in ICU behavior in newer versions. - Steve [1] https://github.com/unicode-org/icu/commit/68340c8464bd988477d6c88f46f9dfe45… [2] https://github.com/unicode-org/icu/blob/master/icu4j/main/classes/core/src/…

2 1

Cancelled: OGF DFDL Working Group call
by Steve Hanson 29 Aug '19

29 Aug '19

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 29 Aug '19

29 Aug '19

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Next call is 4 weeks time - Sept 26th. Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 29 Aug '19

29 Aug '19

1 0

clarification needed: textNumberCheckPolicy lax includes lax-ness about plus signs.
by Mike Beckerle 29 Aug '19

29 Aug '19

Excerpt from daffodil users mailing list indicates that the discussion of how "lax" textNumberCheckPolicy="lax" is w.r.t. plus signs on numbers. ------------------------------ > If you set textNumberCheckPolicy="lax", then > we do ignore leading plus signs in the data The DFDL specification doesn't seem to say that a leading plus sign is ignored. Here's what it says: If 'lax' and dfdl:textNumberRep is 'standard' then grouping separators are ignored, leading and trailing whitespace is ignored, leading zeros are ignored and quoted characters may be omitted. Nothing about ignoring plus signs in that. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Agenda for OGF DFDL WG Call
by Steve Hanson 29 Aug '19

29 Aug '19

Please find agenda for today's call on Redmine at https://redmine.ogf.org/dmsf_files/13593?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Confirmed: OGF DFDL Working Group call (29 Aug 16:00 GDT in https:/ibm.webex.com/meet/smh)
by Steve Hanson 29 Aug '19

29 Aug '19

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

BLOB - binary large object proposal - updated
by Mike Beckerle 29 Aug '19

29 Aug '19

This concept, ,which has been discussed before, is in high demand in the Daffodil user community to enable DFDL to be used to parse image file formats. The use case is to provide uniform image-metadata access without getting bogged down in the large byte-array that makes up most of the file and would be very large (and pointless) if rendered into XML or JSON. So our proposal, (which will get turned into an official Experimental feature document), has been simplified and revised and is described here: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Binary+Lar… Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 5