- dfdl-wg - lists.ogf.org

Proposal: Adopt experimental emptyElementParsePolicy property in DFDL v1.0
by Mike Beckerle 27 Nov '19

27 Nov '19

A feature is described in https://redmine.ogf.org/dmsf_files/13596?download= . It describes a new property to control parser behavior which is already implemented as the standard behavior by IBM DFDL and as an optional experimental behavior by Daffodil. This is desirable functionality and it is needed for some interoperability between these implementations; hence, I propose it be included in the DFDL 1.0 spec., however, aspects that aren't implemented by all implementations of DFDL should be made optional features. We would have to create an erratum for it. I suggest the following are the changes that would be needed. * Section 9.4.2 is amended by inserting a new first paragraph "The property dfdl:emptyElementParsePolicy controls the behavior when empty representation is established while parsing. If dfdl:emptyElementParsePolicy is 'treatAsAbsent', then the empty representation is treated the same as the absent representation, and no default values are considered for addition to the infoset. The remainder of this section describes the behavior for dfdl:emptyElementParsePolicy='treatAsEmpty'." * Section 12.2 - Property description from experimental features document goes right after dfdl:emptyElementDelimiterPolicy. * Section 21 - The row for feature "Defaults" is modified. In the Detection column add "dfdl:emptyElementParsePolicy='treatAsEmpty'". The upshot of this change is that we have the pain in the neck of introducing a new property to implementations, but what we get is that IBM DFDL behavior and Daffodil behavior are then both conforming to the DFDL spec. This is way better than having to document non-conformance or deal with extension features (that not everyone has) to obtain portability. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Errata doc updated with draft errata 5.39 to 5.59 - from all Trackers
by Mike Beckerle 22 Nov '19

22 Nov '19

I have updated our document gwde-dfdl-experience-4-v0.9.docx (The Errata doc) with draft errata for all known issues that had trackers. This adds Errata 5.39 to 5.59. This is mostly just copying language that was in the trackers into the Errata document, but in some cases language had to be consolidated across the original tracker description and subsequent update comments. In some cases I had to compose descriptive language. 5.41 is not complete, as it involves review of all occurrences of MUST, MUST NOT, SHOULD, SHOULD NOT, etc. throughout the document. I plan to handle this after all other errata are incorporated into the spec, with a single sweep, then people can review by looking for those keywords. In reviewing these errata, I suggest they are bucketed into those that are not controversial, and are best reviewed by editing them into the specification, and reviewing there, versus those where there is more concern as to whether the described change is correct and complete. The list of these new errata is below. They are not controversial (to me) except where noted. 5.39 5.40 5.41 - one identified change not controvercial. The sweeping pass to examine all MUST/SHOULD, etc. is separate. 5.42 5.43 5.44 - needs careful review 5.45 - needs careful review 5.46 - new language - needs review. It is a lengthy discussion. Perhaps can be made shorter ? 5.47 - needs careful review 5.48 - do we need all 3 overloads of fn:error()? - also, all behavior here is very implementation-dependent. 5.49 5.50 5.51 5.52 5.53 5.54 5.55 5.56 5.57 5.58 5.59 - needs careful review - description may not match desired/actual behavior of implemetations Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 4

Re: [DFDL-WG] Action 313: Plus '+' sign and lax textNumberCheckPolicy
by Steve Hanson 15 Nov '19

15 Nov '19

https://unicode-org.atlassian.net/browse/ICU-20896 issue raised. I still think we need to pin DFDL 1.0 to a specific release(s). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, slawrence(a)apache.org Cc: DFDL-WG <dfdl-wg(a)ogf.org>, Liam O'Neill/UK/IBM@IBMGB Date: 30/08/2019 15:48 Subject: Re: [DFDL-WG] Action 313: Plus '+' sign and lax textNumberCheckPolicy ICU changing behaviour in an incompatible way is not good. IBM DFDL is way behind, and is still on ICU 51.2. We are limited in what we can do as we try to keep the same level as IBM Integration Bus & WTX as we have had C namespacing issues in the past. Looking at the links, there are other changes that have crept in when lenient. - The string must contain a complete prefix and suffix. For example, if the pattern is "{#};(#)", then "{123}" or "(123)" would match, but "{123", "123}", and "123" would all fail. (The latter strings would be accepted in lenient mode.) - Minus and plus signs can only appear if specified in the pattern. In lenient mode, a plus or minus sign can always precede a number. In typical ICU fashion, even this is not complete. It says nothing about what happens if the pattern has a sign and the data doesn't. I suggest you test all the combos with Daffodil and establish the truth. Then we need to decide what to do. If there is no way of controlling this (eg, parameter or env var) then the safest option is to backoff Daffodil to the latest ICU release that matches the DFDL 1.0 spec, and change the spec so that the link to ICU is specific rather than the generic link which is in the spec today ( http://www.icu-project.org/apiref/icu4c/classDecimalFormat.html#_details) and which floats to the latest release. We can't have a moving target. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 29/08/2019 19:49 Subject: [DFDL-WG] Action 313: Plus '+' sign and lax textNumberCheckPolicy Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Looks like ICU changed behavior.... From: Steve Lawrence <slawrence(a)apache.org> Sent: Thursday, August 29, 2019 1:30 PM To: users(a)daffodil.apache.org Subject: Re: Plus '+' sign and lax textNumberCheckPolicy - was: Re: How to model a fixed-length integer that may be padded with space on the left? I think this is a difference in ICU version? A little grepping through ICU source, I found a change [1] to their number parsing logic in Dec 2017: + if (!isStrict) { + parser.addMatcher(WhitespaceMatcher.getInstance()); + parser.addMatcher(new PlusSignMatcher()); + } That looks to me like a change to make it so plus signs are always matched in lax/lenient mode regardless of the pattern (Daffodils current behavior). A couple minor changes have been made to that section, but nothing that allows you to turn if off if lenient is on. It's hard to tell in the git history what release that was in, but it looks like around version 61, which is relatively new (Daffodil is on version 62). Also, the latest version of DecimalFormatProperties.java (looks to be an internal implementation, so no online javadocs), has javadocs that states that plus signs are always allowed in lenient/lax mode [2]. I think this is a change in ICU behavior in newer versions. - Steve [1] https://github.com/unicode-org/icu/commit/68340c8464bd988477d6c88f46f9dfe45… [2] https://github.com/unicode-org/icu/blob/master/icu4j/main/classes/core/src/… -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

OGF DFDL WG Call on 2019-11-14 - cancelled
by Steve Hanson 14 Nov '19

14 Nov '19

Apologies but my laptop is having network problems. Wireless is broken and for some reason I could not get ethernet to work unless docked at my desk. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Confirmed: OGF DFDL Working Group call (14 Nov 16:00 GMT in https:/ibm.webex.com/meet/smh)
by Steve Hanson 13 Nov '19

13 Nov '19

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 13 Nov '19

13 Nov '19

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13598?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL WG Call on 2019-10-31 - cancelled
by Steve Hanson 31 Oct '19

31 Oct '19

Apologies but I have a personal commitment today so have to leave work early. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 31 Oct '19

31 Oct '19

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 18 Oct '19

18 Oct '19

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Next call is 2 weeks time - Oct 31st. Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

DFDL inputValueCalc restrictions
by Bradd Kadlecik 16 Oct '19

16 Oct '19

Hello, I have one item I'd like to discuss at some point regarding inputValueCalc and not allowing the default property to be specified. I have a use case where the default property would be useful for unparse. It makes perfect sense to not allow it for outputValueCalc however. The use case is in forming a connection between an inputValueCalc element and an outputValueCalc element where the incoming document might not contain the inputValueCalc element for unparse and a default value would be desired for the outputValueCalc use. This is an item our customers using DFDL would like to make use of but don't want to require the element to always exist for unparse. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States

2 6