- dfdl-wg - lists.ogf.org

Conflicting Info: binaryDecimalVirtualPoint
by Mike Beckerle 09 Oct '18

09 Oct '18

The spec has conflicting info about the property binaryDecimalVirtualPoint Section 22.1 shows this property only being examined once binaryNumberRep is packed, bcd, or ibm4690Packed. (Note: doesn't even allow for zoned as this is 'binaryDecimal....", and zoned is considered text) Section 13.7.1.1 explicitly states this property can be used with non-base-10 representations to scale the value. While the functionality in 13.7.1.1 could be useful to people, I believe the intent of this property was to support virtual decimal points, which is, in my experience, a COBOL-oriented feature used with base-10 representations (zoned, text, or packed/bcd). In DFDL v1.0, zoned is considered textual, so this property doesn't apply to zoned numbers. That leaves only the packed/bcd representations. I suggest these changes: 1) language in 13.7.1.1 should be revised to avoid requiring this property to apply to binaryNumberRep='binary' (aka twos-complement or unsigned binary) integers. 2) The definition of the property should explicitly state it applied only to base 10 binary (aka packed/bcd) representations. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Part 2 - Re: Action 307 - Demonstrate implementation interoperability
by Mike Beckerle 09 Oct '18

09 Oct '18

Based on Daffodil JIRA ticket backlog, and documentation at https://daffodil.apache.org/unsupported/, below are DFDL non-core features that are not supported by Daffodil, but that seem to be supported by IBM DFDL (based on my not finding anything that says they aren't implemented), and so are possibly in use in DFDL schemas we will need to use for interoperability testing. Please advise if IBM DFDL does *not* implement any of these. * default, fixed - for defaulting values at parse time - Daffodil support for this is partial at parse time, unsupported at unparse time. The fixed attribute isn't supported at all. * unordered sequences * byte-value entities - in contexts other than fillByte * ICU symbols 'u' and 'I' in calendarPattern * binaryFloatRep 'ibm390Hex' * documentFinalTerminatorCanBeMissing * textStandardBase - with value not equal to 10 * lengthKind 'prefixed', and prefixIncludesPrefixLength, prefixLengthType - Note IBM restricts prefixLengthType to a type that itself cannot be prefixed. * assert with failure type 'recoverableError' * calendarObserveDST * calendarCenturyStart * textNumberPattern 'V' and 'P' symbols * CCSID for specifying dfdl:encoding * nilKind 'literalValue' for binary data * choiceLengthKind 'explicit' and choiceLength * separatorSuppressionPolicy - behaviors for these in Daffodil are known to be both non-standard currently and also different from IBM DFDL. This needs correcting. The above list (after review/correction) needs to be crossed with the published DFDL schemas on github that were published by IBM. The features required to run those DFDL schemas are required for Daffodil to implement before the interoperability demonstration. Below are features of DFDL I believe neither IBM DFDL nor Daffodil implement, and as they are non-core, they need not be implemented by either for interoperability testing: * lengthKind 'endOfParent' * nilKind 'logicalValue' * occursCountKind 'stopValue' (and occursStopValue) * textBiDi - and other related biDi properties * useNilForDefault * floating * fn:exactlyOne function * fn:namespace-URI() function * dfdl:escapeCharacterPolicy 'delimiters' - daffodil doesn't implement this property at all. Below are features of Daffodil that are not implemented by IBM DFDL and so cannot be used in schemas created using Daffodil that intend to be interoperable. These are either easy to work around, or impossible to work around, so are not a big deal, they just have to be kept in mind if considering a DFDL schema for use in interoperability testing. This includes schemas published on github, for image formats, CSV, etc., and a number of the FOUO schema published on DI2E.net/forge.mil - some of those quite possibly can work with IBM DFDL, and if they can do so, they should be modified so that they can be included in the interoperability testing. The list below mostly comes from https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… * calendarTimeZone specified as "" (empty string) - This is the most problematic one, so I've put it first. The predefined DFDL named format that is supplied with Daffodil and used as a starting point by most schemas has calendarTimeZone="". This is because customers didn't like that their datetimes were all being appended with "+00:00" for UTC time zone (in the infoset) when the data simply didn't specify a time zone. Schemas intended for interoperability testing should specify 'UTC' for this property. * calendarTimeZone specified as an Olson format time zone * inputValueCalc, outputValueCalc * hiddenGroupRef * Asserts and discriminators on simple type definitions or global element definitions * fn:concat with more than 4 arguments * non-8-bit charset encodings * bitOrder not mostSignifcantBitFirst * '@' in textNumberPattern (TBD: unsure if Daffodil has this) * "_" in calendarLanguage * calendarLanguage an expression * assert & discriminator messages an expression * binaryBooleanTrueRep as "" (empty string) * checks for binary packed numbers with length units 'bits' and not a multiple of 4 length, and similarly for alignmentUnits bits and alignment not multiple of 4. (relevant to negative tests only) * lengthKind 'implicit' complex elements inside lengthKind 'delimited' complex elements.

2 1

OGF DFDL WG Call on 2018-10-02 - cancelled
by Steve Hanson 02 Oct '18

02 Oct '18

Apologies but I can't make the call today, off work with a bad cold. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 02 Oct '18

02 Oct '18

1 0

OGF DFDL WG Call on 2018-09-18 - cancelled
by Steve Hanson 18 Sep '18

18 Sep '18

Apologies but I can't make the call today. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 18 Sep '18

18 Sep '18

1 0

Re: [DFDL-WG] Proposed feature: lookup tables via simple type unions
by Steve Hanson 12 Sep '18

12 Sep '18

Hi Mike I have taken a look at your revised proposal at https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Features+t… . Can you augment the example with use of the simple type by an element? I think this might highlight a problem. Today we allow the DFDL representation properties for a simple element to be split across the element declaration and the (hierarchy) of simple types. Your proposal is allowing the rep type of an element to be other than the allowed set for the simple type of the element. This prevents the combination of rep properties across the element and the simple type. In fact, it means that all type dependent rep properties would have to be carried by the rep type. That's not going to work because of scoping rules - you can't just switch off property applicability. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 04/04/2017 13:42 Subject: Re: [DFDL-WG] Proposed feature: lookup tables via simple type unions Although signed-up to Confluence I couldn't see a way to comment on the proposal there. I'm not comfortable with the proposal. 1) It relies upon validation being enabled, which is an optional feature. There is a long-held principle that switching on validation does not change the behaviour of a parse, but this will do exactly that. [unionMemberSchema] String. For simple element information items, this member contains an SCD reference to the member of the union that matched the value of the element. Empty if validation is not enabled. Empty if the element's type is not a union. 2) DFDL does not allow the concept of an annotation on a simple type that is a union member. We would be allowing that, but with a completely disjoint property set. Unions; the memberTypes must be derived from the same simple type. DFDL annotations are not permitted on union members 3) The proposal does not help the case where I am not using unions but still would like enums translated into meaningful strings, a far more common situation. I think this processing is best left to either an independent post-parse step or a parser extension via a set of non-DFDL annotations. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 22/02/2017 17:38 Subject: [DFDL-WG] Proposed feature: lookup tables via simple type unions Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> A write up of the proposal, which we are prototyping in Daffodil, is here: https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Enumerations+a… This is needed by a number of data formats we are working with where there are large enumerations having as many as 2000 members. Often these enumerations are a mixture where single values correspond to some enumerated strings, and ranges of values correspond to others. Using expressions to translate representation integers into strings is infeasible, as no constant-time case-statement-like construct is available in DFDL. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Apache Daffodil (Incubating) release 2.2.0
by Mike Beckerle 05 Sep '18

05 Sep '18

The Apache Daffodil (incubating) community is pleased to announce the release of version 2.2.0. Release notes and downloads are available at: https://daffodil.apache.org/releases/2.2.0/ Daffodil is an open source implementation of the DFDL (Data Format Description Language) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 04 Sep '18

04 Sep '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for next OGF DFDL WG Call
by Steve Hanson 04 Sep '18

04 Sep '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13579?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0