October 2018 - dfdl-wg - lists.ogf.org

Latest OGF DFDL WG Call Minutes
by Steve Hanson 16 Oct '18

16 Oct '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for today's OGF DFDL WG Call
by Steve Hanson 16 Oct '18

16 Oct '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13580?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group Call
by Steve Hanson 16 Oct '18

16 Oct '18

1 0

Clarification needed: sequence terminator that exists or not depending on expression
by Mike Beckerle 10 Oct '18

10 Oct '18

Consider the following: <element name="value" type="xs:string" ...../> <sequence dfdl:terminator="{ if (fn:string-length(./value) eq 32) then '%ES;' else '%NUL;' }"/> This is used to add a NUL at the end of a string, if the string length is less than the max length of 32. This comes up often in fixed length or variable-length-with-max data we've seen. I've put this terminator on a separate sequence after the element to emphasize that we're not scanning for terminating markup here. This has nothing to do with lengthKind 'delimited'. However, the DFDL spec says (for terminator property) · ES must not appear as the only DFDL string literal in the property. It can only appear as a member of a list. · Neither the ES entity nor the WSP* entity may appear on their own as one of the string literals in the list when the parser is determining the length of a component by scanning for delimiters. The second bullet doesn't apply to my example. Re: first bullet, I think my terminator expression is illegal... because the '%ES;' is a list of literals containing ES as the only DFDL string literal. But this is a really flawed constraint, as "%ES;%ES;" and "%ES; %ES;" both skirt the constraint, but mean the same thing as just "%ES;" which is illegal. So, if we don't want to allow these hack workarounds, we need a statement that says runs of %ES; adjacent mean the same thing as one %ES;, and that more than one identical-meaning delimiter specified in a list of string literals means the same as just one. Or we can make these hack workarounds illegal. However, why are we disallowing these? The above construct in my example is very useful, and really hard to work around unless we can have a terminator that is '%ES;' as the only string literal. Actually I have no work around for this really. I am guessing I could come up with something, but the various things I've guessed at don't pan out, or prevent the string named 'value' above from being modeled as a simple type. I know we don't want lengthKind='delimited' with terminator="%ES;" as that is most likely just a schema-definition error, but when we're not dealing with a lengthKind, we really do seem to need to specify situations where conditionally the terminator region will be empty. So I think we need to do: 1) clarify that %ES; cannot be used in combination with any other character or entity as a member of a list of string literals. 1a) At the same time I would also disallow combinations of WSP* that are misleading and unnecessary i.e., disallow %WSP*; adjacent to any other WSP, WSP+, or WSP*. 2) clarify that the constraint that %ES; for terminator and separator cannot appear as the only string literal in a list of string literals... applies only when the parser is determining the length of a component by scanning for delimiters. This is just rephrasing the two bullets above so the clause about scanning applies to both, not just the second. I believe this preserves the intent that when lengthKind="delimited" and we are scanning for delimiters, there must be *some* delimiter that is potentially not zero length. You still have to cope with the possible match being zero length due to %ES; being in the list of terminating markup, or WSP* similarly, with no whitespace found. But the notion that there is NO scanning to be done can't happen. That is, the notion that the schema specifies lengthKind delimited, but also specifies no delimiters at all, is still ruled out. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Conflicting Info: binaryDecimalVirtualPoint
by Mike Beckerle 09 Oct '18

09 Oct '18

The spec has conflicting info about the property binaryDecimalVirtualPoint Section 22.1 shows this property only being examined once binaryNumberRep is packed, bcd, or ibm4690Packed. (Note: doesn't even allow for zoned as this is 'binaryDecimal....", and zoned is considered text) Section 13.7.1.1 explicitly states this property can be used with non-base-10 representations to scale the value. While the functionality in 13.7.1.1 could be useful to people, I believe the intent of this property was to support virtual decimal points, which is, in my experience, a COBOL-oriented feature used with base-10 representations (zoned, text, or packed/bcd). In DFDL v1.0, zoned is considered textual, so this property doesn't apply to zoned numbers. That leaves only the packed/bcd representations. I suggest these changes: 1) language in 13.7.1.1 should be revised to avoid requiring this property to apply to binaryNumberRep='binary' (aka twos-complement or unsigned binary) integers. 2) The definition of the property should explicitly state it applied only to base 10 binary (aka packed/bcd) representations. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Part 2 - Re: Action 307 - Demonstrate implementation interoperability
by Mike Beckerle 09 Oct '18

09 Oct '18

Based on Daffodil JIRA ticket backlog, and documentation at https://daffodil.apache.org/unsupported/, below are DFDL non-core features that are not supported by Daffodil, but that seem to be supported by IBM DFDL (based on my not finding anything that says they aren't implemented), and so are possibly in use in DFDL schemas we will need to use for interoperability testing. Please advise if IBM DFDL does *not* implement any of these. * default, fixed - for defaulting values at parse time - Daffodil support for this is partial at parse time, unsupported at unparse time. The fixed attribute isn't supported at all. * unordered sequences * byte-value entities - in contexts other than fillByte * ICU symbols 'u' and 'I' in calendarPattern * binaryFloatRep 'ibm390Hex' * documentFinalTerminatorCanBeMissing * textStandardBase - with value not equal to 10 * lengthKind 'prefixed', and prefixIncludesPrefixLength, prefixLengthType - Note IBM restricts prefixLengthType to a type that itself cannot be prefixed. * assert with failure type 'recoverableError' * calendarObserveDST * calendarCenturyStart * textNumberPattern 'V' and 'P' symbols * CCSID for specifying dfdl:encoding * nilKind 'literalValue' for binary data * choiceLengthKind 'explicit' and choiceLength * separatorSuppressionPolicy - behaviors for these in Daffodil are known to be both non-standard currently and also different from IBM DFDL. This needs correcting. The above list (after review/correction) needs to be crossed with the published DFDL schemas on github that were published by IBM. The features required to run those DFDL schemas are required for Daffodil to implement before the interoperability demonstration. Below are features of DFDL I believe neither IBM DFDL nor Daffodil implement, and as they are non-core, they need not be implemented by either for interoperability testing: * lengthKind 'endOfParent' * nilKind 'logicalValue' * occursCountKind 'stopValue' (and occursStopValue) * textBiDi - and other related biDi properties * useNilForDefault * floating * fn:exactlyOne function * fn:namespace-URI() function * dfdl:escapeCharacterPolicy 'delimiters' - daffodil doesn't implement this property at all. Below are features of Daffodil that are not implemented by IBM DFDL and so cannot be used in schemas created using Daffodil that intend to be interoperable. These are either easy to work around, or impossible to work around, so are not a big deal, they just have to be kept in mind if considering a DFDL schema for use in interoperability testing. This includes schemas published on github, for image formats, CSV, etc., and a number of the FOUO schema published on DI2E.net/forge.mil - some of those quite possibly can work with IBM DFDL, and if they can do so, they should be modified so that they can be included in the interoperability testing. The list below mostly comes from https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… * calendarTimeZone specified as "" (empty string) - This is the most problematic one, so I've put it first. The predefined DFDL named format that is supplied with Daffodil and used as a starting point by most schemas has calendarTimeZone="". This is because customers didn't like that their datetimes were all being appended with "+00:00" for UTC time zone (in the infoset) when the data simply didn't specify a time zone. Schemas intended for interoperability testing should specify 'UTC' for this property. * calendarTimeZone specified as an Olson format time zone * inputValueCalc, outputValueCalc * hiddenGroupRef * Asserts and discriminators on simple type definitions or global element definitions * fn:concat with more than 4 arguments * non-8-bit charset encodings * bitOrder not mostSignifcantBitFirst * '@' in textNumberPattern (TBD: unsure if Daffodil has this) * "_" in calendarLanguage * calendarLanguage an expression * assert & discriminator messages an expression * binaryBooleanTrueRep as "" (empty string) * checks for binary packed numbers with length units 'bits' and not a multiple of 4 length, and similarly for alignmentUnits bits and alignment not multiple of 4. (relevant to negative tests only) * lengthKind 'implicit' complex elements inside lengthKind 'delimited' complex elements.

2 1

OGF DFDL WG Call on 2018-10-02 - cancelled
by Steve Hanson 02 Oct '18

02 Oct '18

Apologies but I can't make the call today, off work with a bad cold. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 02 Oct '18

02 Oct '18

1 0