August 2016 - dfdl-wg - lists.ogf.org

Correction to 12.3.7.1 Length of Simple Elements with Textual Representation
by Mike Beckerle 14 Sep '16

14 Sep '16

This section says "For a textual element, the dfdl:lengthUnits property can be either 'bytes' or 'characters'." I think this statement is just old. It predates the addition of 7-bit and 6-bit characters to DFDL, and the notion that the width of characters might not be a multiple of 8-bits. This section can be removed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

minor spec correction Section 5.2
by Mike Beckerle 14 Sep '16

14 Sep '16

In Section 5.2 The phrase "pattern (for type xs:string and all types descending from xs:string in Figure 3 DFDL simple types)" The entire parenthesized part of that should be dropped, as in that figure it is clear that DFDL does not have any types that are decending from xs:string. So this is just misleading. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

DFDL4S use of wildcard/regex in length path expression
by Mike Beckerle 13 Sep '16

13 Sep '16

(I edited the subject line for this discussion.) Michele, Could you send the parts of the DFDL4S schema that use the types that have these wildcard/regex in them? Also, please send as a zip file, not a jar. Google, in its infinite wisdom, has decided that it is better to save me from a potential computer virus than to let me get my work done, and they block all access to attachments that are jar files, you can't rename them, save to Drive, save to local file system, nor forward them to a different email account. I want to see exactly what this wildcard is doing for you. I suspect it is just allowing these types to be reused in contexts where otherwise you'd have to have separate complex type definitions just because this one step of the path is different at the different points of re-use. I want to really think about whether there is some other way to achieve this same reusability, or if not, just how bad is the workaround. ...mike beckerle ------------------------------ *From:* Steve Hanson [smh(a)uk.ibm.com] *Sent:* Tuesday, July 26, 2016 3:20 AM *To:* Michele Zundo *Cc:* Mike Beckerle; rui.mestre(a)deimos.com.pt *Subject:* Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05 Michele, Rui I assume that this is an example ? dfdl:length="{/Packet_Primary_Header/Packet_Data_Length + 1 - contentLength( /Packet_Data_Field/(.*)Packet_Secondary_Header, 'bytes') - 2}" Firstly, contentLength is a DFDL function so it needs to be in the DFDL namespace, eg, dfdl:contentLength(). Secondly, the first argument to dfdl:contentLength() is a path, so you are effectively still using regular expressions in path steps. Regards Steve Hanson *IBM Integration Bus* <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle(a)tresys.com> Date: 25/07/2016 17:10 Subject: Fwd: OGF DFDL WG Call Minutes 2016-07-05 ------------------------------ Dear Steve, Please find below the answer from our developers and example. Note that we have updated our implementation of DFDL to be as compliant as we can at this point in time with the exception noted below. Michele Begin forwarded message: *From: *"Rui Mestre (DME)" <*rui.mestre(a)deimos.com.pt* <rui.mestre(a)deimos.com.pt>> *Subject: Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05* *Date: *25 July 2016 at 17:45:05 GMT+2 Dear Michele, I believe that after our DFDL compliance effort the mentioned "use of a regex in the path step of a DFDL expression" is no longer in place. Currently the only extension implemented in DFDL4S regarding the use of regular expressions is that implementation of dfdl:contentLength is extended to support also regular expressions when specifying the node. Please find attached a schema file example containing such extension in the use of dfdl:contentLength. Best regards, Rui Begin forwarded message: *From: *Steve Hanson < <smh(a)uk.ibm.com>*smh(a)uk.ibm.com* <smh(a)uk.ibm.com>> *Subject: OGF DFDL WG Call Minutes 2016-07-05* *Date: *5 July 2016 at 17:49:13 GMT+2 *To: * <dfdl-wg(a)ogf.org>*dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> *Cc: *"Mike Beckerle" < <mbeckerle(a)tresys.com>*mbeckerle(a)tresys.com* <mbeckerle(a)tresys.com>>, "Michele Zundo" <*michele.zundo(a)esa.int* <michele.zundo(a)esa.int>> Please find minutes from the above call at <https://redmine.ogf.org/dmsf_files/13537?download=> *https://redmine.ogf.org/dmsf_files/13537?download=* <https://redmine.ogf.org/dmsf_files/13537?download=> * @Michele - please can you send to the WG a schema that shows your use of a regex in the path step of a DFDL expression ?* Next call *Aug 2nd* Regards Steve Hanson Architect, IBM DFDL, Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> *tel:+44-1962-815848* <+44-1962-815848> ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: <michele.zundo(a)esa.int>*michele.zundo(a)esa.int* <michele.zundo(a)esa.int> This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: *michele.zundo(a)esa.int* <michele.zundo(a)esa.int> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases
by Mike Beckerle 13 Sep '16

13 Sep '16

Given the limited set of required encodings for a conforming DFDL processor, I believe dfdl:utf16Width='variable' should be an optional feature. That's just consistency with what is optional already. But it is also quite hard to implement. There are other situations that are very hard to implement, probably never used by real users, yet which are non optional in the spec: I would suggest that dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding like utf-8 is very problematic to implement. I am pretty sure IBM DFDL has no implementation of this per email threads, and I know I don't want to implement this in Daffodil even though we're trying to be very comprehensive in the implementation eventually. I think implementations should be free to just not implement this. These sorts of cases often exist just because we're trying to preserve some orthogonality of composition in the language. So it's possible to do quite a few things that probably aren't ever needed by anyone, that reflect ill-defined data formats, etc. I'd rather not document a bunch of "non-conformances" for Daffodil or other implementations for these sorts of things. I'd like to say we don't implement them, but they're optional, and so that's allowed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call 2016-08-30 - cancelled
by Steve Hanson 26 Aug '16

26 Aug '16

Steve not available so cancelling. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 26 Aug '16

26 Aug '16

Steve unable to make call

1 0

need clarification on unparsing with textPadKind='padChar'
by Mike Beckerle 23 Aug '16

23 Aug '16

The spec for textPadKind has this sentence about unparsing: "When dfdl:lengthKind is 'explicit' (and dfdl:length is an expression), 'delimited', 'prefixed', 'pattern' the data value is padded to the length given by the XSD minLength facet for type 'xs:string' or dfdl:textOutputMinLength property for other types." My concern is the case where dfdl:lengthKind is 'explicit' and dfdl:length is an expression. Example 1: Suppose my infoset element contains an element at path '../x' of type xs:string containing 10 characters "1234567890" Suppose the length expression evaluates to 20, and lengthUnits is 'characters' Suppose the XSD minLength facet is 100. Suppose the textPadKind is padChar. The above sentence says the target length would be 100 and we would pad to length 100, not 20, and there is no error. I want to be sure this is correct and what we intended. To me this seems inconsistent with what we have stated in Erratum 5.18 which says that whether or not the length is a constant or an expression, this element is treated as fixed length, and when unparsing, the length expression is evaluated, not ignored. It's not technically incorrect in erratum 5.18 because that doesn't deal with this minLength complexity. It's just not terribly clear. We may want to update this sentence of the spec, and the erratum to both use the term target length, which is in our glossary. Furthermore, In this case, the dfdl:contentLength for this element should return 100 for units 'characters'. In this case the dfdl:valueLength for this element should return 10 for units 'characters' Expression fn:string-length(../x) should return 10. The length expression value of 20 is effectively being ignored because it is less than the minLength. The length expression *would* be used if it was greater than the minLength. Example 2: My infoset contains an element at path "../x" of type xs:string of length 200 The length expression evaluates to 20 dfdl:truncateSpecifiedLengthString is true XSD minLength is 40. In this case I believe we would have a target length of 40, and we would truncate the string to length 40. dfdl:contentLength of this element is 40 dfdl:valueLength of this element is 40 Expression fn:string-length(../x) should return 200. Is this the correct interpretation? Example 3: Same as example 2, except XSD minLength is 0. Now the target length would be 20 and we would truncate the length to 20. dfdl:contentLength of this element is 20 characters dfdl:valueLength of this element is 20 characters. Expression fn:string-length(../x) should return 200. Is this the correct interpretation? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

OGF DFDL WG Call Minutes 2016-08-02
by Steve Hanson 03 Aug '16

03 Aug '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13538?download= Next call Aug 30th due to summer vacations Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Action 286 - Daffodil v IBM DFDL feature comparison
by Steve Hanson 03 Aug '16

03 Aug '16

Action 286 closed. 286 Create table that compares Daffodil features to IBM DFDL features (Steve/Mike) 16/2: Take table from section 22 of DFDL 1.0 spec and add columns showing implementations. 1/3: No progress 10/5: No progress. Work on this for next call. ... 7/6: No further progress 5/7: Steve has added column for IBM DFDL and emailed to WG. Mike will add column for Daffodil. 2/8: Closed. Mike has added column and discussion for Daffodil and emailed to WG. Steve to convert to an experience document and add to Redmine @ https://redmine.ogf.org/dmsf_files/13539?download=. Document can be updated as required. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 19/07/2016 20:13 Subject: Re: [DFDL-WG] Action 286 - Daffodil v IBM DFDL feature comparison Table with Daffodil column added, and additional details below. Feature Detection IBM DFDL Daffodil v2.0 Validation External switch Yes partial. Parsing only. Named Formats dfdl:defineFormat or dfdl:ref Yes Yes Choices xs:choice in xsd Yes Partial - choiceDispatchKey and choiceBranchKey not supported. Arrays where size not known in advance dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' Partial - not 'stopValue' Partial – not ‘stopValue’ Expressions Use of a DFDL expression in any property value Yes Yes. Limitation - query-style expressions not supported. These are expressions which require iteration over children of an element rather than constant-time access. For example fn:count(/a/b) where element 'a' has multiple element 'b' children separated by other element declarations. Hence, gathering all the b children requires iterating. End of parent dfdl:lengthKind "endOfParent" No No Simple type restrictions xs:simpleType in xsd Partial - not unions Partial – not unions Text representation for types other than String dfdl:representation "text" for Number, Calendar or Boolean types Yes Partial – type Boolean not implemented Delimiters dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or dfdl:lengthKind "delimited" Yes Yes Nils XSDL nillable 'true' in xsd Yes Partial - nilKind 'logicalValue' not supported Defaults XSDL default or fixed in xsd Partial - unparsing only Partial Simple types - ?? (Some support in parsing and in unparsing) Complex types – No Bi-Directional text. dfdl:textBiDi 'yes' No No Lengths in Bits dfdl:alignmentUnits 'bits' or dfdl:lengthUnits 'bits' Yes Yes Delimited lengths and representation binary element dfdl:representation 'binary' (or implied binary) and dfdl:lengthKind 'delimited' Yes Partial – delimiters must not use character class entities like %NL or %WSP*. Regular expressions dfdl:lengthKind 'pattern', dfdl:assert with dfdl:testkind 'pattern' , dfdl:discriminator with dfdl:testkind 'pattern' Partial - only for dfdl:lengthKind Yes Zoned numbers dfdl:textNumberRep 'zoned' Yes No IBM 390 packed numbers dfdl:binaryNumberRep 'packed' Yes No IBM 390 packed calendars dfdl:binaryCalendarRep 'packed' No No IBM 390 floats dfdl:binaryFloatRep 'ibm390Hex' Yes No Unordered sequences dfdl:sequenceKind 'unordered' Yes No Floating elements dfdl:floating 'yes' No No dfdl functions in expression language dfdl:functions in expression No Yes Hidden groups dfdl:hiddenGroupRef <> '' No Yes Calculated values dfdl:inputValueCalc <> '' or dfdl:outputValueCalc <> '' No Yes Escape schemes dfd:defineEscapeScheme in xsd Yes Yes Extended encodings Any dfdl:encoding value beyond the core list Yes Partial - iso-8859-1 works. Others - TBD. Asserts dfdl:assert in xsd Partial - not on global elements or simple types Yes Discriminators dfdl:discriminator in xsd Partial - not on global elements or simple types Yes Prefixed lengths dfdl:lengthKind 'prefixed' Partial - not nested prefixed lengths No Variables dfdl:defineVariable, dfdl:newVariableInstances, dfdl:setVariable Variables in DFDL expression language Note that variables as a feature is dependent on the Expressions feature. Partial - not newVariableinstance or external variables Partial – not newVariableInstance BCD calendars dfdl:binaryCalendarRep "bcd" Yes No BCD numbers dfdl:binaryNumberRep "bcd" Yes No Multiple schemas xs:include or xs:import in xsd Yes Yes IBM 4690 packed numbers dfdl:binaryNumberRep "ibm4690Packed" Yes No IBM 4690 packed calendars dfdl:binaryCalendarRep "ibm4690Packed" Yes No DFDL Byte Value Entities Use of %#r syntax in a DFDL String Literal other than the dfdl:fillByte property Yes No DFDL Standard Character Set Encodings dfdl:encoding name begins with "X-DFDL-". No Partial: X-DFDL-US-ASCII-7-BIT-PACKED, but not the 6-bit variant. Bit Order - Least Significant Bit First dfdl:bitOrder with value 'leastSignificantBitFirst' No Yes There is a list of implementation specific things not implemented for Daffodil also. The list needs updating, but includes at least: * xs:boolean - not supported * dfdl:ignoreCase - not supported * binary form for xs:date, xs:time, and xs:dateTime are not supported. * dfdl:utf16Width='variable' not supported * extended ICU symbols 'u' and 'I' in calendarPattern - not supported * dfdl:textStandardBase - not supported (behaves as 10) W.r.t, the additional you listed for IBM: fn:count(), fn:exactlyOne(), fn:name(), fn:local-name(), fn:namespace-uri() Daffodil has fn:count(), fn:exactlyOne(), but not the other 3. BOMs Daffodil does not yet support BOMs Correct distinction between 'empty' and 'missing' when parsing (action 140) No known limitation here - I would be curious as to a test that illustrates this limitation so we could try it on Daffodil Use of namespaces in expression paths Daffodil supports prefixed QNames in path steps. However, Daffodil's interpretation of NCNames in path steps does NOT currently take any default namespace into account. This makes Daffodil incompatible with IBM DFDL, which does not allow QNames, and so depends on those names being made meaningful by way of default namespace. Making Daffodil compatible with IBM's behavior here is Daffodil JIRA ticket https://opensource.ncsa.illinois.edu/jira/browse/DFDL-1446 dfdl:separatorSuppressionPolicy "trailingEmptyStrict" Daffodil supports this property value. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Mon, Jul 4, 2016 at 11:49 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: 286 Create table that compares Daffodil features to IBM DFDL features (Steve/Mike) 16/2: Take table from section 22 of DFDL 1.0 spec and add columns showing implementations. 1/3: No progress 10/5: No progress. Work on this for next call. ... 7/6: No further progress 5/7: To kick this off here is the optional features table from spec section 21 with a column added for IBM DFDL Feature Detection IBM DFDL Validation External switch Yes Named Formats dfdl:defineFormat or dfdl:ref Yes Choices xs:choice in xsd Yes Arrays where size not known in advance dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' Partial - not 'stopValue' Expressions Use of a DFDL expression in any property value Yes End of parent dfdl:lengthKind "endOfParent" No Simple type restrictions xs:simpleType in xsd Partial - not unions Text representation for types other than String dfdl:representation "text" for Number, Calendar or Boolean types Yes Delimiters dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or dfdl:lengthKind "delimited" Yes Nils XSDL nillable 'true' in xsd Yes Defaults XSDL default or fixed in xsd Partial - unparsing only Bi-Directional text. dfdl:textBiDi 'yes' No Lengths in Bits dfdl:alignmentUnits 'bits' or dfdl:lengthUnits 'bits' Yes Delimited lengths and representation binary element dfdl:representation 'binary' (or implied binary) and dfdl:lengthKind 'delimited' Yes Regular expressions dfdl:lengthKind 'pattern', dfdl:assert with dfdl:testkind 'pattern' , dfdl:discriminator with dfdl:testkind 'pattern' Partial - only for dfdl:lengthKind Zoned numbers dfdl:textNumberRep 'zoned' Yes IBM 390 packed numbers dfdl:binaryNumberRep 'packed' Yes IBM 390 packed calendars dfdl:binaryCalendarRep 'packed' No IBM 390 floats dfdl:binaryFloatRep 'ibm390Hex' Yes Unordered sequences dfdl:sequenceKind 'unordered' Yes Floating elements dfdl:floating 'yes' No dfdl functions in expression language dfdl:functions in expression No Hidden groups dfdl:hiddenGroupRef <> '' No Calculated values dfdl:inputValueCalc <> '' or dfdl:outputValueCalc <> '' No Escape schemes dfd:defineEscapeScheme in xsd Yes Extended encodings Any dfdl:encoding value beyond the core list Yes Asserts dfdl:assert in xsd Partial - not on global elements or simple types Discriminators dfdl:discriminator in xsd Partial - not on global elements or simple types Prefixed lengths dfdl:lengthKind 'prefixed' Partial - not nested prefixed lengths Variables dfdl:defineVariable, dfdl:newVariableInstances, dfdl:setVariable Variables in DFDL expression language Note that variables as a feature is dependent on the Expressions feature. Partial - not newVariableinstance or external variables BCD calendars dfdl:binaryCalendarRep "bcd" Yes BCD numbers dfdl:binaryNumberRep "bcd" Yes Multiple schemas xs:include or xs:import in xsd Yes IBM 4690 packed numbers dfdl:binaryNumberRep "ibm4690Packed" Yes IBM 4690 packed calendars dfdl:binaryCalendarRep "ibm4690Packed" Yes DFDL Byte Value Entities Use of %#r syntax in a DFDL String Literal other than the dfdl:fillByte property Yes DFDL Standard Character Set Encodings dfdl:encoding name begins with "X-DFDL-". No Bit Order - Least Significant Bit First dfdl:bitOrder with value 'leastSignificantBitFirst' No Some IBM DFDL restrictions are not captured in the table, mainly due to non-implementation of many of the later errata. Full details here https://www.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mf… but the key omissions are: fn:count(), fn:exactlyOne(), fn:name(), fn:local-name(), fn:namespace-uri() BOMs Correct distinction between 'empty' and 'missing' when parsing (action 140) Use of namespaces in expression paths dfdl:separatorSuppressionPolicy "trailingEmptyStrict" Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0