- dfdl-wg - lists.ogf.org

clarification on behavior of DFDL encodingErrorPolicy='error' and pre-decoding by implementations
by Mike Beckerle 06 Dec '18

06 Dec '18

The DFDL spec isn't clear on when encodingErrorPolicy 'error' is allowed to cause an error, and when one must be suppressed, if the implementation pre-decodes data into characters. Example: Suppose you have what turns out to be 8 characters of text, followed by some binary data. Suppose a DFDL implementation happens to always try to fill a buffer of 64 decoded characters, just for efficiency reasons. Depending on what is in the binary data, this may parse the 8 characters of text without error, but subsequently hit a decode error, because it has strayed into binary data past the text. There is no actual decode error in the data stream, because parsing should determine there are only 8 characters of text, and then switch to parsing the binary data using binary means. The DFDL spec doesn't say this isn't allowed to cause a decode error. Perhaps it is implied somewhere? But I didn't find it. The DFDL spec does point out that for asserts/discriminators with testKind pattern, that pattern matching may cause decode errors. But again, suppose the regex matching library an implementation uses happens to pre-fetch and pre-decode a bunch of characters, but the regex matching library then finds a match that is quite short, and stops well before the characters that were pre-decoded that caused a decode error. It would seem to me that this sort of pre-decoding should not cause decode errors. but the DFDL spec doesn't state that explicitly. comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Latest OGF DFDL WG Call Minutes
by Steve Hanson 30 Nov '18

30 Nov '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Note extra call next Thursday 6th December @ 16:00 UK Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Extra OGF DFDL WG call
by Steve Hanson 30 Nov '18

30 Nov '18

Extra Call. Same dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 Netherlands Toll-Free 0-800-363-6036 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg DFDL Getting Started: http://www.ibm.com/developerworks/library/se-dfdl/ DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf

1 0

Agenda for today's OGF DFDL WG Call
by Steve Hanson 29 Nov '18

29 Nov '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13583?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Action 242 - Progress - Single TDML format - Can cross-test now.
by Mike Beckerle 21 Nov '18

21 Nov '18

Good progress to report on TDML and on creating tools so we can do interoperability testing more easily and push for our demonstration of 2-interoperable implementations. I am able to run TDML test suites against IBM DFDL now from the command-line/sbt. The ibmCrossTestRig is not part of Daffodil (because it links against IBM DFDL), but is open source Apache License v2, and is currently in review at: https://github.com/OpenDFDL/ibmDFDLCrossTester/pull/1 This may not be its permanent home, but for now that's where it is. There's a README.txt on how to set it up. It requires that one obtain IBM DFDL, and put the jars in lib, etc. This requires a daffodil 2.3.0 dev branch snapshot build, as the guts of the TDML runner are in one of daffodil's jars, and that depends on daffodil's IO and other libraries for synthesizing and comparing test data, etc. In fact as of this writing, the daffodil changes to support this testing rig are still also in PR review at https://github.com/apache/incubator-daffodil/pull/140 But we hope to have them on the main development 2.3.0 branch in about a week, and an official release within a month or so. Individual TDML parser/unparser test cases, or whole TDML test suites can be marked with what implementations they should be run against. A few tests are built in to the test rig. These will drive the IBM DFDL samples that come with IBM DFDL from TDML as tests. The tests in daffodil's daffodil-test-ibm1 module are the first target for cross-testing. You basically modify one file (build.sbt) to point the test rig at suites of tests to run. These tests are mostly set to run on both daffodil and IBM DFDL, but some are daffodil-only because they use computed values or hidden groups. Those tests (18 of them) get skipped if you run against IBM DFDL. This little test suite includes most of the 70+ tests that were originally authored by IBM, and given to the "world" way back when as part of DFDL working-group promotion of TDML. There are also around 30 tests that were part of the original Univ. of Illinois NCSA Daffodil project when it was first started. We don't currently know if these tests are still "as were contributed" or through the years we've tweeked them to get them to work on Daffodil. But in any case, they're a useful set to start from. Current state of portability of this first small suite of 112 tests is this JUnit summary: [error] Failed: Total 112, Failed 15, Errors 0, Passed 79, Skipped 18 That is, of 112 tests, 18 are skipped. 15 failed (are not portable, but we think they should be). These need some analysis. We've already created JIRA tickets for some of these non-portability issues. They need analysis to see what the proper behavior is, and which implementation is correct or if there really is ambiguity/laxity and both are possibly correct. https://issues.apache.org/jira/browse/DAFFODIL-2018 https://issues.apache.org/jira/browse/DAFFODIL-2017 The above include separator/terminator issues, and a number of issues with date/time formats and when "Z" vs "+00:00" are to be used, and when fractional seconds are supposed to be in the infoset, etc. We also already have a test which failed on IBM DFDL, but in fact that is correct behavior. This was the one where Daffodil was making the test work by inserting coercions that it shouldn't have. https://issues.apache.org/jira/browse/DAFFODIL-2021 So we're starting to get some initial benefits from the cross testing already. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

clarification needed: conversions to required type of expressions - implicit or explicit
by Mike Beckerle 21 Nov '18

21 Nov '18

We have a test for Daffodil that does this: <element name="y" type="xsd:boolean" maxOccurs="unbounded" dfdl:textBooleanTrueRep="{ 1*10 }" dfdl:textBooleanFalseRep="{'a b c'}" /> We found that we must change this to: dfdl:textBooleanTrueRep="{ xsd:string(1*10) }" For the test to be portable and work on IBM DFDL as well as Daffodil. So the issue to discuss is, are string conversions, or in general conversions to the required property's type (e.g., integer for dfdl:length), are these implicit, or do they have to be explicit? I didn't find language in the DFDL spec that makes this clear. The properties tend to be clear on what the type needs to be (string or integer typically). The XPath language that is what we derive DFDL's expression language from, has lots of conversion/coercion rules. Everything can be implicitly converted to a string. We say our DFDL expression language is a subset of XPath 2.0, and that we explicitly say we use XPath 2.0's type system and respect its errors, and we say DFDL implementations may use XPath 2.0 implementations if they do some pre-processing. So the clarification is reduced to this: Does an entire DFDL expression for the value of a property behave as if it was an actual argument to a function, that formal argument having the type required by the property? In that case coercions would be inserted to convert say, numbers to strings. Alternatively, does the expression for a property have to explicitly convert to the type that the property requires? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy

2 2

Action 292 - Write up of hexBinary with lengthUnits 'bits'
by Mike Beckerle 20 Nov '18

20 Nov '18

The proposed change is to allow lengthUnits 'bits' for hexBinary data. This turned out to be more complex to describe than I originally suspected because of the need to deal with XSD minLength and maxLength facets, which are always measured in bytes. Those in conjunction with dfdl:lengthKind='explicit' and dfdl:lengthUnits='bits' create some minor complexities. The below changes match the Daffodil implementation of this proposed feature. These sentences in the description of dfdl:lengthUnits in Section 12.3 must change. - 'bits' may only be used for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt, and xs:unsignedLong simple types with binary representation. - 'bytes' must be used for type xs:hexBinary. The text should read - 'bits' may only be used for xs:hexBinary and for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt, and xs:unsignedLong simple types with binary representation. Later in the section, the paragraph: When unparsing a simple element with binary representation, then for hexBinary the length is the number of bytes in the infoset value padded to the XSD minLength facet value using dfdl:fillByte, and for the other types the length is the minimum number of bytes to represent the value and any sign. Must change to: When unparsing a simple element with binary representation, then for types other than hexBinary the length is the minimum number of bytes to represent the value and any sign. For type hexBinary when the dfdl:lengthUnits is 'bytes' then the length is the number of bytes in the infoset value padded to the XSD minLength facet value using dfdl:fillByte. For type hexBinary when the dfdl:lengthUnits is 'bits': - First the data is padded to XSD minLength bytes as if the dfdl:lengthUnits was 'bytes'. - When dfdl:lengthKind is other than 'explicit', the length is the number of bytes times 8. - When the dfdl:lengthKind is 'explicit' then the value is further padded or truncated to fit the target length, in bits. - if the data does not have sufficient bytes to supply the target length in bits it is a processing error. - if the data is longer than the minimum number of bytes needed to supply the target length in bits, it is a processing error. - If the explicit length in bits is not a multiple of 8, then the final byte is only partially unparsed according to the current dfdl:byteOrder. Section 12.3.7.2.7 Length of Binary Opaque Elements, the first sentence must be modified from: "The dfdl:lengthUnits property must be 'bytes'. It is a schema definition error otherwise." to "The dfdl:lengthUnits property must be 'bytes' or 'bits'. It is a schema definition error otherwise. Note that even when the dfdl:lengthUnits property is 'bits', the values of the XSD minLength and XSD maxLength facets are still always interpreted as constraining the length in units of bytes. That's the end of the actual proposed language. Note about alternatives: I considered the alternative to make it a schema definition error when dfdl:lengthUnits is 'bits' type is 'xs:hexBinary', and the XSD:minLength or XSD:maxLength facets are defined. I decided to go with the description above to support the use case where a data item is, in hex, some number of bytes long for a valid XML infoset, but the representation is explicitly a partial byte smaller. E.g., in a hexBinary with 8 bytes, but fewer than 64 bits in the representation. Ex: 63 bits as the explicit target length, so 1 bit will be unused from the xs:hexBinary logical value, but the above rules insure no more than 7 bits go unused from the final byte of the hexBinary logical value. This is trying to be consistent with the notion that we do not truncate data to fit into the available length except for xs:string when properties explicitly allow it. It is also trying to be consistent with our treatment of binary integers where a xs:long value may be output into an element having room for any number of bits, and any extra bits in the logical value are ignored. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Latest OGF DFDL WG Call Minutes
by Steve Hanson 15 Nov '18

15 Nov '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for today's OGF DFDL WG Call
by Steve Hanson 15 Nov '18

15 Nov '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13582?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Fw: Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases (action 290)
by Steve Hanson 15 Nov '18

15 Nov '18

Hi Mike I've been looking into IBM DFDL's treatment of property dfdl:utf16Width. While we claim to support 'variable' and have a few tests that use this, there is not the number of tests that I would expect to test the property fully. The intent to support 'variable' is clear in the code, though; for example, when parsing we check each char for being part of a surrogate pair and adjust length accordingly. The code uses java.nio.charset for its encoders & decoders, which we wrap in our own class which notes whether utf16 is fixed or variable, but this information is not passed to the encoder/decoder as there is no way to do so. Hmm. We will add some more tests and see if everything is behaving. Back to your original question, should 'variable' be an optional feature of the spec. I have discussed with implementation team members and we think that is a sensible thing to do. To handle surrogates does require extra code to be written, and for a minimal implementation it should not be necessary to do that. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 04/04/2017 11:56 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Some light on action 291 - see the last sentence of this extract from the original errata document (experience doc 1): 3.9. Section 12.3.5, 7.3.1, 7.3.2. The spec originally allows lengthKind ‘pattern’ to be used when the representation of the current element, or of a child element, is binary, but imposes restrictions on the encoding that can be in force. Clarify that the encoding property must be defined for the element (else schema definition error), and that a decoding processing error is possible if the match of the regex encounters data that does not decode in that encoding, dependent on the setting of encodingErrorPolicy. Remove section 12.3.5.1. Same clarifications needed for testKind ”pattern” property for asserts and discriminators. For consistency, the restriction that a complex element of specified length and lengthUnits ‘characters’ must have children that are all text and that have the same encoding as the complex element, is dropped So that explains how IBM DFDL's error message CTDV1524E came about, it was policing a restriction in the original GFD.174 spec, a restriction which no longer exists. IBM DFDL has not yet implemented the erratum. It wasn't an extra IBM restriction. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 14/09/2016 08:44 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Actions 290 and 291 raised to investigate further - see minutes. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/09/2016 13:14 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Mike I am assuming that the processing for utf-16 'fixed' or 'variable' is entirely handled by ICU so there should be no coding overhead. IBM DFDL works ok for dfdl:lengthKind='explicit' for an element of complex type with dfdl:lengthUnits='characters' and dfdl:encoding="utf-8". But there are conditions the content of the complex type must satisfy otherwise an SDE results, such as: CTDV1524E : For a complex element, when 'lengthKind' is 'explicit' or 'prefixed', and 'lengthUnits' is characters, all simple child elements must have text representation, 'lengthUnits' set to 'characters' and the same encoding. So we insist that the properties of the children are consistent with the properties of the parent. If you recall, IBM DFDL does all these kinds of validation checks in a pre-processing phase. That seems a pretty sensible rule but I am not sure if the rule appears in the spec as such - I just had a quick look but didn't spot anything. So I guess I don't see a need for these things to be optional features? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 10/08/2016 18:57 Subject: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Given the limited set of required encodings for a conforming DFDL processor, I believe dfdl:utf16Width='variable' should be an optional feature. That's just consistency with what is optional already. But it is also quite hard to implement. There are other situations that are very hard to implement, probably never used by real users, yet which are non optional in the spec: I would suggest that dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding like utf-8 is very problematic to implement. I am pretty sure IBM DFDL has no implementation of this per email threads, and I know I don't want to implement this in Daffodil even though we're trying to be very comprehensive in the implementation eventually. I think implementations should be free to just not implement this. These sorts of cases often exist just because we're trying to preserve some orthogonality of composition in the language. So it's possible to do quite a few things that probably aren't ever needed by anyone, that reflect ill-defined data formats, etc. I'd rather not document a bunch of "non-conformances" for Daffodil or other implementations for these sorts of things. I'd like to say we don't implement them, but they're optional, and so that's allowed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0