- dfdl-wg - lists.ogf.org

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 12 Dec '18

12 Dec '18

1 0

Ambiguity or Correct ? Behavior of time zone in infoset for xs:date
by Mike Beckerle 11 Dec '18

11 Dec '18

I have a case where IBM DFDL and Daffodil differ, but I think both are correct so this is just a case where users have to tolerate variation in behavior: <dfdl:format calendarPatternKind="implicit" calendarTimeZone="UTC" /> <xs:element name="date1" type="xs:date" /> The data is "2010-12-30" The DFDL spec says the parser behavior is as for pattern "yyyy-MM-dd", which does not have any character for time zone information. So the time zone information is coming from the dfdl:calendarTimeZone="UTC" property. But this question isn't about the parsing/unparsing behavior. It's about what date/time components end up in the infoset and how those get projected into XML by different DFDL implementations. The problem is that there are multiple equivalent (according to XML/XSD and ISO 8601) representations in the infoset for this information. Ex: <date1>2010-12-30+00:00</date1> (Note: current Daffodil behavior) or <date1>2010-12-30Z</date1> (Note: current IBM DFDL behavior) I believe these are 100% equivalent according to ISO8601 which is the standard for date/time referenced from XML/XSD. If this is acceptable then there's no action here. It's just an equivalence that users have to expect. So I'm just raising this to get opinions. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Action 292 - version 2 proposal for hexBinary with lengthUnits bits
by Mike Beckerle 07 Dec '18

07 Dec '18

Users want a way to express an arbitrary unaligned string of bits, with the appearance in the infoset being hexadecimal, not base 10. Right now the only way I can see to meet this requirement while retaining backward compatibility would be a new DFDL property. So here's the new idea: Property dfdl:hexBinaryRep with values 'bytes' or 'bits'. New property, so defaulting (with suppressible warning) to 'bytes' for backward compatibility in schemas not having the property. When set to 'bits', then type xs:hexBinary would behave just like xs:nonNegativeInteger, and all properties relevant to that type would be applicable, and any use of XSD length facets on such elements would be an SDE. The hexBinary string would be exactly same as if you took the numeric value for a nonNegativeInteger and instead of presenting it as base 10 digits, you use base 16 digits. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 4

Latest OGF DFDL WG Call Minutes
by Steve Hanson 07 Dec '18

07 Dec '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

clarification on behavior of DFDL encodingErrorPolicy='error' and pre-decoding by implementations
by Mike Beckerle 06 Dec '18

06 Dec '18

The DFDL spec isn't clear on when encodingErrorPolicy 'error' is allowed to cause an error, and when one must be suppressed, if the implementation pre-decodes data into characters. Example: Suppose you have what turns out to be 8 characters of text, followed by some binary data. Suppose a DFDL implementation happens to always try to fill a buffer of 64 decoded characters, just for efficiency reasons. Depending on what is in the binary data, this may parse the 8 characters of text without error, but subsequently hit a decode error, because it has strayed into binary data past the text. There is no actual decode error in the data stream, because parsing should determine there are only 8 characters of text, and then switch to parsing the binary data using binary means. The DFDL spec doesn't say this isn't allowed to cause a decode error. Perhaps it is implied somewhere? But I didn't find it. The DFDL spec does point out that for asserts/discriminators with testKind pattern, that pattern matching may cause decode errors. But again, suppose the regex matching library an implementation uses happens to pre-fetch and pre-decode a bunch of characters, but the regex matching library then finds a match that is quite short, and stops well before the characters that were pre-decoded that caused a decode error. It would seem to me that this sort of pre-decoding should not cause decode errors. but the DFDL spec doesn't state that explicitly. comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Latest OGF DFDL WG Call Minutes
by Steve Hanson 30 Nov '18

30 Nov '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Note extra call next Thursday 6th December @ 16:00 UK Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Extra OGF DFDL WG call
by Steve Hanson 30 Nov '18

30 Nov '18

Extra Call. Same dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 Netherlands Toll-Free 0-800-363-6036 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg DFDL Getting Started: http://www.ibm.com/developerworks/library/se-dfdl/ DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf

1 0

Agenda for today's OGF DFDL WG Call
by Steve Hanson 29 Nov '18

29 Nov '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13583?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Action 242 - Progress - Single TDML format - Can cross-test now.
by Mike Beckerle 21 Nov '18

21 Nov '18

Good progress to report on TDML and on creating tools so we can do interoperability testing more easily and push for our demonstration of 2-interoperable implementations. I am able to run TDML test suites against IBM DFDL now from the command-line/sbt. The ibmCrossTestRig is not part of Daffodil (because it links against IBM DFDL), but is open source Apache License v2, and is currently in review at: https://github.com/OpenDFDL/ibmDFDLCrossTester/pull/1 This may not be its permanent home, but for now that's where it is. There's a README.txt on how to set it up. It requires that one obtain IBM DFDL, and put the jars in lib, etc. This requires a daffodil 2.3.0 dev branch snapshot build, as the guts of the TDML runner are in one of daffodil's jars, and that depends on daffodil's IO and other libraries for synthesizing and comparing test data, etc. In fact as of this writing, the daffodil changes to support this testing rig are still also in PR review at https://github.com/apache/incubator-daffodil/pull/140 But we hope to have them on the main development 2.3.0 branch in about a week, and an official release within a month or so. Individual TDML parser/unparser test cases, or whole TDML test suites can be marked with what implementations they should be run against. A few tests are built in to the test rig. These will drive the IBM DFDL samples that come with IBM DFDL from TDML as tests. The tests in daffodil's daffodil-test-ibm1 module are the first target for cross-testing. You basically modify one file (build.sbt) to point the test rig at suites of tests to run. These tests are mostly set to run on both daffodil and IBM DFDL, but some are daffodil-only because they use computed values or hidden groups. Those tests (18 of them) get skipped if you run against IBM DFDL. This little test suite includes most of the 70+ tests that were originally authored by IBM, and given to the "world" way back when as part of DFDL working-group promotion of TDML. There are also around 30 tests that were part of the original Univ. of Illinois NCSA Daffodil project when it was first started. We don't currently know if these tests are still "as were contributed" or through the years we've tweeked them to get them to work on Daffodil. But in any case, they're a useful set to start from. Current state of portability of this first small suite of 112 tests is this JUnit summary: [error] Failed: Total 112, Failed 15, Errors 0, Passed 79, Skipped 18 That is, of 112 tests, 18 are skipped. 15 failed (are not portable, but we think they should be). These need some analysis. We've already created JIRA tickets for some of these non-portability issues. They need analysis to see what the proper behavior is, and which implementation is correct or if there really is ambiguity/laxity and both are possibly correct. https://issues.apache.org/jira/browse/DAFFODIL-2018 https://issues.apache.org/jira/browse/DAFFODIL-2017 The above include separator/terminator issues, and a number of issues with date/time formats and when "Z" vs "+00:00" are to be used, and when fractional seconds are supposed to be in the infoset, etc. We also already have a test which failed on IBM DFDL, but in fact that is correct behavior. This was the one where Daffodil was making the test work by inserting coercions that it shouldn't have. https://issues.apache.org/jira/browse/DAFFODIL-2021 So we're starting to get some initial benefits from the cross testing already. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

clarification needed: conversions to required type of expressions - implicit or explicit
by Mike Beckerle 21 Nov '18

21 Nov '18

We have a test for Daffodil that does this: <element name="y" type="xsd:boolean" maxOccurs="unbounded" dfdl:textBooleanTrueRep="{ 1*10 }" dfdl:textBooleanFalseRep="{'a b c'}" /> We found that we must change this to: dfdl:textBooleanTrueRep="{ xsd:string(1*10) }" For the test to be portable and work on IBM DFDL as well as Daffodil. So the issue to discuss is, are string conversions, or in general conversions to the required property's type (e.g., integer for dfdl:length), are these implicit, or do they have to be explicit? I didn't find language in the DFDL spec that makes this clear. The properties tend to be clear on what the type needs to be (string or integer typically). The XPath language that is what we derive DFDL's expression language from, has lots of conversion/coercion rules. Everything can be implicitly converted to a string. We say our DFDL expression language is a subset of XPath 2.0, and that we explicitly say we use XPath 2.0's type system and respect its errors, and we say DFDL implementations may use XPath 2.0 implementations if they do some pre-processing. So the clarification is reduced to this: Does an entire DFDL expression for the value of a property behave as if it was an actual argument to a function, that formal argument having the type required by the property? In that case coercions would be inserted to convert say, numbers to strings. Alternatively, does the expression for a property have to explicitly convert to the type that the property requires? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy

2 2