September 2016 - dfdl-wg

Announcement: Daffodil 2.0.0 Release Candidate 1 - Release to Test
by Steve Lawrence 29 Sep '16

29 Sep '16

We are pleased to announce the first release candidate of Daffodil 2.0.0! Notable changes since Daffodil 1.1.0 include unparse achieving full feature parity with parse, support for additional unparse only properties such as outputValueCalc, fillByte, truncateSpecifiedLengthString, etc., numerous bug fixes in parse, and improved parse performance. The release candidate has been tagged as 2.0.0-rc1 in the git repo: https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil… The release is also available for downloaded from the Daffodil Artifacts page: https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&versio… Daffodil will now enter the test phase, which we expect to last approximately 3 weeks, followed by the official release of Daffodil 2.0.0. Please give the release candidate a test and let us know if you find any issues. To report bugs or ask any questions, the development team is available on the Daffodil users mailing list: http://oss.tresys.com/mailman/listinfo/daffodil-users as well as XMPP HipChat: http://hipchat.ncsa.illinois.edu/gvZdmJHmq or via the JIRA Daffodil bug tracker: https://opensource.ncsa.illinois.edu/jira/projects/DFDL/summary On behalf of the Daffodil team, - Steve

3 2

OGF DFDL WG Call 2016-09-27 - cancelled
by Steve Hanson 27 Sep '16

27 Sep '16

Steve not available so cancelling. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 26 Sep '16

26 Sep '16

Unable to make DFDL call this week so cancelling

1 0

Re: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group
by Steve Hanson 14 Sep '16

14 Sep '16

...and also missed this one. Next call ... meantime if you agree with my conclusion then please feel free to compile some extra words for the spec. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 07/09/2016 10:28 Subject: Re: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group I agree. Btw the language was added by several erratum so is fairly new. Perhaps what is missing is some words in the spec that state the resolved set of format annotation properties for a simple element describe SimpleElement in the data syntax grammar, and so on. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 06/09/2016 20:47 Subject: Re: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group Perhaps this is a minor clarification I need. Perhaps this is obvious. The language you excerpted probably pre-dates the data syntax grammar in the spec. What grammar regions does point (4) " dfdl:sequence or dfdl:choice or dfdl:group following property scoping rules" apply to? dfdl:group for group references are non-issue since those turn into sequence or choice. I think the answer is perhaps, obvious in retrospect.: it is the Sequence or Choice regions, and for simple types it is SimpleElement (not SimpleContent) for complex elements it is ComplexElement (not ComplexContent). That way it is clear that in every case, framing, alignment regions etc., are included. So for sequences, what textually appears above/before the dfdl:setVariable statement is framing properties of the sequence (in short form, or in long form if the dfdl:sequence element is placed above the setVariable statemnts). Those can use runtime-valued expressions which refer to the variable, and will see the variable after the set has executed. This is disconcerting only because of the lexical ordering mis-match with the order of execution. However, the same is true for asserts and discriminators using expressions. Those execute after the setVariables, no matter where they appear in lexical order vis a vis the setVariable statement annotations. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 6, 2016 at 12:58 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: Mike >From section 9.5: For sequences, choices and group refs: 1. dfdl:discriminator or dfdl:assert(s) with testKind 'pattern' (parsing only) 2. dfdl:newVariableInstance(s) - in lexical order, innermost schema component first 3. dfdl:setVariable(s) - in lexical order, innermost schema component first 4. dfdl:sequence or dfdl:choice or dfdl:group following property scoping rules 5. dfdl:discriminator or dfdl:assert(s) with testKind 'expression' (parsing only) This says that a setVariable on a group is processed before the format annotations of the group, so your example would be allowed. For discriminators and asserts, when they are executed depends of the testKind. If 'pattern' they are executed first. If 'expression' they are executed last. All seems clear to me - am I missing some subtlety somewhere? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 02/09/2016 16:56 Subject: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> The spec defines where setVariable executes as a statement relative to element processing and model-group processing. Question is: what does element and model group processing mean: the value region of the element, the content region of the element, or the whole element including all framing? So, for an element, spec says that setVariables run after the element. Rhetorical question: Does "after" mean after the terminator (including any expression for the terminator) has completed? Yes, because otherwise a element could contain the value of its own terminator, and that wouldn't work for lengthKind='delimited' as you would need the value to get the terminator to parse the value. So 'after' has to mean after the terminator, meaning all framing, and any other expression that can affect the length or interpretation in forming the element. For a model group, the spec says that setVariables run before the model group. So, can a sequence carry a set-variable that sets a variable value that is then used in the initiator property of that very sequence? E.g., <xs:sequence dfdl:initiator="{ $init }"> <xs:annotation><xs:appinfo ...> <dfdl:setVariable ref="init">{ ../someplace/else }</dfdl:setVariable> </xs:appinfo></xs:annotation> ..... reset of sequence </xs:sequence> If this is not allowed for the initiator, what about for the terminator or separator, or outputNewLine or encoding or any other property that affects the syntax of the model group? What about discriminators/asserts. Do those run before the sequence's initiator is matched, or after? E.g., can a discriminator indicate "true", but then the initiator fails to match? When unparsing, you don't execute discriminator/assert at all, so no guidance there. Maybe this is clear already, but I am not sure if setVariables should run before any framing of a model group, or after the framing, but before the content region, or before the value region. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Correction to 12.3.7.1 Length of Simple Elements with Textual Representation
by Mike Beckerle 14 Sep '16

14 Sep '16

This section says "For a textual element, the dfdl:lengthUnits property can be either 'bytes' or 'characters'." I think this statement is just old. It predates the addition of 7-bit and 6-bit characters to DFDL, and the notion that the width of characters might not be a multiple of 8-bits. This section can be removed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

minor spec correction Section 5.2
by Mike Beckerle 14 Sep '16

14 Sep '16

In Section 5.2 The phrase "pattern (for type xs:string and all types descending from xs:string in Figure 3 DFDL simple types)" The entire parenthesized part of that should be dropped, as in that figure it is clear that DFDL does not have any types that are decending from xs:string. So this is just misleading. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Re: [DFDL-WG] need clarification on unparsing with textPadKind='padChar'
by Steve Hanson 14 Sep '16

14 Sep '16

Mike Apologies - missed this when compiling the agenda for yesterday's call - will add it to agenda for next call. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 23/08/2016 13:19 Subject: Re: [DFDL-WG] need clarification on unparsing with textPadKind='padChar' Erratum 5.18 is not yet incorporated into the spec. When it is then dfdl:lengthKind 'explicit' with an expression becomes the same as dfdl:lengthKind 'explicit' with a literal. I believe that means the expression result is the fixed length for unparsing, and minLength is only used for validation. However I would like to run a test with IBM DFDL to make sure everything is as expected. fn:string-length() always returns the length of the value in the infoset item. It has no schema awareness. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 10/08/2016 17:27 Subject: [DFDL-WG] need clarification on unparsing with textPadKind='padChar' Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> The spec for textPadKind has this sentence about unparsing: "When dfdl:lengthKind is 'explicit' (and dfdl:length is an expression), 'delimited', 'prefixed', 'pattern' the data value is padded to the length given by the XSD minLength facet for type 'xs:string' or dfdl:textOutputMinLength property for other types." My concern is the case where dfdl:lengthKind is 'explicit' and dfdl:length is an expression. Example 1: Suppose my infoset element contains an element at path '../x' of type xs:string containing 10 characters "1234567890" Suppose the length expression evaluates to 20, and lengthUnits is 'characters' Suppose the XSD minLength facet is 100. Suppose the textPadKind is padChar. The above sentence says the target length would be 100 and we would pad to length 100, not 20, and there is no error. I want to be sure this is correct and what we intended. To me this seems inconsistent with what we have stated in Erratum 5.18 which says that whether or not the length is a constant or an expression, this element is treated as fixed length, and when unparsing, the length expression is evaluated, not ignored. It's not technically incorrect in erratum 5.18 because that doesn't deal with this minLength complexity. It's just not terribly clear. We may want to update this sentence of the spec, and the erratum to both use the term target length, which is in our glossary. Furthermore, In this case, the dfdl:contentLength for this element should return 100 for units 'characters'. In this case the dfdl:valueLength for this element should return 10 for units 'characters' Expression fn:string-length(../x) should return 10. The length expression value of 20 is effectively being ignored because it is less than the minLength. The length expression *would* be used if it was greater than the minLength. Example 2: My infoset contains an element at path "../x" of type xs:string of length 200 The length expression evaluates to 20 dfdl:truncateSpecifiedLengthString is true XSD minLength is 40. In this case I believe we would have a target length of 40, and we would truncate the string to length 40. dfdl:contentLength of this element is 40 dfdl:valueLength of this element is 40 Expression fn:string-length(../x) should return 200. Is this the correct interpretation? Example 3: Same as example 2, except XSD minLength is 0. Now the target length would be 20 and we would truncate the length to 20. dfdl:contentLength of this element is 20 characters dfdl:valueLength of this element is 20 characters. Expression fn:string-length(../x) should return 200. Is this the correct interpretation? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases
by Steve Hanson 14 Sep '16

14 Sep '16

Actions 290 and 291 raised to investigate further - see minutes. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/09/2016 13:14 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Mike I am assuming that the processing for utf-16 'fixed' or 'variable' is entirely handled by ICU so there should be no coding overhead. IBM DFDL works ok for dfdl:lengthKind='explicit' for an element of complex type with dfdl:lengthUnits='characters' and dfdl:encoding="utf-8". But there are conditions the content of the complex type must satisfy otherwise an SDE results, such as: CTDV1524E : For a complex element, when 'lengthKind' is 'explicit' or 'prefixed', and 'lengthUnits' is characters, all simple child elements must have text representation, 'lengthUnits' set to 'characters' and the same encoding. So we insist that the properties of the children are consistent with the properties of the parent. If you recall, IBM DFDL does all these kinds of validation checks in a pre-processing phase. That seems a pretty sensible rule but I am not sure if the rule appears in the spec as such - I just had a quick look but didn't spot anything. So I guess I don't see a need for these things to be optional features? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 10/08/2016 18:57 Subject: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Given the limited set of required encodings for a conforming DFDL processor, I believe dfdl:utf16Width='variable' should be an optional feature. That's just consistency with what is optional already. But it is also quite hard to implement. There are other situations that are very hard to implement, probably never used by real users, yet which are non optional in the spec: I would suggest that dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding like utf-8 is very problematic to implement. I am pretty sure IBM DFDL has no implementation of this per email threads, and I know I don't want to implement this in Daffodil even though we're trying to be very comprehensive in the implementation eventually. I think implementations should be free to just not implement this. These sorts of cases often exist just because we're trying to preserve some orthogonality of composition in the language. So it's possible to do quite a few things that probably aren't ever needed by anyone, that reflect ill-defined data formats, etc. I'd rather not document a bunch of "non-conformances" for Daffodil or other implementations for these sorts of things. I'd like to say we don't implement them, but they're optional, and so that's allowed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2016-09-13
by Steve Hanson 13 Sep '16

13 Sep '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13541?download= Next call Sept 26th Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

DFDL4S use of wildcard/regex in length path expression
by Mike Beckerle 13 Sep '16

13 Sep '16

(I edited the subject line for this discussion.) Michele, Could you send the parts of the DFDL4S schema that use the types that have these wildcard/regex in them? Also, please send as a zip file, not a jar. Google, in its infinite wisdom, has decided that it is better to save me from a potential computer virus than to let me get my work done, and they block all access to attachments that are jar files, you can't rename them, save to Drive, save to local file system, nor forward them to a different email account. I want to see exactly what this wildcard is doing for you. I suspect it is just allowing these types to be reused in contexts where otherwise you'd have to have separate complex type definitions just because this one step of the path is different at the different points of re-use. I want to really think about whether there is some other way to achieve this same reusability, or if not, just how bad is the workaround. ...mike beckerle ------------------------------ *From:* Steve Hanson [smh(a)uk.ibm.com] *Sent:* Tuesday, July 26, 2016 3:20 AM *To:* Michele Zundo *Cc:* Mike Beckerle; rui.mestre(a)deimos.com.pt *Subject:* Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05 Michele, Rui I assume that this is an example ? dfdl:length="{/Packet_Primary_Header/Packet_Data_Length + 1 - contentLength( /Packet_Data_Field/(.*)Packet_Secondary_Header, 'bytes') - 2}" Firstly, contentLength is a DFDL function so it needs to be in the DFDL namespace, eg, dfdl:contentLength(). Secondly, the first argument to dfdl:contentLength() is a path, so you are effectively still using regular expressions in path steps. Regards Steve Hanson *IBM Integration Bus* <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle(a)tresys.com> Date: 25/07/2016 17:10 Subject: Fwd: OGF DFDL WG Call Minutes 2016-07-05 ------------------------------ Dear Steve, Please find below the answer from our developers and example. Note that we have updated our implementation of DFDL to be as compliant as we can at this point in time with the exception noted below. Michele Begin forwarded message: *From: *"Rui Mestre (DME)" <*rui.mestre(a)deimos.com.pt* <rui.mestre(a)deimos.com.pt>> *Subject: Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05* *Date: *25 July 2016 at 17:45:05 GMT+2 Dear Michele, I believe that after our DFDL compliance effort the mentioned "use of a regex in the path step of a DFDL expression" is no longer in place. Currently the only extension implemented in DFDL4S regarding the use of regular expressions is that implementation of dfdl:contentLength is extended to support also regular expressions when specifying the node. Please find attached a schema file example containing such extension in the use of dfdl:contentLength. Best regards, Rui Begin forwarded message: *From: *Steve Hanson < <smh(a)uk.ibm.com>*smh(a)uk.ibm.com* <smh(a)uk.ibm.com>> *Subject: OGF DFDL WG Call Minutes 2016-07-05* *Date: *5 July 2016 at 17:49:13 GMT+2 *To: * <dfdl-wg(a)ogf.org>*dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> *Cc: *"Mike Beckerle" < <mbeckerle(a)tresys.com>*mbeckerle(a)tresys.com* <mbeckerle(a)tresys.com>>, "Michele Zundo" <*michele.zundo(a)esa.int* <michele.zundo(a)esa.int>> Please find minutes from the above call at <https://redmine.ogf.org/dmsf_files/13537?download=> *https://redmine.ogf.org/dmsf_files/13537?download=* <https://redmine.ogf.org/dmsf_files/13537?download=> * @Michele - please can you send to the WG a schema that shows your use of a regex in the path step of a DFDL expression ?* Next call *Aug 2nd* Regards Steve Hanson Architect, IBM DFDL, Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> *tel:+44-1962-815848* <+44-1962-815848> ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: <michele.zundo(a)esa.int>*michele.zundo(a)esa.int* <michele.zundo(a)esa.int> This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: *michele.zundo(a)esa.int* <michele.zundo(a)esa.int> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2