- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] need clarification on unparsing with textPadKind='padChar'
by Steve Hanson 14 Sep '16

14 Sep '16

Mike Apologies - missed this when compiling the agenda for yesterday's call - will add it to agenda for next call. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 23/08/2016 13:19 Subject: Re: [DFDL-WG] need clarification on unparsing with textPadKind='padChar' Erratum 5.18 is not yet incorporated into the spec. When it is then dfdl:lengthKind 'explicit' with an expression becomes the same as dfdl:lengthKind 'explicit' with a literal. I believe that means the expression result is the fixed length for unparsing, and minLength is only used for validation. However I would like to run a test with IBM DFDL to make sure everything is as expected. fn:string-length() always returns the length of the value in the infoset item. It has no schema awareness. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 10/08/2016 17:27 Subject: [DFDL-WG] need clarification on unparsing with textPadKind='padChar' Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> The spec for textPadKind has this sentence about unparsing: "When dfdl:lengthKind is 'explicit' (and dfdl:length is an expression), 'delimited', 'prefixed', 'pattern' the data value is padded to the length given by the XSD minLength facet for type 'xs:string' or dfdl:textOutputMinLength property for other types." My concern is the case where dfdl:lengthKind is 'explicit' and dfdl:length is an expression. Example 1: Suppose my infoset element contains an element at path '../x' of type xs:string containing 10 characters "1234567890" Suppose the length expression evaluates to 20, and lengthUnits is 'characters' Suppose the XSD minLength facet is 100. Suppose the textPadKind is padChar. The above sentence says the target length would be 100 and we would pad to length 100, not 20, and there is no error. I want to be sure this is correct and what we intended. To me this seems inconsistent with what we have stated in Erratum 5.18 which says that whether or not the length is a constant or an expression, this element is treated as fixed length, and when unparsing, the length expression is evaluated, not ignored. It's not technically incorrect in erratum 5.18 because that doesn't deal with this minLength complexity. It's just not terribly clear. We may want to update this sentence of the spec, and the erratum to both use the term target length, which is in our glossary. Furthermore, In this case, the dfdl:contentLength for this element should return 100 for units 'characters'. In this case the dfdl:valueLength for this element should return 10 for units 'characters' Expression fn:string-length(../x) should return 10. The length expression value of 20 is effectively being ignored because it is less than the minLength. The length expression *would* be used if it was greater than the minLength. Example 2: My infoset contains an element at path "../x" of type xs:string of length 200 The length expression evaluates to 20 dfdl:truncateSpecifiedLengthString is true XSD minLength is 40. In this case I believe we would have a target length of 40, and we would truncate the string to length 40. dfdl:contentLength of this element is 40 dfdl:valueLength of this element is 40 Expression fn:string-length(../x) should return 200. Is this the correct interpretation? Example 3: Same as example 2, except XSD minLength is 0. Now the target length would be 20 and we would truncate the length to 20. dfdl:contentLength of this element is 20 characters dfdl:valueLength of this element is 20 characters. Expression fn:string-length(../x) should return 200. Is this the correct interpretation? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases
by Steve Hanson 14 Sep '16

14 Sep '16

Actions 290 and 291 raised to investigate further - see minutes. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/09/2016 13:14 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Mike I am assuming that the processing for utf-16 'fixed' or 'variable' is entirely handled by ICU so there should be no coding overhead. IBM DFDL works ok for dfdl:lengthKind='explicit' for an element of complex type with dfdl:lengthUnits='characters' and dfdl:encoding="utf-8". But there are conditions the content of the complex type must satisfy otherwise an SDE results, such as: CTDV1524E : For a complex element, when 'lengthKind' is 'explicit' or 'prefixed', and 'lengthUnits' is characters, all simple child elements must have text representation, 'lengthUnits' set to 'characters' and the same encoding. So we insist that the properties of the children are consistent with the properties of the parent. If you recall, IBM DFDL does all these kinds of validation checks in a pre-processing phase. That seems a pretty sensible rule but I am not sure if the rule appears in the spec as such - I just had a quick look but didn't spot anything. So I guess I don't see a need for these things to be optional features? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 10/08/2016 18:57 Subject: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Given the limited set of required encodings for a conforming DFDL processor, I believe dfdl:utf16Width='variable' should be an optional feature. That's just consistency with what is optional already. But it is also quite hard to implement. There are other situations that are very hard to implement, probably never used by real users, yet which are non optional in the spec: I would suggest that dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding like utf-8 is very problematic to implement. I am pretty sure IBM DFDL has no implementation of this per email threads, and I know I don't want to implement this in Daffodil even though we're trying to be very comprehensive in the implementation eventually. I think implementations should be free to just not implement this. These sorts of cases often exist just because we're trying to preserve some orthogonality of composition in the language. So it's possible to do quite a few things that probably aren't ever needed by anyone, that reflect ill-defined data formats, etc. I'd rather not document a bunch of "non-conformances" for Daffodil or other implementations for these sorts of things. I'd like to say we don't implement them, but they're optional, and so that's allowed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2016-09-13
by Steve Hanson 13 Sep '16

13 Sep '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13541?download= Next call Sept 26th Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

DFDL4S use of wildcard/regex in length path expression
by Mike Beckerle 13 Sep '16

13 Sep '16

(I edited the subject line for this discussion.) Michele, Could you send the parts of the DFDL4S schema that use the types that have these wildcard/regex in them? Also, please send as a zip file, not a jar. Google, in its infinite wisdom, has decided that it is better to save me from a potential computer virus than to let me get my work done, and they block all access to attachments that are jar files, you can't rename them, save to Drive, save to local file system, nor forward them to a different email account. I want to see exactly what this wildcard is doing for you. I suspect it is just allowing these types to be reused in contexts where otherwise you'd have to have separate complex type definitions just because this one step of the path is different at the different points of re-use. I want to really think about whether there is some other way to achieve this same reusability, or if not, just how bad is the workaround. ...mike beckerle ------------------------------ *From:* Steve Hanson [smh(a)uk.ibm.com] *Sent:* Tuesday, July 26, 2016 3:20 AM *To:* Michele Zundo *Cc:* Mike Beckerle; rui.mestre(a)deimos.com.pt *Subject:* Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05 Michele, Rui I assume that this is an example ? dfdl:length="{/Packet_Primary_Header/Packet_Data_Length + 1 - contentLength( /Packet_Data_Field/(.*)Packet_Secondary_Header, 'bytes') - 2}" Firstly, contentLength is a DFDL function so it needs to be in the DFDL namespace, eg, dfdl:contentLength(). Secondly, the first argument to dfdl:contentLength() is a path, so you are effectively still using regular expressions in path steps. Regards Steve Hanson *IBM Integration Bus* <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle(a)tresys.com> Date: 25/07/2016 17:10 Subject: Fwd: OGF DFDL WG Call Minutes 2016-07-05 ------------------------------ Dear Steve, Please find below the answer from our developers and example. Note that we have updated our implementation of DFDL to be as compliant as we can at this point in time with the exception noted below. Michele Begin forwarded message: *From: *"Rui Mestre (DME)" <*rui.mestre(a)deimos.com.pt* <rui.mestre(a)deimos.com.pt>> *Subject: Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05* *Date: *25 July 2016 at 17:45:05 GMT+2 Dear Michele, I believe that after our DFDL compliance effort the mentioned "use of a regex in the path step of a DFDL expression" is no longer in place. Currently the only extension implemented in DFDL4S regarding the use of regular expressions is that implementation of dfdl:contentLength is extended to support also regular expressions when specifying the node. Please find attached a schema file example containing such extension in the use of dfdl:contentLength. Best regards, Rui Begin forwarded message: *From: *Steve Hanson < <smh(a)uk.ibm.com>*smh(a)uk.ibm.com* <smh(a)uk.ibm.com>> *Subject: OGF DFDL WG Call Minutes 2016-07-05* *Date: *5 July 2016 at 17:49:13 GMT+2 *To: * <dfdl-wg(a)ogf.org>*dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> *Cc: *"Mike Beckerle" < <mbeckerle(a)tresys.com>*mbeckerle(a)tresys.com* <mbeckerle(a)tresys.com>>, "Michele Zundo" <*michele.zundo(a)esa.int* <michele.zundo(a)esa.int>> Please find minutes from the above call at <https://redmine.ogf.org/dmsf_files/13537?download=> *https://redmine.ogf.org/dmsf_files/13537?download=* <https://redmine.ogf.org/dmsf_files/13537?download=> * @Michele - please can you send to the WG a schema that shows your use of a regex in the path step of a DFDL expression ?* Next call *Aug 2nd* Regards Steve Hanson Architect, IBM DFDL, Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> *tel:+44-1962-815848* <+44-1962-815848> ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: <michele.zundo(a)esa.int>*michele.zundo(a)esa.int* <michele.zundo(a)esa.int> This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: *michele.zundo(a)esa.int* <michele.zundo(a)esa.int> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

OGF DFDL WG Call Agenda 2016-09-16
by Steve Hanson 13 Sep '16

13 Sep '16

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13541?download= Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases
by Mike Beckerle 13 Sep '16

13 Sep '16

Given the limited set of required encodings for a conforming DFDL processor, I believe dfdl:utf16Width='variable' should be an optional feature. That's just consistency with what is optional already. But it is also quite hard to implement. There are other situations that are very hard to implement, probably never used by real users, yet which are non optional in the spec: I would suggest that dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding like utf-8 is very problematic to implement. I am pretty sure IBM DFDL has no implementation of this per email threads, and I know I don't want to implement this in Daffodil even though we're trying to be very comprehensive in the implementation eventually. I think implementations should be free to just not implement this. These sorts of cases often exist just because we're trying to preserve some orthogonality of composition in the language. So it's possible to do quite a few things that probably aren't ever needed by anyone, that reflect ill-defined data formats, etc. I'd rather not document a bunch of "non-conformances" for Daffodil or other implementations for these sorts of things. I'd like to say we don't implement them, but they're optional, and so that's allowed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

clarification needed: setVariable - executes before or after framing of the element/model group
by Mike Beckerle 07 Sep '16

07 Sep '16

The spec defines where setVariable executes as a statement relative to element processing and model-group processing. Question is: what does element and model group processing mean: the value region of the element, the content region of the element, or the whole element including all framing? So, for an element, spec says that setVariables run after the element. Rhetorical question: Does "after" mean after the terminator (including any expression for the terminator) has completed? Yes, because otherwise a element could contain the value of its own terminator, and that wouldn't work for lengthKind='delimited' as you would need the value to get the terminator to parse the value. So 'after' has to mean after the terminator, meaning all framing, and any other expression that can affect the length or interpretation in forming the element. For a model group, the spec says that setVariables run before the model group. So, can a sequence carry a set-variable that sets a variable value that is then used in the initiator property of that very sequence? E.g., <xs:sequence dfdl:initiator="{ $init }"> <xs:annotation><xs:appinfo ...> <dfdl:setVariable ref="init">{ ../someplace/else }</dfdl:setVariable> </xs:appinfo></xs:annotation> ..... reset of sequence </xs:sequence> If this is not allowed for the initiator, what about for the terminator or separator, or outputNewLine or encoding or any other property that affects the syntax of the model group? What about discriminators/asserts. Do those run before the sequence's initiator is matched, or after? E.g., can a discriminator indicate "true", but then the initiator fails to match? When unparsing, you don't execute discriminator/assert at all, so no guidance there. Maybe this is clear already, but I am not sure if setVariables should run before any framing of a model group, or after the framing, but before the content region, or before the value region. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

OGF DFDL WG Call 2016-08-30 - cancelled
by Steve Hanson 26 Aug '16

26 Aug '16

Steve not available so cancelling. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 26 Aug '16

26 Aug '16

Steve unable to make call

1 0

need clarification on unparsing with textPadKind='padChar'
by Mike Beckerle 23 Aug '16

23 Aug '16

The spec for textPadKind has this sentence about unparsing: "When dfdl:lengthKind is 'explicit' (and dfdl:length is an expression), 'delimited', 'prefixed', 'pattern' the data value is padded to the length given by the XSD minLength facet for type 'xs:string' or dfdl:textOutputMinLength property for other types." My concern is the case where dfdl:lengthKind is 'explicit' and dfdl:length is an expression. Example 1: Suppose my infoset element contains an element at path '../x' of type xs:string containing 10 characters "1234567890" Suppose the length expression evaluates to 20, and lengthUnits is 'characters' Suppose the XSD minLength facet is 100. Suppose the textPadKind is padChar. The above sentence says the target length would be 100 and we would pad to length 100, not 20, and there is no error. I want to be sure this is correct and what we intended. To me this seems inconsistent with what we have stated in Erratum 5.18 which says that whether or not the length is a constant or an expression, this element is treated as fixed length, and when unparsing, the length expression is evaluated, not ignored. It's not technically incorrect in erratum 5.18 because that doesn't deal with this minLength complexity. It's just not terribly clear. We may want to update this sentence of the spec, and the erratum to both use the term target length, which is in our glossary. Furthermore, In this case, the dfdl:contentLength for this element should return 100 for units 'characters'. In this case the dfdl:valueLength for this element should return 10 for units 'characters' Expression fn:string-length(../x) should return 10. The length expression value of 20 is effectively being ignored because it is less than the minLength. The length expression *would* be used if it was greater than the minLength. Example 2: My infoset contains an element at path "../x" of type xs:string of length 200 The length expression evaluates to 20 dfdl:truncateSpecifiedLengthString is true XSD minLength is 40. In this case I believe we would have a target length of 40, and we would truncate the string to length 40. dfdl:contentLength of this element is 40 dfdl:valueLength of this element is 40 Expression fn:string-length(../x) should return 200. Is this the correct interpretation? Example 3: Same as example 2, except XSD minLength is 0. Now the target length would be 20 and we would truncate the length to 20. dfdl:contentLength of this element is 20 characters dfdl:valueLength of this element is 20 characters. Expression fn:string-length(../x) should return 200. Is this the correct interpretation? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1