July 2016 - dfdl-wg - lists.ogf.org

DFDL4S use of wildcard/regex in length path expression
by Mike Beckerle 13 Sep '16

13 Sep '16

(I edited the subject line for this discussion.) Michele, Could you send the parts of the DFDL4S schema that use the types that have these wildcard/regex in them? Also, please send as a zip file, not a jar. Google, in its infinite wisdom, has decided that it is better to save me from a potential computer virus than to let me get my work done, and they block all access to attachments that are jar files, you can't rename them, save to Drive, save to local file system, nor forward them to a different email account. I want to see exactly what this wildcard is doing for you. I suspect it is just allowing these types to be reused in contexts where otherwise you'd have to have separate complex type definitions just because this one step of the path is different at the different points of re-use. I want to really think about whether there is some other way to achieve this same reusability, or if not, just how bad is the workaround. ...mike beckerle ------------------------------ *From:* Steve Hanson [smh(a)uk.ibm.com] *Sent:* Tuesday, July 26, 2016 3:20 AM *To:* Michele Zundo *Cc:* Mike Beckerle; rui.mestre(a)deimos.com.pt *Subject:* Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05 Michele, Rui I assume that this is an example ? dfdl:length="{/Packet_Primary_Header/Packet_Data_Length + 1 - contentLength( /Packet_Data_Field/(.*)Packet_Secondary_Header, 'bytes') - 2}" Firstly, contentLength is a DFDL function so it needs to be in the DFDL namespace, eg, dfdl:contentLength(). Secondly, the first argument to dfdl:contentLength() is a path, so you are effectively still using regular expressions in path steps. Regards Steve Hanson *IBM Integration Bus* <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle(a)tresys.com> Date: 25/07/2016 17:10 Subject: Fwd: OGF DFDL WG Call Minutes 2016-07-05 ------------------------------ Dear Steve, Please find below the answer from our developers and example. Note that we have updated our implementation of DFDL to be as compliant as we can at this point in time with the exception noted below. Michele Begin forwarded message: *From: *"Rui Mestre (DME)" <*rui.mestre(a)deimos.com.pt* <rui.mestre(a)deimos.com.pt>> *Subject: Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05* *Date: *25 July 2016 at 17:45:05 GMT+2 Dear Michele, I believe that after our DFDL compliance effort the mentioned "use of a regex in the path step of a DFDL expression" is no longer in place. Currently the only extension implemented in DFDL4S regarding the use of regular expressions is that implementation of dfdl:contentLength is extended to support also regular expressions when specifying the node. Please find attached a schema file example containing such extension in the use of dfdl:contentLength. Best regards, Rui Begin forwarded message: *From: *Steve Hanson < <smh(a)uk.ibm.com>*smh(a)uk.ibm.com* <smh(a)uk.ibm.com>> *Subject: OGF DFDL WG Call Minutes 2016-07-05* *Date: *5 July 2016 at 17:49:13 GMT+2 *To: * <dfdl-wg(a)ogf.org>*dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> *Cc: *"Mike Beckerle" < <mbeckerle(a)tresys.com>*mbeckerle(a)tresys.com* <mbeckerle(a)tresys.com>>, "Michele Zundo" <*michele.zundo(a)esa.int* <michele.zundo(a)esa.int>> Please find minutes from the above call at <https://redmine.ogf.org/dmsf_files/13537?download=> *https://redmine.ogf.org/dmsf_files/13537?download=* <https://redmine.ogf.org/dmsf_files/13537?download=> * @Michele - please can you send to the WG a schema that shows your use of a regex in the path step of a DFDL expression ?* Next call *Aug 2nd* Regards Steve Hanson Architect, IBM DFDL, Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> *tel:+44-1962-815848* <+44-1962-815848> ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: <michele.zundo(a)esa.int>*michele.zundo(a)esa.int* <michele.zundo(a)esa.int> This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: *michele.zundo(a)esa.int* <michele.zundo(a)esa.int> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

Spec bug: omission of mention of textBooleanJustification property
by Mike Beckerle 02 Aug '16

02 Aug '16

The sentence "The padding characters populate the *LeftPadding* and/or *RightPadding* regions depending on dfdl:textStringJustification, dfdl:textNumberJustification, or dfdl:textCalendarJustification, depending on the type of the element." Appears in the definition of textPadKind and textTrimKind. It omits the dfdl:textBooleanJustification case. The proper phrasing to match the wording of the prior sentence is "The padding characters populate the *LeftPadding* and/or *RightPadding* regions depending on dfdl:textStringJustification, dfdl:textNumberJustification, dfdl:textBooleanJustification, or dfdl:textCalendarJustification, depending on the type of the element." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

unparsing with dfdl:setVariable referring to dfdl:outputValueCalc element
by Mike Beckerle 02 Aug '16

02 Aug '16

I somehow think we already discussed this, but am unable to find the thread. Suppose: <xs:element name="len" type="xs:int" dfdl:outputValueCalc="{ dfdl:valueLength(../data) + 10 }"/> <xs:sequence><xs:annotation><xs:appinfo ...> <dfdl:setVariable name="var">{ ../len }" /> </xs:appinfo></xs:annotation></xs:setVariable> <xs:element name="data" dfdl:length="{ $var }"> .... </xs:element> In the above, when unparsing, the output value calc for len can be evaluated, but we must delay its evaluation and unparsing until the subsequent data element is available. The next thing the unparse has to do, after delaying the unparsing of 'len' is set the var variable. This requires the value of len, which has been deferred. We have real schemas (e.g., even PCAP) where this occurs. This breaks the rule that when variables are evaluated, things they reference must have already been evaluated. Basically, when we delay evaluating the OVC for len we are suspending anything that depends on len having a value as well. Did we discuss and/or agree on this behavior? I don't have any work around for this. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Need dfdl:inputValueCalc and dfdl:outputValueCalc on same element
by Mike Beckerle 02 Aug '16

02 Aug '16

So in PCAP format we find something like this: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../payloadLength }"> <complexType>.....</complexType> </element> To make this unparse we really need the payloadLength element to carry both an IVC and an OVC. Otherwise when unparsing the dfdl:length expression on the payload element can't be meaningful. We have an erratum that states that length expressions are always re-evaluated when unparsing, so as to produce a target length that can be used to insert padding/fillByte. However, the spec currently says you cannot have both IVC and OVC on the same element. I do not recall why we have this restriction, other than trying to prohibit things we didn't think were necessary, but it seems in this situation to be needed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Action 242: valueLength of complexType - contentLength and valueLength functions
by Mike Beckerle 01 Aug '16

01 Aug '16

(There are a few separate email threads related to Action 242. They have different subject lines, but all mention Action 242.) Experience with the Daffodil unparser implementation and dfdl:outputValueCalc has led us to a number of conclusions: 1) we need the concept of valueLength of a complexType which excludes the ElementUnused region, so that contentLength of a complexType can include the ElementUnused region. The example that motivates this is from PCAP where there are length fields that aren't the exact length while unparsing but contain other factors: <element name="len" type="xs:int" dfdl:outputValueCalc="{ fn:max(dfdl:valueLength(../ctElem, 'bytes'), 60) }" .../> ... <element name="ctElem" dfdl:length="{ ../len }" .../> In the above, we have a complex type element that has a minimum length of 60. So when unparsing if the actual length of the choice/sequence that is the model group of the ctElem is of length less than 60, we need to fill out the ElementUnused region of the DFDL grammar so as to achieve the min length of 60 bytes. If the dfdl:outputValueCalc of the len element was to call dfdl:contentLength, then we would have a circularity. Erratum 5.18 clarifies that length expressions are evaluated when unparsing to produce the target length that is used for pad/fill. This concept must be extended to complex types so that it also addresses the ElementUnused region that can follow the model group of a complex type. To clarify all this I suggest the DFDL data grammar must be changed as follows: ComplexNormalRep = LeftFraming PrefixLength ComplexContent RightFraming # Unchanged. Just here for context ComplexContent = ComplexValue ElementUnused ComplexValue = Sequence | Choice The above change basically moves ELementUnused out of the ComplexNormalRep production into a separate ComplexContent production, and distinguishes ComplexValue from ComplexContent. With the above grammar rule changes, I believe the spec as modified for Erratum 5.18 will be correct. (Everyplace where lengthKind explicit is lumped in with prefixed etc must be broken out, as it is a different case. ) 2) Spec says escapeCharacter, escapeBlockStart, and excapeBlockEnd contribute to the content length of an element. (Note: escapeEscapeCharacter is missing this stipulation and should be made consistent.) However, I believe this is incorrect. These all contribute to the *value* length, that is to say, they appear within the "SimpleValue | NilLogicalValue" region of the grammar. 3) Definition of dfdl:valueLength must be changed to specify that it returns the length of the ComplexValue region for complex types, and excludes any filling of the ElementUnused region. The definition of dfdl:contentLength is acceptable as is. However, both require clarification as to when "characters" and "bytes" are allowed relative to the lengthUnits of the element that is the argument. (This length units topic is a separate matter and will be a separate message.) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Action 286 - Daffodil v IBM DFDL feature comparison
by Steve Hanson 19 Jul '16

19 Jul '16

286 Create table that compares Daffodil features to IBM DFDL features (Steve/Mike) 16/2: Take table from section 22 of DFDL 1.0 spec and add columns showing implementations. 1/3: No progress 10/5: No progress. Work on this for next call. ... 7/6: No further progress 5/7: To kick this off here is the optional features table from spec section 21 with a column added for IBM DFDL Feature Detection IBM DFDL Validation External switch Yes Named Formats dfdl:defineFormat or dfdl:ref Yes Choices xs:choice in xsd Yes Arrays where size not known in advance dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' Partial - not 'stopValue' Expressions Use of a DFDL expression in any property value Yes End of parent dfdl:lengthKind "endOfParent" No Simple type restrictions xs:simpleType in xsd Partial - not unions Text representation for types other than String dfdl:representation "text" for Number, Calendar or Boolean types Yes Delimiters dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or dfdl:lengthKind "delimited" Yes Nils XSDL nillable 'true' in xsd Yes Defaults XSDL default or fixed in xsd Partial - unparsing only Bi-Directional text. dfdl:textBiDi 'yes' No Lengths in Bits dfdl:alignmentUnits 'bits' or dfdl:lengthUnits 'bits' Yes Delimited lengths and representation binary element dfdl:representation 'binary' (or implied binary) and dfdl:lengthKind 'delimited' Yes Regular expressions dfdl:lengthKind 'pattern', dfdl:assert with dfdl:testkind 'pattern' , dfdl:discriminator with dfdl:testkind 'pattern' Partial - only for dfdl:lengthKind Zoned numbers dfdl:textNumberRep 'zoned' Yes IBM 390 packed numbers dfdl:binaryNumberRep 'packed' Yes IBM 390 packed calendars dfdl:binaryCalendarRep 'packed' No IBM 390 floats dfdl:binaryFloatRep 'ibm390Hex' Yes Unordered sequences dfdl:sequenceKind 'unordered' Yes Floating elements dfdl:floating 'yes' No dfdl functions in expression language dfdl:functions in expression No Hidden groups dfdl:hiddenGroupRef <> '' No Calculated values dfdl:inputValueCalc <> '' or dfdl:outputValueCalc <> '' No Escape schemes dfd:defineEscapeScheme in xsd Yes Extended encodings Any dfdl:encoding value beyond the core list Yes Asserts dfdl:assert in xsd Partial - not on global elements or simple types Discriminators dfdl:discriminator in xsd Partial - not on global elements or simple types Prefixed lengths dfdl:lengthKind 'prefixed' Partial - not nested prefixed lengths Variables dfdl:defineVariable, dfdl:newVariableInstances, dfdl:setVariable Variables in DFDL expression language Note that variables as a feature is dependent on the Expressions feature. Partial - not newVariableinstance or external variables BCD calendars dfdl:binaryCalendarRep "bcd" Yes BCD numbers dfdl:binaryNumberRep "bcd" Yes Multiple schemas xs:include or xs:import in xsd Yes IBM 4690 packed numbers dfdl:binaryNumberRep "ibm4690Packed" Yes IBM 4690 packed calendars dfdl:binaryCalendarRep "ibm4690Packed" Yes DFDL Byte Value Entities Use of %#r syntax in a DFDL String Literal other than the dfdl:fillByte property Yes DFDL Standard Character Set Encodings dfdl:encoding name begins with "X-DFDL-". No Bit Order - Least Significant Bit First dfdl:bitOrder with value 'leastSignificantBitFirst' No Some IBM DFDL restrictions are not captured in the table, mainly due to non-implementation of many of the later errata. Full details here https://www.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mf… but the key omissions are: fn:count(), fn:exactlyOne(), fn:name(), fn:local-name(), fn:namespace-uri() BOMs Correct distinction between 'empty' and 'missing' when parsing (action 140) Use of namespaces in expression paths dfdl:separatorSuppressionPolicy "trailingEmptyStrict" Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Action 242: units 'characters' and encodingErrorPolicy 'error' - valueLength and contentLength functions
by Mike Beckerle 19 Jul '16

19 Jul '16

In analysis of the valueLength and contentLength email discussion threads, we're not converged on whether these functions when measured in units 'characters', are allowed to compute the length without checking for decode errors when (a) the encoding is fixed width - so a character unit is just an alias for some number of bytes (b) encodingErrorPolicy='error'. I think we need to clarify that just because encodingErrorPolicy is 'error', doesn't mean all data in scope of that will be scanned to be sure there are no decode errors. Other features with similar issues are regex pattern asserts. In this case the regex is matching against text, and that match might or might not encounter a decode error, but the entire scope of data it's talking about is NOT going to get converted just to insure no chance of a decode error. Can we go so far as to say DFDL implementations can (or maybe even must) optimize performance by avoiding character decoding when possible? This means that some character decode errors may not be detected even though dfdl:encodingErrorPolicy is 'error'. I would suggest the language should say that only character decoding that results in a character being placed into the DFDL Infoset is guaranteed to cause an error should that character not be decodable. (Similarly for unparsing, it is only if we actually unparse an unmapped character from the infoset to the output stream, that a encoding error is guaranteed to occur. I might not have worded it well, but I think the above is what we're trying to allow - implementations are free to exploit fixed character width, and just jumping around the bytes and not decoding/encoding anything - whenever they can, because we all expect, and think our users will expect, this level of performance. If you use UTF-8, certainly a common thing,( or any other variable-width encoding) then you are likely to get some cases where implementations say "can't do that with utf-8" because it's just a limitation of the implementation. One may also get cases where switching from utf-16 to utf-8 for your data changes the behavior of the processor because utf-16 wouldn't detect some decoding errors because of fixed width, whereas when using utf-8 will have to measure length in characters by decoding and so will detect the error. Basically, we've tried to make a consistent position around a messy area: The schema contains a complex type element which is a mixture of character encodings, binary stuff, and may have decode errors in the corresponding data stream (or characters in the infoset that have no mapping into the representation of the encoding - for unparsing). Given this mess, there are ways that a schema can look at it, and foolishly-perhaps, treat it as characters. Asserts with test patterns are one. Specified length with units of 'characters' is another, and the dfdl:contentLength and dfdl:valueLength functions is yet another, since one can specify the units 'characters' as that 2nd argument. The only consistent position is that dfdl:contentLength or dfdl:valueLength of an element, with units 'character' does NOT necessarily imply those characters will be decoded/encoded. Anyway, it's moot in the scenario where an earlier OVC wants the dfdl:contentLength of something later,.... if when we unparse the thing later, we get the decode error at that point. We're just getting what is arguably an incorrect OVC computation, but followed by a later decode error. .... Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Action 242 - complexType with interior alignment - valueLength and contentLength
by Mike Beckerle 19 Jul '16

19 Jul '16

I have reexamined this material from prior discussions in this group, and considering the experience of implementing this in Daffodil. We're not yet converged on whether the dfdl:contentLength of a complex element, is allowed to be a function of its starting position due to variable-sized alignment regions interior to the complex representation. This creates a situation where a element with dfdl:outputValueCalc can have variable-length itself, and hence a complex element after it, that it forward references, can't compute its content's length, because it can't know the start position due to the variable length OVC element preceding it. (Got that?) All I can say is ..... gaaaak. This scenario is possible due to the way that features compose in DFDL, which is trying to be fairly orthogonal to each other. Alignment causes representation lengths to break this orthogonality because length now depends on the length of previous things. Our choices: 1) yup must detect this crazy deadlock. Runtime SDE. 2) rule it out by disallowing OVC elements that call dfdl:contentLength which reference complex elements that have interior alignment, to be variable length themselves. It's an SDE. (Gaaaaak) 3) rule it out by saying OVC elements can never be variable length (... .but I think I've seen prefix lengths in textual data before, and those are naturally variable length... and might not have been adjacent, so not lengthKind 'prefixed', but needs OVC..... double Gaaaaak) 4) punt it down the road by saying it's 'implementation dependent' behavior. Still have to say exactly what it is that is implementation behavior, but this could be a paragraph/subsection in the description of dfdl:contentLength, dfdl:valueLength, and a reference from the dfdl:outputValueCalc section. Better options? Ideas? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 06 Jul '16

06 Jul '16

steve away

1 0