March 2016 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

Clarification for nil processing and zero length
by Steve Hanson 11 May '16

11 May '16

The DFDL 1.0 spec current says: 9.3.2.1 Simple element If the result is length zero as described above, the representation is then established by checking, in order for: 1. nil representation (if %ES; is a literal nil value). 2. empty representation. 3. normal representation (xs:string or xs:hexBinary only) 4. absent representation (if none of the prior representations apply). But should bullet 1 be: 1. nil representation (if either %ES; or %WSP*; on its own is a literal nil value). I added a test to IBM DFDL and found that setting dfdl:nilKind="literalValue" & dfdl:nilValue="%WSP*;" did not match an element value of empty string. That surprised me, and I think the IBM DFDL code is strictly implementing bullet 1. Using "%WSP*;" is useful for allowing zero or more white space to mean <nil>. I could use "%WSP+; %ES" to achieve the same goal but I'm not sure that was the intent here. Quick response appreciated. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Cancelled: OGF DFDL Working Group call
by Steve Hanson 17 Mar '16

17 Mar '16

Easter vacation

1 0

Re: [DFDL-WG] [DFDL4S]: Results of OGF DFDL WG Call Agenda 2016-02-16
by Michele Zundo 16 Mar '16

16 Mar '16

Thanks Steve, your input is very appreciated. We will check internally how to proceed. > On 03 Mar 2016, at 18:05 , Steve Hanson <smh(a)uk.ibm.com> wrote: > > Michele, Rui > > 1) This is a deviation from the DFDL 1.0 specification and should be corrected. The context for the evaluation of a DFDL discriminator (or assert or indeed any DFDL expression) is the object on which the annotation occurs (if expression is on an element) or the nearest parent element (if expression is on a sequence or choice). > > 2) Looking at the kinds of tests you are doing with dmx:assertExpression, I think you could use dfdl:assert with a recoverableException. > > Regards > > Steve Hanson > IBM Integration Bus <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, Hursley, UK > Architect, IBM DFDL <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, OGF DFDL Working Group <http://www.ogf.org/dfdl/> > smh(a)uk.ibm.com <mailto:smh@uk.ibm.com> > tel:+44-1962-815848 > mob:+44-7717-378890 > > > > From: Michele Zundo <michele.zundo(a)esa.int> > To: Steve Hanson/UK/IBM@IBMGB, Mike Beckerle <mbeckerle(a)tresys.com>, Montserrat Piñol <mpinol(a)eopp.esa.int>, Maurizio De Bartolomei <mdebartolomei(a)eopp.esa.int>, Rui Mestre <rui.mestre(a)deimos.com.pt>, DFDL-WG <dfdl-wg(a)ogf.org> > Date: 03/03/2016 16:05 > Subject: [DFDL4S]: Results of OGF DFDL WG Call Agenda 2016-02-16 > > > > Dear Steve/Mike, > > here few considerations regarding the 2 points in the MoM: > > > > 1) Regarding the evaluation of the XPath expressions in the dfdl:discriminator tests: > > > From: "Rui Mestre (DME)" <rui.mestre(a)deimos.com.pt <mailto:rui.mestre@deimos.com.pt>> > > Our implementation of the DFDL parser is based on a slightly different interpretation of the way the paths are applied. > Our approach is based on the fact that the discriminator expression is not being evaluated at the level of the element > containing the discriminator but instead at the level of the type containing such element. > > In the example at hand the type TypeMasterChId is defined with an element "Transfer_Frame_Version" followed by a "choice" among several alternative elements Spacecraft_Id_V*. So the decision of which "choice" branch to choose is done while "located" at the root level of TypeMasterChId. While parsing the sub-components of TypeMasterChId the several branches of the "choice" construct are evaluated (in order) and the first one evaluating to "true" is the one selected. Hence the relative path is written in reference to the top level of the type. Therefore in such cases the paths start with "./" instead of "../". > > > 2) dfdl:assert vs dmx:assertExpression > > The question from Steve: > “Also you said that you saw no need to implement DFDL assert - so what are the dmx:assertExpression attributes doing? “ > > (Maurizio De Bartolomei) : as far as I understood, the DFDL assert shall be used to check that data is well formed (i.e. the data is compliant to a given structure and can be parsed) and not for data validation (i.e. inspecting the content of a given element and comparing against an expected value). > The dmx:assertExpression does not check that the data is well formed but only checks the value of a given item against an expected value, if the value is different a message is returned. > This can be improved by either changing the expression name (i.e. dmx:checkExpression) or investigate if DFDL provides already suitable expressions for content validation without loosing the compactness and clarity of dmx:assertExpression). > > From: "Rui Mestre (DME)" <rui.mestre(a)deimos.com.pt <mailto:rui.mestre@deimos.com.pt>> > > Regarding the use of dmx:assertExpression (as opposed to dfdl:assert) we fully agree with Maurizio's answer above > About his statement "(...)or investigate if DFDL provides already suitable expressions for content validation without loosing the compactness and clarity of dmx:assertExpression" we had in fact identified an alternative: a possibility is to use XML Schema facets to capture the assert expression by a restriction to the data type. > However that approach does not cover the semantics currently available in our "custom" operators (in,inrange, et al.). So until the DFDL standard is updated to contain the corresponding semantics it would not be possible to completely replace the dmx:assertExpression with a standard DFDL (or in this case XML) construct. > > > ******* > > > > > > > From: Steve Hanson <smh(a)uk.ibm.com <mailto:smh@uk.ibm.com>> > Subject: Re: OGF DFDL WG Call Agenda 2016-02-16 > Date: 1 March 2016 at 19:12:13 GMT+1 > To: Michele Zundo <michele.zundo(a)esa.int <mailto:michele.zundo@esa.int>> > Cc: DFDL-WG <dfdl-wg(a)ogf.org <mailto:dfdl-wg@ogf.org>>, Mike Beckerle <mbeckerle(a)tresys.com <mailto:mbeckerle@tresys.com>> > > Hi Michele > > Here is an example of where I do not think the relative path is correct. The discriminator expression should be {../Transfer_Frame_Version in [0]}. In infoset terms, Spacecraft_Id_V1 is a sibling of Transfer_Frame_Version. > > Also you said that you saw no need to implement DFDL assert - so what are the dmx:assertExpression attributes doing? > > > <xs:complexType name="TypeMasterChId"> > <xs:sequence> > <xs:element name="Transfer_Frame_Version" type="xs:int" dfdl:lengthKind="explicit" dfdl:lengthUnits="bits" dfdl:length="2" dmx:representation="Integer16" > dmx:assertExpression="../Transfer_Frame_Version eq 1" dmx:assertMessage="Invalid TF version detected." dmx:assertPropagate="true"/> > <xs:choice> > <xs:element name="Spacecraft_Id_V1" type="xs:int" dfdl:lengthKind="explicit" dfdl:lengthUnits="bits" dfdl:length="10" dmx:representation="Integer32" > > <xs:annotation> > <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/ <http://www.ogf.org/dfdl/dfdl-1.0/>"> > <dfdl:discriminator test="{./Transfer_Frame_Version in [0]}" /> > </xs:appinfo> > </xs:annotation> > </xs:element> > <xs:element name="Spacecraft_Id_V2" type="xs:int" dfdl:lengthKind="explicit" dfdl:lengthUnits="bits" dfdl:length="8" dmx:representation="UInteger8" > dmx:assertExpression="../Spacecraft_Id_V2 in [82, 83, 84]" dmx:assertMessage="Invalid Spacecraft ID detected."> > <xs:annotation> > <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/ <http://www.ogf.org/dfdl/dfdl-1.0/>"> > <dfdl:discriminator test="{true}" /> > </xs:appinfo> > </xs:annotation> > </xs:element> > </xs:choice> > </xs:sequence> > </xs:complexType> > > > > Regards > > Steve Hanson > IBM Integration Bus <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, Hursley, UK > Architect, IBM DFDL <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, OGF DFDL Working Group <http://www.ogf.org/dfdl/> > smh(a)uk.ibm.com <mailto:smh@uk.ibm.com> > tel:+44-1962-815848 <tel:+44-1962-815848> > mob:+44-7717-378890 > > > -- > ----------------------------------------- > Michele Zundo > > Head of Ground System Definition and Verification Office > EOP-PEP > European Space Agency, ESTEC > e-mail: michele.zundo(a)esa.int <mailto:michele.zundo@esa.int> > > > > > > > > > This message and any attachments are intended for the use of the addressee or addressees only. > The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its > content is not permitted. > If you received this message in error, please notify the sender and delete it from your system. > Emails can be altered and their integrity cannot be guaranteed by the sender. > > Please consider the environment before printing this email. > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: michele.zundo(a)esa.int This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email.

1 0

OGF DFDL WG Call 2016-03-15 - cancelled
by Steve Hanson 15 Mar '16

15 Mar '16

Mike not available short notice so cancelling. Next meeting due to Easter is April 12th. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group call
by Steve Hanson 15 Mar '16

15 Mar '16

Mike unable to make call

1 0

OGF DFDL WG Call Agenda 2016-03-15
by Steve Hanson 14 Mar '16

14 Mar '16

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13524?download= Note 1hr earlier start for European participants. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group call (15 Mar 14:00 GMT in IBM: Hursley DE3S09)
by Steve Hanson 13 Mar '16

13 Mar '16

Calls for 2016. OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 Netherlands Toll-Free 0-800-363-6036 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg DFDL Getting Started: http://www.ibm.com/developerworks/library/se-dfdl/ DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf US DST has started bt UK DST has not, so changing the time to keep the US time the same.

1 0

interesting bit-order variation
by Mike Beckerle 09 Mar '16

09 Mar '16

So back when we were working on bit order, we decided that 3 of 4 combinations actually exist in the real world: byte order bigEndian bit order mostSignificantBitFirst byte order littleEndian bit order mostSignificantBitFirst byte order littleEndian bit order leastSignificantBitFirst I have, surprise surprise, run into a data format specification called Asterix, which is public, and which uses the fourth combination. This spec is all big-endian, except when they discuss the bits within a byte they present them in order from most-significant bit first, to least,...... but the labels they assign are least-significant first, so for example they explain the bits of a byte starting by explaining bit 8, then bit 7, etc. down to bit 1. For example: (There's an image below. If you can't see it. it's from page 24 of the spec, URL for spec is below) [image: Inline image 2] Translating this into DFDL with mostSignificantBitFirst is possible. It's just rather inconvenient. you have to take the data item size into account. Some items are 16-bit words so the bits start from bit-16 first, some things are 32-bits and start from bit 32, etc. etc. You have to subtract from the word size to get the "mostSignificantBitFirst" bit position. The spec is relatively modern in that it is byte/octet oriented. Uses 8-bits per character, and pays attention to not spanning byte boundaries with fields except for 2's complement integers, of which a few are 14-bits, i.e., not a multiple of 8-bits long. It's a big spec, but the part I have reviewed is called "cat062" and it is available here: http://www.eurocontrol.int/sites/default/files/content/documents/nm/asterix… ...mike Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Re: [DFDL-WG] Clarification for nil processing and zero length
by Steve Hanson 04 Mar '16

04 Mar '16

No, because for a complex element to be nil the only allowable value is empty string, so WSP* is too permissive. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: Alex Wood1/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 04/03/2016 13:06 Subject: Re: Clarification for nil processing and zero length I agree with the proposed change. Would it also apply to 9.3.2.2 for complex elements? That has very similar words. Cheers, Andy Edwards Software Engineer E-mail: andy.edwards(a)uk.ibm.com Snail-mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Phone (internal): 247222 Phone (external): 44-1962-817222 Desk: DE3 F16 The Feynman problem solving Algorithm: 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times ----- Original message ----- From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Cc: Alex Wood1/UK/IBM@IBMGB, Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM Subject: Clarification for nil processing and zero length Date: Fri, Mar 4, 2016 11:35 AM The DFDL 1.0 spec current says: 9.3.2.1 Simple element If the result is length zero as described above, the representation is then established by checking, in order for: 1. nil representation (if %ES; is a literal nil value). 2. empty representation. 3. normal representation (xs:string or xs:hexBinary only) 4. absent representation (if none of the prior representations apply). But should bullet 1 be: 1. nil representation (if either %ES; or %WSP*; on its own is a literal nil value). I added a test to IBM DFDL and found that setting dfdl:nilKind="literalValue" & dfdl:nilValue="%WSP*;" did not match an element value of empty string. That surprised me, and I think the IBM DFDL code is strictly implementing bullet 1. Using "%WSP*;" is useful for allowing zero or more white space to mean <nil>. I could use "%WSP+; %ES" to achieve the same goal but I'm not sure that was the intent here. Quick response appreciated. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0