July 2014 - dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Unparsing and choices - (related - Re: Fw: Action 233 (deferred) - "byte order not sufficient..." - draft document on experience with binary format MIL-STD-2045
by Mike Beckerle 16 Jul '14

16 Jul '14

We'll I am a bit concerned if we cannot generally implement an asymetric parser and unparser inside the same schema. That is, if you have calculated element named C, then I think one should be able to have hidden element A which is used only when parsing (C computed from A), and hidden element B which is used only when unparsing, (B computed from C), where B is non-existing on parse, and A is non-existing on unparse. I think this trick may work:  <element name="A" type="xs:string" minOccurs="0" dfdl:occursCountKind=...whatever.. />  <dfdl:newVariableInstance ref="areParsing" value ="{ fn:exists(../A) }"/>  <element name="B" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="{ if ($areParsing) then 0 else fn:string-length(../C) }" dfdl:outputValueCalc="{ ../C }" />  <element name="C" type="xs:string" dfdl:inputValueCalc="{ ../A }" /> On Mon, Jul 14, 2014 at 3:23 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: > The proposal for action 261 is that the dfdl:length expression is > evaluated, so that will stop what you propose from working. > > For the MIL-STD-2045 'fixedOrTerminated' case then the sequence suggestion > in the other email is much simpler. > > Regards > > Steve Hanson > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Steve Hanson/UK/IBM@IBMGB, > Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> > Date: 14/07/2014 19:51 > Subject: Re: Unparsing and choices - (related - Re: [DFDL-WG] Fw: > Action 233 (deferred) - "byte order not sufficient..." - draft document on > experience with binary format MIL-STD-2045 > ------------------------------ > > > > There has to be a way to get zero-length on parsing, but an > outputValueCalc string on unparsing. > > <defineVariable name="dummyLength" defaultValue="0" external='true'/> > > <element name="outputRaw" type="xs:string" > dfdl:lengthKind="explicit" dfdl:length="{ $tns:dummyLength }" > dfdl:outputValueCalc="{.....}" /> > > Would this work? The length expression isn't used when unparsing I believe > in that some other elements are supposed to be storing the length > information. That's the case here though the length value information isn't > being stored, what is being stored is a way to determine the length. > > Maybe there's something simpler? > > > > On Mon, Jul 14, 2014 at 1:42 PM, Steve Hanson <*smh(a)uk.ibm.com* > <smh(a)uk.ibm.com>> wrote: > Mike > > Section 17 of the spec says "It is a schema definition error if > dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same > element. " So outputRaw will give an SDE. > > Regards > > Steve Hanson > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:*+44-1962-815848* <%2B44-1962-815848> > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com* > <mbeckerle.dfdl(a)gmail.com>> > To: Steve Hanson/UK/IBM@IBMGB, > Cc: "*dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>" <*dfdl-wg(a)ogf.org* > <dfdl-wg(a)ogf.org>> > Date: 14/07/2014 16:43 > Subject: Unparsing and choices - (related - Re: [DFDL-WG] Fw: > Action 233 (deferred) - "byte order not sufficient..." - draft document on > experience with binary format MIL-STD-2045 > ------------------------------ > > > > Breaking out this one issue for a separate discussion thread. > > Consider the choice below. There is a choice of two different elements > named raw1 and raw2, then a value element which is computed based on one or > the other of the alternatives. > > <choice> >  > <element name="raw1" type="xs:string" > dfdl:lengthKind='pattern' > dfdl:lengthPattern="[^\x7F]{0,99}" > dfdl:terminator="%DEL;"/> > <element name="raw2" type="xs:string" > dfdl:lengthKind="explicit" > dfdl:length="100"/> > </choice> > <element name='value' type='xs:string' > dfdl:inputValueCalc='{ if (fn:exists( ../raw1 ) then ../raw1 else > ../raw2 }'/> > > Now, given the statement below: > > SMH: There is no way to make a choice branch fail when unparsing. (The > only 'backtracking' when unparsing a choice is when the infoset contains no > branch at all then the spec states that each branch is examined in turn > until one is found that successfully applies defaults. But that's not > really backtracking, as you can statically deduce the branch from the > schema alone, so the 'default' branch to use can be computed up front). > > So is there any way to fix this and make the above choice work? > > This is the only thing I can think of, which ends up with entirely > separate elements for parsing and unparsing. > > I am not sure this works or should work. > > <group name="hidG1"> > <sequence> >  > <element name="raw1" type="xs:string" > dfdl:lengthKind='pattern' > dfdl:lengthPattern="[^\x7F]{0,99}" > dfdl:terminator="%DEL;" > minOccurs='0' dfdl:occursCountKind='implicit'/> >  > <element name="raw2" type="xs:string" > dfdl:lengthKind="explicit" > dfdl:length="100" > minOccurs='0' dfdl:occursCountKind='expression' > dfdl:occursCount="{ if (fn:exists(../raw1) then 0 else 1 }"/> >  > <element name="outputRaw" type="xs:string" > dfdl:lengthKind="explicit" dfdl:length="{ 0 }" > dfdl:outputValueCalc="{ if (fn:string-length(../value) eq 100 then > ../value else fn:concat(../value, '') }" /> > <sequence> > </group> > >  > <sequence dfdl:hiddenGroupRef="tns:hidG1"/> > <element name='value' type='xs:string' > dfdl:inputValueCalc='{ if (fn:exists( ../raw1 ) then ../raw1 else > ../raw2 }'/> > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >

2 1

stop bit encoding
by Mike Beckerle 16 Jul '14

16 Jul '14

The FIX protocols (financial interchange) make use of stop bit encoding. I have heard of other places using this also. This is modern, current, non-legacy usage. This is the length encoding where the most-significant bit of a byte is used to determine whether there are more ASCII characters or not. The last byte will have the most-significant bit of 1, prior bytes most-significant bit of 0. Minimum length in this length kind is 1. It seems we are missing this lengthKind, which applies only to text in the ASCII/US-ASCII encoding. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call Minutes 2014-07-15
by Steve Hanson 15 Jul '14

15 Jul '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13280?download= Please review spec draft r15 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Unparsing and choices - (related - Re: Fw: Action 233 (deferred) - "byte order not sufficient..." - draft document on experience with binary format MIL-STD-2045
by Mike Beckerle 14 Jul '14

14 Jul '14

Breaking out this one issue for a separate discussion thread. Consider the choice below. There is a choice of two different elements named raw1 and raw2, then a value element which is computed based on one or the other of the alternatives. > > <choice> >  > <element name="raw1" type="xs:string" > dfdl:lengthKind='pattern' > dfdl:lengthPattern="[^\x7F]{0,99}" > dfdl:terminator="%DEL;"/> > <element name="raw2" type="xs:string" > dfdl:lengthKind="explicit" > dfdl:length="100"/> > </choice> > <element name='value' type='xs:string' > dfdl:inputValueCalc='{ if (fn:exists( ../raw1 ) then ../raw1 else > ../raw2 }'/> > > Now, given the statement below: > > SMH: There is no way to make a choice branch fail when unparsing. (The > only 'backtracking' when unparsing a choice is when the infoset contains no > branch at all then the spec states that each branch is examined in turn > until one is found that successfully applies defaults. But that's not > really backtracking, as you can statically deduce the branch from the > schema alone, so the 'default' branch to use can be computed up front). > > So is there any way to fix this and make the above choice work? This is the only thing I can think of, which ends up with entirely separate elements for parsing and unparsing. I am not sure this works or should work. <group name="hidG1"> <sequence>  <element name="raw1" type="xs:string" dfdl:lengthKind='pattern' dfdl:lengthPattern="[^\x7F]{0,99}" dfdl:terminator="%DEL;" minOccurs='0' dfdl:occursCountKind='implicit'/>  <element name="raw2" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="100" minOccurs='0' dfdl:occursCountKind='expression' dfdl:occursCount="{ if (fn:exists(../raw1) then 0 else 1 }"/>  <element name="outputRaw" type="xs:string" dfdl:inputValueCalc='{ "" }' dfdl:outputValueCalc="{ if (fn:string-length(../value) eq 100 then ../value else fn:concat(../value, '') }" /> <sequence> </group>  <sequence dfdl:hiddenGroupRef="tns:hidG1"/> <element name='value' type='xs:string' dfdl:inputValueCalc='{ if (fn:exists( ../raw1 ) then ../raw1 else ../raw2 }'/>

2 3

OGF DFDL WG Call Agenda 2014-07-15
by Steve Hanson 14 Jul '14

14 Jul '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13280?download= Please review spec draft r15 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 233 (deferred) - "byte order not sufficient..." - draft document on experience with binary format MIL-STD-2045
by Steve Hanson 14 Jul '14

14 Jul '14

Mike Some further thoughts from IBM on your recommendations, after more internal discussion here. Preferable to have dfdl:bitOrder as a separate property rather to handle it via new dfdl:byteOrder enums. Although new properties pose validation issues for existing schemas, this should not compromise the language design. DFDL can choose what bitOrder/byteOrder combinations are supported. OK with with new dfdl:byteOrder enum for littleEndianAtomic16Bit though can we improve the name? dfdl:encoding has an architected system for extra encodings so US-ASCII-7-Bit-Packed should be x-US-ASCII-7-Bit-Packed, and the spec updated to remove specific mention of US-ASCII-7-Bit-Packed. We discussed proposed new dfdl:lengthKind 'fixedLengthOrTerminated'. A new enum implies that it can be used in any scenario, so the following need to be specified. dfdl:terminator must be set and can not be empty string or contain ES on its own If xs:string or xs:hexBinary, can maxLength facet be used instead of dfdl:length? (Suggest no - this is variable length data so min/maxLength are for validation only). Can dfdl:length be an expression? (Suggest no unless specific use case identified) Any special rules for emptyValueDelimiterPolicy and nilValueDelimiterPolicy ? Use on complex element. Presumably dfdl:length is first used to extract a 'box' but within that box does parser immediately scan for the dfdl:terminator or does it descend into the complex type and parse the children, expecting to either consume all the box or to find the terminator at the end? (Suggest the latter). Use on complex element. Last child can not be dfdl:lengthKind 'endOfParent'. Scanning rules: Use of this new dfdl:lengthKind switches off any in-scope stack of terminating markup in force at that point. Put another way, when we are scanning for the dfdl:terminator, we are not looking for any markup from an outer scope. So there's plenty to think about with this new dfdl:lengthKind. A good rule for deciding whether a new dfdl:length or dfdl:occursCountKind should be added is whether it bends some other part of the spec out of shape. The new dfdl:lengthKind looks ok so far. However we *think* we have come up with an alternative model which is simpler than you one you state in the document. Example for field 'varstr' with max length 100: <xs:sequence dfdl:terminator="{if (fn:str-len(varstr) eq 100) then '%ES;' else '%DEL'}" ...> <xs:element name="varstr" type="xs:string" dfdl:lengthKind="pattern" dfdl:pattern="([^\x7F].\x7F)|(.{100})" ... /> </xs:sequence> Can't put dfdl:terminator with a self-referencing expression on the element. Might need fn:exists in the dfdl:terminator expression to handle optionality. Does that work? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/07/2014 13:09 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 08/07/2014 13:31 Subject: Re: [DFDL-WG] Action 233 (deferred) - "byte order not sufficient..." - draft document on experience with binary format MIL-STD-2045 Mike Please find attached IBM's initial comments to your experience document, as Word comments. We only got as far as the 3 x required extensions, not looked at the optional usability stuff in detail yet. We think we have our collective heads around the least significant bit ordering concept, but we think the explanation could be clearer and show the bits on-the-wire. Some debate as to whether this could be considered some variation of byteOrder but you've obviously thought this through and concluded a separate property is best. Also should bit order apply to text reps, given that byteOrder is binary rep only and any byte ordering variations in encodings are handled as separate encodings (eg, UTF-16LE and UTF-16BE). Regarding the US-ASCII-7-Bit-Packed encoding enum, this was added via erratum previously using the idea of DFDL-specific named encoding. But we are thinking that this could have been handled as an x- encoding, rather than specifically adding it to the spec. And thinking further on that same thread, should byteOrder be made to work like encoding and allow x- enums, then the new byteOrder would become a x- enum. The Wikipedia article you cite on Endianness mentions other byte orders (eg, Middle-Endian, PDP-Endian). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 24/06/2014 20:27 Subject: [DFDL-WG] Action 233 (deferred) - "byte order not sufficient..." - draft document on experience with binary format MIL-STD-2045 Sent by: dfdl-wg-bounces(a)ogf.org I have created an experience document about the "bit order" issue, which was a deferred action 233, and the subject of a public comment. The document is here: http://redmine.ogf.org/dmsf_files/13268. The public comment item is http://redmine.ogf.org/boards/15/topics/43. It recommends a new dfdl:bitOrder property, and a new dfdl:byteOrder enum value, without which it is impossible to model these data formats. It also recommends several other improvements to DFDL to facilitate handling these data formats. The formats in question are a variety of MIL-STD formats which are all densely packed binary data. These formats are in broad use. MIL-STD-2045 is one part of this family and this particular format specification is generally available without any restrictions from a US DoD web site ( http://assistdocs.com) so I made this specific format the subject of the document as it illustrates all the problematic issues. We have implemented the dfdl:bitOrder property in Daffodil, and it works with some useful tests now passing. We have also enhanced our TDML implementation to enable creation of tests for this feature (and in the process actually found two bugs in the MIL-STD-2045 spec!). Both the property and this TDML enhancement are described in the document. The sponsors of the Daffodil project are extremely keen to get this needed binary support into the DFDL v1.0 standard so as to have multiple DFDL implementations support it. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Spec section 12.3.7.2 - binary reps and zero length
by Steve Hanson 10 Jul '14

10 Jul '14

Section 9 of the specification talks about empty etc representations, where the meanings of a length of zero are defined. However section 12.3.7.2 has sub-sections for certain binary reps where it is illegal to have a length of zero. Is this deliberate? The cases where 0 is allowed are also those which can have lengthKind 'delimited' so that might be the reason. Note that disallowing length of zero prevents use of a default value. 12.3.7.2.1: Binary numbers. Table 22 has minimum length in bits => 0 not allowed. 12.3.7.2.2: Floating point numbers: Says length must be exactly 4 bytes or 8 bytes => 0 not allowed. 12.3.7.2.3: Packed numbers. No minimum length stated => 0 allowed. 12.3.7.2.4: Binary booleans. Refers to 12.3.7.2.1 => 0 not allowed. 12.3.7.2.5: Binary calendars. Says length must be exactly 4 bytes or 8 bytes => 0 not allowed. 12.3.7.2.6: Packed calendars. No minimum length stated => 0 allowed. 12.3.7.2.7: Opaque binary. No minimum length stated => 0 allowed. Noticed this when modelling an example of a header with user-defined information at the end with the length given by a binary count. <xs:complexType name="Type_Header"> <xs:sequence> <xs:element dfdl:length="75" name="Filler" type="xs:hexBinary"/> <xs:element dfdl:length="2" name="HeaderLength" type="xs:nonNegativeInteger"/> <xs:element dfdl:length="30" name="Filler2" type="xs:hexBinary"/> <xs:element dfdl:length="{../HeaderLength - 108}" name="UserInfo" type="xs:hexBinary" minOccurs="0" dfdl:occursCountKind="implicit"/> </xs:sequence> </xs:complexType> This could equally be modelled using occursCountKind 'expression' but that seems unnecessary if the length is 0: <xs:complexType name="Type_Header"> <xs:sequence> <xs:element dfdl:length="75" name="Filler" type="xs:hexBinary"/> <xs:element dfdl:length="2" name="HeaderLength" type="xs:nonNegativeInteger"/> <xs:element dfdl:length="30" name="Filler2" type="xs:hexBinary"/> <xs:element dfdl:length="{../HeaderLength - 108}" name="UserInfo" type="xs:hexBinary" minOccurs="0" dfdl:occursCountKind="expression" dfdl:occursCount="if (../HeaderLength - 108 eq 0) then 0 else 1"/> </xs:sequence> </xs:complexType> Also what do the minimum lengths in Table 22 mean when lengthUnits is bytes ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

errata on number of fractional second digits
by Cranford, Jonathan W. 09 Jul '14

09 Jul '14

Not sure if this has been addressed by a subsequent errata or not, but I just noticed something that should be corrected in the spec. Section 13.11.1 is on the dfdl:calendarPattern property. The following two statements are made about the number of fractional second digits. - "Any number of fractional seconds "S" may by specified in the pattern and accepted by implementations, but an implementation is free to represent a limited number of fractional seconds internally. Excess fractional seconds are truncated, not rounded up. At least millisecond accuracy must be implemented." (Section 13.11.1) - "The number of fractional second digits supported is implementation dependent but must be at least one." (Section 13.11.1) These two statements are not consistent. I believe the second one should be corrected to say that the minimum # of fractional second digits is three (to support a minimum of millisecond accuracy). I'm working through the last version that was put out for public comment (v1.0.4), so my apologies if this has already been corrected. FYI, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

2 1

guidelines for good schema design
by Mike Beckerle 09 Jul '14

09 Jul '14

This short article makes a number of good points about design of XML Schemas and they also apply to DFDL schemas. http://www.xfront.com/HideVersusExpose.pdf I would summarize it as "define types, not elements", though there is a little bit more to it than that. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Direct Dispatch case insensitivity on matching
by Andrew Edwards 09 Jul '14

09 Jul '14

In section 15 Choices, we define the direct dispatch method of choice resolution. For the choiceDispatchKey, we make the statements that... The expression must evaluate to an xs:string which must not be the empty string. and The resultant string must match the dfdl:choiceBranchKey property value of one of the element branches of the choice. This match is case insensitive. The use of a case insensitive match can be a bit of a performance inhibitor, and given that direct dispatch is intended to aid performance optimising the model, it kinda gets in the way. If an end user needs case insensitive searches then they can always define all of their branchKeys in one case and manually fold the dispatchKey into that case in their expression (toLower/toUpper). Can I propose that we remove the criteria that the match be case insensitive. The ignoreCase option may also play a part here. Cheers, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1