- dfdl-wg - lists.ogf.org

lengthUnits bits not allowed for strings, binary floats, hexBinary
by Mike Beckerle 25 Jul '14

25 Jul '14

DFDL spec currently says this w.r.t. lengthUnits property: - 'bits' may only be used for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt, and xs:unsignedLong simple types with binary representation. This feels like a hold over from when we only had strings made up of 8-bit byte code-units. Now that we have 7-bit and 6-bit characters, this restriction seems unnecessary, and is in fact awkward because many specs specify the lengths of strings in bits (which are used universally for the length of everything in these formats). This is a real concern. In many cases DFDL Schemas will be generated from other specifications by programs. Having to conditionally convert the length as specified into different units for strings is just one more place to have to test, one more way the DFDL schema doesn't obviously match the specification from which it was derived, etc. Similarly: - 'bytes' must be used for type xs:hexBinary. - 'bytes' must be used for types xs:float and xs:double with binary representation. These are to prevent the user misunderstanding the limitations of these types. I.e., that we dont support hexbinary that is not a multiple of 8-bits in size, and float and double that are not exactly 4 and 8 bytes respectively. But now this restriction just seems annoying. If my data format specification base has all these values in bits, then it is painful when creating a DFDL schema to have to transform the values for just those element declarations that are of these types. I'm not suggesting we lift the actual restrictions. I'm good with hexBinary requiring whole bytes, and that float and double are exactly 32-bits and 64-bits respectively. I just think having to use bytes as the length units is just arbitrary. We thought it would be preventing people from making mistakes, but in fact it is likely to have the opposite effect, forcing them to have to interpret the length differently based on a type that might not even be defined in the same file where they see the dfdl:length property. Consider: <element name="x" type="foo:xType" dfdl:length="448"/> Is that 448 correct? It depends on the definition of foo:xType. If it's a simple type derived from string, then length units has to be characters or bytes, but in all the formats where I see these 448's. They are measured in bits. This is 56 bytes, holding 64 characters. But when I write out this element I don't have the information right there to know whether to divide by 8 or 7 or not without knowledge of the type. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

draft r18 on redmine
by Mike Beckerle 25 Jul '14

25 Jul '14

I posted spec r18 to redmine. This incorporates all the errata as of the 1.5 experience doc 1. - One exception - the errata about implementation-defined and implementation-dependent. I went through and updated all the places where this was not correctly stated per Jonathan's document. - However, I did not create an appendix also listing all these things. That turns out to be hard to do, and redundant (you end up copying text blocks - then they're in two places....) - I think if we want an index to everyplace that "implementation-defined" and "implementation-dependent" appear in the document, that's what an index is for, so we should generate one and put those in as indexed terms. I added the appendix of DFDL Standard Character Set Encodings, which includes both 7-bit and 6-bit flavors now. I believe the last thing to do is bit order, but that should be reviewed as a separate document first. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2014-07-22
by Steve Hanson 23 Jul '14

23 Jul '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13301?download= Please review spec draft r17 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

new documents
by Mike Beckerle 22 Jul '14

22 Jul '14

I pushed two new documents: http://redmine.ogf.org/dmsf/dfdl-wg?folder_id=5485 They are draft-gwdrp-dfdl-v1.0.4-r17 of the spec - mostly this is SMH changes, but a few others. Also has the US-ASCII-7-Bit-Packed moved out into a separate spec. draft-gwdi-dfdl-standard-encodings-v01.docx - first cut at a 2nd spec document which specifies only DFDL Standard character set encodings. Only entry currently is x-dfdl-us-ascii-7-bit-packed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 2

Fw: errata on number of fractional second digits
by Steve Hanson 22 Jul '14

22 Jul '14

Erratum 2.68 updated to reflect below. Spec draft r17 contains the change. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2014 14:43 ----- From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 09/07/2014 18:21 Subject: Re: [DFDL-WG] errata on number of fractional second digits Jonathan The two statements apply to different pattern symbols. The first is for the S symbol, the second is for the I symbol. However, it is clearer if the implementer uses the same precision for each, so the I symbol should change to match the S symbol. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 08/07/2014 16:07 Subject: [DFDL-WG] errata on number of fractional second digits Sent by: dfdl-wg-bounces(a)ogf.org Not sure if this has been addressed by a subsequent errata or not, but I just noticed something that should be corrected in the spec. Section 13.11.1 is on the dfdl:calendarPattern property. The following two statements are made about the number of fractional second digits. - "Any number of fractional seconds "S" may by specified in the pattern and accepted by implementations, but an implementation is free to represent a limited number of fractional seconds internally. Excess fractional seconds are truncated, not rounded up. At least millisecond accuracy must be implemented." (Section 13.11.1) - "The number of fractional second digits supported is implementation dependent but must be at least one." (Section 13.11.1) These two statements are not consistent. I believe the second one should be corrected to say that the minimum # of fractional second digits is three (to support a minimum of millisecond accuracy). I'm working through the last version that was put out for public comment (v1.0.4), so my apologies if this has already been corrected. FYI, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org) -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2014-07-22
by Steve Hanson 22 Jul '14

22 Jul '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13301?download= Please review spec draft r17 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Please review - New version of bit order/MIL-STD-2045 document
by Mike Beckerle 18 Jul '14

18 Jul '14

I posted this to our documents. http://redmine.ogf.org/dmsf_files/13300 This version is quite different. This version proposes that there are actually two different variations on the least-significant-bit first format. One is where the bits are actually reversed (logical 0xE2 becomes 0x47 when unparsed to a file) The other is where the bits are simply interpreted as LSB first, but the bytes are stored 'normally'. (0xE2 is in the file, but the 'first' 3 bits of it are 010, not 111.) Turns out it is this latter form that MIL-STD-2045 and related standards need, and that the Daffodil code implements today. I hope this version clarifies some things. You'll see there is a need to come up with useful names for these concepts when you read the document. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Re: [DFDL-WG] Unparsing and choices - (related - Re: Fw: Action 233 (deferred) - "byte order not sufficient..." - draft document on experience with binary format MIL-STD-2045
by Mike Beckerle 16 Jul '14

16 Jul '14

We'll I am a bit concerned if we cannot generally implement an asymetric parser and unparser inside the same schema. That is, if you have calculated element named C, then I think one should be able to have hidden element A which is used only when parsing (C computed from A), and hidden element B which is used only when unparsing, (B computed from C), where B is non-existing on parse, and A is non-existing on unparse. I think this trick may work:  <element name="A" type="xs:string" minOccurs="0" dfdl:occursCountKind=...whatever.. />  <dfdl:newVariableInstance ref="areParsing" value ="{ fn:exists(../A) }"/>  <element name="B" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="{ if ($areParsing) then 0 else fn:string-length(../C) }" dfdl:outputValueCalc="{ ../C }" />  <element name="C" type="xs:string" dfdl:inputValueCalc="{ ../A }" /> On Mon, Jul 14, 2014 at 3:23 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: > The proposal for action 261 is that the dfdl:length expression is > evaluated, so that will stop what you propose from working. > > For the MIL-STD-2045 'fixedOrTerminated' case then the sequence suggestion > in the other email is much simpler. > > Regards > > Steve Hanson > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Steve Hanson/UK/IBM@IBMGB, > Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> > Date: 14/07/2014 19:51 > Subject: Re: Unparsing and choices - (related - Re: [DFDL-WG] Fw: > Action 233 (deferred) - "byte order not sufficient..." - draft document on > experience with binary format MIL-STD-2045 > ------------------------------ > > > > There has to be a way to get zero-length on parsing, but an > outputValueCalc string on unparsing. > > <defineVariable name="dummyLength" defaultValue="0" external='true'/> > > <element name="outputRaw" type="xs:string" > dfdl:lengthKind="explicit" dfdl:length="{ $tns:dummyLength }" > dfdl:outputValueCalc="{.....}" /> > > Would this work? The length expression isn't used when unparsing I believe > in that some other elements are supposed to be storing the length > information. That's the case here though the length value information isn't > being stored, what is being stored is a way to determine the length. > > Maybe there's something simpler? > > > > On Mon, Jul 14, 2014 at 1:42 PM, Steve Hanson <*smh(a)uk.ibm.com* > <smh(a)uk.ibm.com>> wrote: > Mike > > Section 17 of the spec says "It is a schema definition error if > dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same > element. " So outputRaw will give an SDE. > > Regards > > Steve Hanson > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:*+44-1962-815848* <%2B44-1962-815848> > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com* > <mbeckerle.dfdl(a)gmail.com>> > To: Steve Hanson/UK/IBM@IBMGB, > Cc: "*dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>" <*dfdl-wg(a)ogf.org* > <dfdl-wg(a)ogf.org>> > Date: 14/07/2014 16:43 > Subject: Unparsing and choices - (related - Re: [DFDL-WG] Fw: > Action 233 (deferred) - "byte order not sufficient..." - draft document on > experience with binary format MIL-STD-2045 > ------------------------------ > > > > Breaking out this one issue for a separate discussion thread. > > Consider the choice below. There is a choice of two different elements > named raw1 and raw2, then a value element which is computed based on one or > the other of the alternatives. > > <choice> >  > <element name="raw1" type="xs:string" > dfdl:lengthKind='pattern' > dfdl:lengthPattern="[^\x7F]{0,99}" > dfdl:terminator="%DEL;"/> > <element name="raw2" type="xs:string" > dfdl:lengthKind="explicit" > dfdl:length="100"/> > </choice> > <element name='value' type='xs:string' > dfdl:inputValueCalc='{ if (fn:exists( ../raw1 ) then ../raw1 else > ../raw2 }'/> > > Now, given the statement below: > > SMH: There is no way to make a choice branch fail when unparsing. (The > only 'backtracking' when unparsing a choice is when the infoset contains no > branch at all then the spec states that each branch is examined in turn > until one is found that successfully applies defaults. But that's not > really backtracking, as you can statically deduce the branch from the > schema alone, so the 'default' branch to use can be computed up front). > > So is there any way to fix this and make the above choice work? > > This is the only thing I can think of, which ends up with entirely > separate elements for parsing and unparsing. > > I am not sure this works or should work. > > <group name="hidG1"> > <sequence> >  > <element name="raw1" type="xs:string" > dfdl:lengthKind='pattern' > dfdl:lengthPattern="[^\x7F]{0,99}" > dfdl:terminator="%DEL;" > minOccurs='0' dfdl:occursCountKind='implicit'/> >  > <element name="raw2" type="xs:string" > dfdl:lengthKind="explicit" > dfdl:length="100" > minOccurs='0' dfdl:occursCountKind='expression' > dfdl:occursCount="{ if (fn:exists(../raw1) then 0 else 1 }"/> >  > <element name="outputRaw" type="xs:string" > dfdl:lengthKind="explicit" dfdl:length="{ 0 }" > dfdl:outputValueCalc="{ if (fn:string-length(../value) eq 100 then > ../value else fn:concat(../value, '') }" /> > <sequence> > </group> > >  > <sequence dfdl:hiddenGroupRef="tns:hidG1"/> > <element name='value' type='xs:string' > dfdl:inputValueCalc='{ if (fn:exists( ../raw1 ) then ../raw1 else > ../raw2 }'/> > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >

2 1

stop bit encoding
by Mike Beckerle 16 Jul '14

16 Jul '14

The FIX protocols (financial interchange) make use of stop bit encoding. I have heard of other places using this also. This is modern, current, non-legacy usage. This is the length encoding where the most-significant bit of a byte is used to determine whether there are more ASCII characters or not. The last byte will have the most-significant bit of 1, prior bytes most-significant bit of 0. Minimum length in this length kind is 1. It seems we are missing this lengthKind, which applies only to text in the ASCII/US-ASCII encoding. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call Minutes 2014-07-15
by Steve Hanson 15 Jul '14

15 Jul '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13280?download= Please review spec draft r15 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0