Action 233 (deferred) - "byte order not sufficient..." - draft document on experience with binary format MIL-STD-2045
I have created an experience document about the "bit order" issue, which was a deferred action 233, and the subject of a public comment. The document is here: http://redmine.ogf.org/dmsf_files/13268. The public comment item is http://redmine.ogf.org/boards/15/topics/43. It recommends a new dfdl:bitOrder property, and a new dfdl:byteOrder enum value, without which it is impossible to model these data formats. It also recommends several other improvements to DFDL to facilitate handling these data formats. The formats in question are a variety of MIL-STD formats which are all densely packed binary data. These formats are in broad use. MIL-STD-2045 is one part of this family and this particular format specification is generally available without any restrictions from a US DoD web site ( http://assistdocs.com) so I made this specific format the subject of the document as it illustrates all the problematic issues. We have implemented the dfdl:bitOrder property in Daffodil, and it works with some useful tests now passing. We have also enhanced our TDML implementation to enable creation of tests for this feature (and in the process actually found two bugs in the MIL-STD-2045 spec!). Both the property and this TDML enhancement are described in the document. The sponsors of the Daffodil project are extremely keen to get this needed binary support into the DFDL v1.0 standard so as to have multiple DFDL implementations support it. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>
Mike Please find attached IBM's initial comments to your experience document, as Word comments. We only got as far as the 3 x required extensions, not looked at the optional usability stuff in detail yet. We think we have our collective heads around the least significant bit ordering concept, but we think the explanation could be clearer and show the bits on-the-wire. Some debate as to whether this could be considered some variation of byteOrder but you've obviously thought this through and concluded a separate property is best. Also should bit order apply to text reps, given that byteOrder is binary rep only and any byte ordering variations in encodings are handled as separate encodings (eg, UTF-16LE and UTF-16BE). Regarding the US-ASCII-7-Bit-Packed encoding enum, this was added via erratum previously using the idea of DFDL-specific named encoding. But we are thinking that this could have been handled as an x- encoding, rather than specifically adding it to the spec. And thinking further on that same thread, should byteOrder be made to work like encoding and allow x- enums, then the new byteOrder would become a x- enum. The Wikipedia article you cite on Endianness mentions other byte orders (eg, Middle-Endian, PDP-Endian). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org>, Date: 24/06/2014 20:27 Subject: [DFDL-WG] Action 233 (deferred) - "byte order not sufficient..." - draft document on experience with binary format MIL-STD-2045 Sent by: dfdl-wg-bounces@ogf.org I have created an experience document about the "bit order" issue, which was a deferred action 233, and the subject of a public comment. The document is here: http://redmine.ogf.org/dmsf_files/13268. The public comment item is http://redmine.ogf.org/boards/15/topics/43. It recommends a new dfdl:bitOrder property, and a new dfdl:byteOrder enum value, without which it is impossible to model these data formats. It also recommends several other improvements to DFDL to facilitate handling these data formats. The formats in question are a variety of MIL-STD formats which are all densely packed binary data. These formats are in broad use. MIL-STD-2045 is one part of this family and this particular format specification is generally available without any restrictions from a US DoD web site ( http://assistdocs.com) so I made this specific format the subject of the document as it illustrates all the problematic issues. We have implemented the dfdl:bitOrder property in Daffodil, and it works with some useful tests now passing. We have also enhanced our TDML implementation to enable creation of tests for this feature (and in the process actually found two bugs in the MIL-STD-2045 spec!). Both the property and this TDML enhancement are described in the document. The sponsors of the Daffodil project are extremely keen to get this needed binary support into the DFDL v1.0 standard so as to have multiple DFDL implementations support it. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
participants (2)
-
Mike Beckerle -
Steve Hanson