- dfdl-wg - lists.ogf.org

Action Item: Versioning mechanism for DFDL v2.0 features/changes
by Mike Beckerle 23 Jan '18

23 Jan '18

There are some features now under discussion which will most likely not be part of the DFDL v1.0 standard, but a subsequent version 2.0 of the standard. Implementations may have these features prior to the existence of that draft version of the DFDL standard; hence, a mechanism for indicating the support, and requirements that implies, on DFDL schemas is needed in order for these v2.0 features to be introduced. The purpose of the versioning mechanism is to provide access to DFDL v2.0 features, while maintaining also backward compatibility with DFDL v1.0 schemas. For example, new DFDL format properties may be added to the DFDL v2.0 version; however, those properties are not required to be specified unless the version of the schema is specifically set to indicate it is using version 2.0 features. Also, deprecated features should only generate deprecation warnings if the user is requesting version 2.0 of DFDL, and should be silent otherwise. Proposal: New attribute on the dfdl:format annotation: DFDLVersion with valid values "1", "2". All values are reserved for future use. The DFDLVersion, if unspecified, defaults to "1" which means the implementation is of the DFDL v1.0 specification. If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL may be enforced on the schema, and schema definition errors will result if version 2.0 requirements are not met. DFDL Version 1.0 features that are deprecated will generate warnings. Example: <dfdl:format DFDLVersion="2" .../> Example: <dfdl:defineFormat name="myV2Format"> <dfdl:format DFDLVersion="2" .../> </dfdl:defineFormat> The DFDLVersion is not a format property and is not scoped. It can be added to a named format definition which is then referenced in the usual manner for named format definitions. Backward Compatibility Requirement DFDL Implementations that provide version 2.0 functionality must also provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends version 1.0. Mixed Version Schemas If a schema consists of multiple schema documents, and their respective dfdl:format annotations do not all contain the same value for the DFDLVersion attribute, then the schema is said to be "mixed version". This is allowed. A named dfdl:format carrying DFDLVersion "2" may not be referenced from a DFDL Schema document having DFDLVersion "1". The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or lacking the DFDLVersion attribute may be referenced from a DFDL Schema having DFDLVersion="2". There are really 3 ways that DFDL version 2.0 can differ from version 1.0, and these are that new features are added, old features are deprecated, or old features are removed. This latter is quite undesirable, but will be considered a possibility for purpose of discussion this versioning system. These will commonly be in combination such as if an old feature is deprecated with the replacement feature being added. These will now be discussed separately, with the emphasis on how mixed version schemas must work. New Features Added Each new required DFDL property introduced for version 2.0 will be defined along with what is called its "version 1.0 default value". When a mixed version schema exists, the DFDL v1.0 parts of it behave as if any required DFDLVersion 2 properties have their version 1.0 default value defined, at top level of the schema (as if defined with that default value in each DFDLVersion 1 schema document's dfdl:format annotation). Features Removed Incompatibilities are to be avoided wherever possible, and may not be needed at all. However, there is still the possibility that some feature will want to be entirely removed and replaced by something better, and in that case the versioning scheme needs to be able to support this. In that situation, the schema definition error for version 2 is only generated when a schema has DFDLVersion 2 specified (in dfdl:format explicitly, or by reference to a named format carrying or referencing DFDLVersion 2). Mixed schemas that contain schema documents entirely using version 1 of DFDL do not generate this schema definition error. Features deprecated Deprecation warnings are generated only when a DFDLVersion="2" attribute is present or is incorporated by reference (to a named format). A version 1 schema that is incorporated with a version 2 DFDL schema but is not modified in any way, does not generate deprecation warnings about version 1.0 features that have been deprecated in version 2.0. (Most likely deprecations would be revising property names or value enums for clarity.) Implementations may provide mechanisms for suppressing warnings. These are implementation dependent. Version 1.0 Existing Deprecations and Compatibility During the process of standardization for DFDL v1.0 a number of properties names were changed. DFDL version 1.0 implementations accept both the old and new property names. (Example: dfdl:separatorSuppressionPolicy, which replaced an older property name.) DFDL schemas that specify DFDLVersion="2" will not accept the older property names, and it is a schema definition error if the old (pre DFDL v1.0) property names are used. Properly constructed mixed version schemas can still depend on the older property names in the version 1.0 parts of the schema. ------------------------------ Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Latest OGF DFDL WG Call Minutes
by Steve Hanson 23 Jan '18

23 Jan '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for next OGF DFDL WG Call
by Steve Hanson 23 Jan '18

23 Jan '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13571?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Example format with bitOrder leastSignificantBitFirst, but byteOrder bigEndian
by Mike Beckerle 11 Jan '18

11 Jan '18

This format, which is a envelope format for an envelope-and-payload style of data, describes data as bigEndian, but LSBF. https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Comman… Thanks to Marshall Miller of Quark Security for this insight. This extracted from the dev(a)daffodil.apache.org mailing list. One can workaround this problem in Daffodil by way of dfdl:inputValueCalc/dfdl:outputValueCalc and hidden groups. But that is, perhaps, unsatisfying. Example: Data is comprised of 3 unsigned integer bitfields: slot1 is 12 bits slot2 is 32 bits slot3 is 20 bits Given an input with the following bytes in hex: 01 23 45 67 89 AB CD EF Parsed results need to be (in hex): slot1: 0567 slot2: DEF01234 slot3: 089ABC Explanation of results: Since the bit fields need to consume data in a leastSignificantBitFirst order, it is easier to represent the input in right-to-left order where bits are consumed starting at the right and moving toward the left. Since words are 4 bytes, and in big endian order, the lowest bit in the 4th byte is the first bit to be consumed. Reordering the input data and consuming data from right to left yields the correct field boundaries. 89 AB CD EF 01 23 45 67 parse fields right to left 05 06 07 08 01 02 03 04 input byte position slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles, and slot3 consumes the left 5 nibbles. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 09 Jan '18

09 Jan '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for next OGF DFDL WG Call
by Steve Hanson 09 Jan '18

09 Jan '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13570?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group Call
by Steve Hanson 08 Jan '18

08 Jan '18

Calls for 2018. OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 Netherlands Toll-Free 0-800-363-6036 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg DFDL Getting Started: http://www.ibm.com/developerworks/library/se-dfdl/ DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 12 Dec '17

12 Dec '17

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for next OGF DFDL WG Call
by Steve Hanson 12 Dec '17

12 Dec '17

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13569?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 12 Dec '17

12 Dec '17

1 0