Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes
So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it. I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features. The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as http://www.xml.com/pub/a/2004/07/21/design.html These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself. The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <smh@uk.ibm.com> wrote:
Mike,
Your scheme as proposed has a couple of problems. a) It is possible to create a schema that does not use dfdl:format at all. Unlikely but possible. With such a schema there is no way to indicate version. b) What are the version implications for annotations that have nothing to do with dfdl:format? Asserts, discriminators, variables.
Most of the language you use refers to versions of *schemas*. That's not the same as version of a dfdl:format. Any reason you didn't just change the DFDL namespace URL to be *http://www.ogf.org/dfdl/dfdl-2.0/* <http://www.ogf.org/dfdl/dfdl-2.0/>, which must be declared before any DFDL feature can be used? I think that was always the intent of the DFDL namespace.
The key piece of work is validating any proposal against the scoping rules in section 8 of the spec.
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 <+44%201962%20815848> mob:+44-7717-378890 <+44%207717%20378890> Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: dfdl-wg@ogf.org Date: 23/01/2018 16:03 Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> ------------------------------
There are some features now under discussion which will most likely not be part of the DFDL v1.0 standard, but a subsequent version 2.0 of the standard.
Implementations may have these features prior to the existence of that draft version of the DFDL standard; hence, a mechanism for indicating the support, and requirements that implies, on DFDL schemas is needed in order for these v2.0 features to be introduced.
The purpose of the versioning mechanism is to provide access to DFDL v2.0 features, while maintaining also backward compatibility with DFDL v1.0 schemas.
For example, new DFDL format properties may be added to the DFDL v2.0 version; however, those properties are not required to be specified unless the version of the schema is specifically set to indicate it is using version 2.0 features. Also, deprecated features should only generate deprecation warnings if the user is requesting version 2.0 of DFDL, and should be silent otherwise.
Proposal:
New attribute on the dfdl:format annotation: DFDLVersion with valid values "1", "2". All values are reserved for future use.
The DFDLVersion, if unspecified, defaults to "1" which means the implementation is of the DFDL v1.0 specification.
If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL may be enforced on the schema, and schema definition errors will result if version 2.0 requirements are not met. DFDL Version 1.0 features that are deprecated will generate warnings.
Example:
<dfdl:format DFDLVersion="2" .../>
Example:
<dfdl:defineFormat name="myV2Format"> <dfdl:format DFDLVersion="2" .../> </dfdl:defineFormat>
The DFDLVersion is not a format property and is not scoped. It can be added to a named format definition which is then referenced in the usual manner for named format definitions.
Backward Compatibility Requirement
DFDL Implementations that provide version 2.0 functionality must also provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends version 1.0.
Mixed Version Schemas
If a schema consists of multiple schema documents, and their respective dfdl:format annotations do not all contain the same value for the DFDLVersion attribute, then the schema is said to be "mixed version". This is allowed.
A named dfdl:format carrying DFDLVersion "2" may not be referenced from a DFDL Schema document having DFDLVersion "1".
The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or lacking the DFDLVersion attribute may be referenced from a DFDL Schema having DFDLVersion="2".
There are really 3 ways that DFDL version 2.0 can differ from version 1.0, and these are that new features are added, old features are deprecated, or old features are removed. This latter is quite undesirable, but will be considered a possibility for purpose of discussion this versioning system.
These will commonly be in combination such as if an old feature is deprecated with the replacement feature being added.
These will now be discussed separately, with the emphasis on how mixed version schemas must work.
New Features Added
Each new required DFDL property introduced for version 2.0 will be defined along with what is called its "version 1.0 default value". When a mixed version schema exists, the DFDL v1.0 parts of it behave as if any required DFDLVersion 2 properties have their version 1.0 default value defined, at top level of the schema (as if defined with that default value in each DFDLVersion 1 schema document's dfdl:format annotation).
Features Removed
Incompatibilities are to be avoided wherever possible, and may not be needed at all. However, there is still the possibility that some feature will want to be entirely removed and replaced by something better, and in that case the versioning scheme needs to be able to support this.
In that situation, the schema definition error for version 2 is only generated when a schema has DFDLVersion 2 specified (in dfdl:format explicitly, or by reference to a named format carrying or referencing DFDLVersion 2). Mixed schemas that contain schema documents entirely using version 1 of DFDL do not generate this schema definition error.
Features deprecated
Deprecation warnings are generated only when a DFDLVersion="2" attribute is present or is incorporated by reference (to a named format). A version 1 schema that is incorporated with a version 2 DFDL schema but is not modified in any way, does not generate deprecation warnings about version 1.0 features that have been deprecated in version 2.0.
(Most likely deprecations would be revising property names or value enums for clarity.)
Implementations may provide mechanisms for suppressing warnings. These are implementation dependent.
Version 1.0 Existing Deprecations and Compatibility
During the process of standardization for DFDL v1.0 a number of properties names were changed. DFDL version 1.0 implementations accept both the old and new property names. (Example: dfdl:separatorSuppressionPolicy, which replaced an older property name.) DFDL schemas that specify DFDLVersion="2" will not accept the older property names, and it is a schema definition error if the old (pre DFDL v1.0) property names are used.
Properly constructed mixed version schemas can still depend on the older property names in the version 1.0 parts of the schema.
------------------------------
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=lDIa8xvu2n-e8PXWCJmajfYaXESMqvvOWpg1HMWeuGQ&s=PgUsR-jesgrC3vQjvdBMwkK8vL7yDFnY4bQ3w7-sXBQ&e=> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5Fpolicies.php&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=lDIa8xvu2n-e8PXWCJmajfYaXESMqvvOWpg1HMWeuGQ&s=RTtCEaAWUKcmc-Fiv2pJtx8i9kJ97-J7rgB0zvQTfsc&e=> -- dfdl-wg mailing list dfdl-wg@ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www. ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_ iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=lDIa8xvu2n-e8 PXWCJmajfYaXESMqvvOWpg1HMWeuGQ&s=hdO_VQ_p-1f7JfZPcSTPR1S_Y1a nSBXJWarqPdJZdPc&e=
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Mike I'd like to make some progress on this if possible? The IBM z/TPF team are a heavy user of IBM DFDL. They are interested in adding offset support and also support for pointers (more on the latter on next WG call). A problem with using DFDLVersion="2" is that it is pre-judging what is going to be in 2.0. I would prefer that new features appeared in DFDL 1.0 schemas as 'experimental' with their DFDL properties in a separate namespace. They would switch to the DFDL namespace only in a 2.0 schema. This gives us flexibility around property naming and behaviour - we might decide to change things when formalising 2.0. For example this is what z/TPF have done for pointers: <xs:element name="myString" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="../len" tddt:indirectKind="pointer" tddt:indirectLength="4" /> It means that when a user migrates to DFDL 2.0 they would need to change their schemas but I think that is preferable to the current proposal where 2.0 bleeds into 1.0. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: dfdl-wg@ogf.org Date: 02/02/2018 22:07 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it. I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features. The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as http://www.xml.com/pub/a/2004/07/21/design.html These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself. The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <smh@uk.ibm.com> wrote: Mike, Your scheme as proposed has a couple of problems. a) It is possible to create a schema that does not use dfdl:format at all. Unlikely but possible. With such a schema there is no way to indicate version. b) What are the version implications for annotations that have nothing to do with dfdl:format? Asserts, discriminators, variables. Most of the language you use refers to versions of schemas. That's not the same as version of a dfdl:format. Any reason you didn't just change the DFDL namespace URL to be http://www.ogf.org/dfdl/dfdl-2.0/, which must be declared before any DFDL feature can be used? I think that was always the intent of the DFDL namespace. The key piece of work is validating any proposal against the scoping rules in section 8 of the spec. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: dfdl-wg@ogf.org Date: 23/01/2018 16:03 Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> There are some features now under discussion which will most likely not be part of the DFDL v1.0 standard, but a subsequent version 2.0 of the standard. Implementations may have these features prior to the existence of that draft version of the DFDL standard; hence, a mechanism for indicating the support, and requirements that implies, on DFDL schemas is needed in order for these v2.0 features to be introduced. The purpose of the versioning mechanism is to provide access to DFDL v2.0 features, while maintaining also backward compatibility with DFDL v1.0 schemas. For example, new DFDL format properties may be added to the DFDL v2.0 version; however, those properties are not required to be specified unless the version of the schema is specifically set to indicate it is using version 2.0 features. Also, deprecated features should only generate deprecation warnings if the user is requesting version 2.0 of DFDL, and should be silent otherwise. Proposal: New attribute on the dfdl:format annotation: DFDLVersion with valid values "1", "2". All values are reserved for future use. The DFDLVersion, if unspecified, defaults to "1" which means the implementation is of the DFDL v1.0 specification. If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL may be enforced on the schema, and schema definition errors will result if version 2.0 requirements are not met. DFDL Version 1.0 features that are deprecated will generate warnings. Example: <dfdl:format DFDLVersion="2" .../> Example: <dfdl:defineFormat name="myV2Format"> <dfdl:format DFDLVersion="2" .../> </dfdl:defineFormat> The DFDLVersion is not a format property and is not scoped. It can be added to a named format definition which is then referenced in the usual manner for named format definitions. Backward Compatibility Requirement DFDL Implementations that provide version 2.0 functionality must also provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends version 1.0. Mixed Version Schemas If a schema consists of multiple schema documents, and their respective dfdl:format annotations do not all contain the same value for the DFDLVersion attribute, then the schema is said to be "mixed version". This is allowed. A named dfdl:format carrying DFDLVersion "2" may not be referenced from a DFDL Schema document having DFDLVersion "1". The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or lacking the DFDLVersion attribute may be referenced from a DFDL Schema having DFDLVersion="2". There are really 3 ways that DFDL version 2.0 can differ from version 1.0, and these are that new features are added, old features are deprecated, or old features are removed. This latter is quite undesirable, but will be considered a possibility for purpose of discussion this versioning system. These will commonly be in combination such as if an old feature is deprecated with the replacement feature being added. These will now be discussed separately, with the emphasis on how mixed version schemas must work. New Features Added Each new required DFDL property introduced for version 2.0 will be defined along with what is called its "version 1.0 default value". When a mixed version schema exists, the DFDL v1.0 parts of it behave as if any required DFDLVersion 2 properties have their version 1.0 default value defined, at top level of the schema (as if defined with that default value in each DFDLVersion 1 schema document's dfdl:format annotation). Features Removed Incompatibilities are to be avoided wherever possible, and may not be needed at all. However, there is still the possibility that some feature will want to be entirely removed and replaced by something better, and in that case the versioning scheme needs to be able to support this. In that situation, the schema definition error for version 2 is only generated when a schema has DFDLVersion 2 specified (in dfdl:format explicitly, or by reference to a named format carrying or referencing DFDLVersion 2). Mixed schemas that contain schema documents entirely using version 1 of DFDL do not generate this schema definition error. Features deprecated Deprecation warnings are generated only when a DFDLVersion="2" attribute is present or is incorporated by reference (to a named format). A version 1 schema that is incorporated with a version 2 DFDL schema but is not modified in any way, does not generate deprecation warnings about version 1.0 features that have been deprecated in version 2.0. (Most likely deprecations would be revising property names or value enums for clarity.) Implementations may provide mechanisms for suppressing warnings. These are implementation dependent. Version 1.0 Existing Deprecations and Compatibility During the process of standardization for DFDL v1.0 a number of properties names were changed. DFDL version 1.0 implementations accept both the old and new property names. (Example: dfdl:separatorSuppressionPolicy, which replaced an older property name.) DFDL schemas that specify DFDLVersion="2" will not accept the older property names, and it is a schema definition error if the old (pre DFDL v1.0) property names are used. Properly constructed mixed version schemas can still depend on the older property names in the version 1.0 parts of the schema. ------------------------------ Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=lDIa8xvu2n-e8PXWCJmajfYaXESMqvvOWpg1HMWeuGQ&s=hdO_VQ_p-1f7JfZPcSTPR1S_Y1anSBXJWarqPdJZdPc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=uWXWZYFvwxVucKlexXAl0l-fbTxrG8hnuudlaTlVXHM&s=_2LlmN7csTB474Wj0hWGssBKMzIdZqVVXl0QBdc5e2U&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
This seems reasonable to me. A few questions and suggestions. How are these properties expressed in long form annotations or property-element form annotations? I.e., <dfdl:format ..../> normally the attributes that short form have "dfdl:" would not have any namespace. Would the extension properties that show up inside dfdl:format carry their namespace there also? As in <dfdl:format tddt:pointerEnable="true"/> ? We also would need a way to make ultra long form properties work, e.g., <dfdl:element> <dfdl:property name="tddt:pointerEnable">true</dfdl:property> </dfdl:element> It's odd at first in any XSD-embedded language for a name attribute to take a QName as value. In this case I think it is proper. Really the name attribute isn't creating a name here, it's referencing a name. Assuming all the above notational issues are resolvable,..... Having a agreed upon URI for extension properties makes sense, because we can check if it is bound, and can validate short-form properties in that namespace. We need some way of looking at the schema at the top, and saying "this has extensions". There is a precedent for requiring a namespace binding to be at the root of a schema file, which is the feature where we allow include/import of non-DFDL XSD schemas that define other annotation languages. Such annotation languages can use attributes for example, though DFDL schemas cannot. When there is no binding of the dfdl namespace on the root of a schema, we don't include it for schema compilation purposes, though it is included for schema validation purposes. Given that precedent, we could require a binding of this extension URI at the root of the schema, rather than just allowing it anywhere. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Fri, Feb 1, 2019 at 4:34 AM Steve Hanson <smh@uk.ibm.com> wrote:
Mike
I'd like to make some progress on this if possible? The IBM z/TPF team are a heavy user of IBM DFDL. They are interested in adding offset support and also support for pointers (more on the latter on next WG call).
A problem with using DFDLVersion="2" is that it is pre-judging what is going to be in 2.0. I would prefer that new features appeared in DFDL 1.0 schemas as 'experimental' with their DFDL properties in a separate namespace. They would switch to the DFDL namespace only in a 2.0 schema. This gives us flexibility around property naming and behaviour - we might decide to change things when formalising 2.0.
For example this is what z/TPF have done for pointers:
<xs:element name="myString" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="../len" tddt:indirectKind="pointer" tddt:indirectLength="4" />
It means that when a user migrates to DFDL 2.0 they would need to change their schemas but I think that is preferable to the current proposal where 2.0 bleeds into 1.0.
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: dfdl-wg@ogf.org Date: 02/02/2018 22:07 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> ------------------------------
So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it.
I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features.
The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as *http://www.xml.com/pub/a/2004/07/21/design.html * <http://www.xml.com/pub/a/2004/07/21/design.html>
These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself.
The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: Mike,
Your scheme as proposed has a couple of problems. a) It is possible to create a schema that does not use dfdl:format at all. Unlikely but possible. With such a schema there is no way to indicate version. b) What are the version implications for annotations that have nothing to do with dfdl:format? Asserts, discriminators, variables.
Most of the language you use refers to versions of *schemas*. That's not the same as version of a dfdl:format. Any reason you didn't just change the DFDL namespace URL to be *http://www.ogf.org/dfdl/dfdl-2.0/* <http://www.ogf.org/dfdl/dfdl-2.0/>, which must be declared before any DFDL feature can be used? I think that was always the intent of the DFDL namespace.
The key piece of work is validating any proposal against the scoping rules in section 8 of the spec.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:*+44-1962-815848* <+44%201962%20815848> mob:*+44-7717-378890* <+44%207717%20378890> Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> Date: 23/01/2018 16:03 Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
There are some features now under discussion which will most likely not be part of the DFDL v1.0 standard, but a subsequent version 2.0 of the standard.
Implementations may have these features prior to the existence of that draft version of the DFDL standard; hence, a mechanism for indicating the support, and requirements that implies, on DFDL schemas is needed in order for these v2.0 features to be introduced.
The purpose of the versioning mechanism is to provide access to DFDL v2.0 features, while maintaining also backward compatibility with DFDL v1.0 schemas.
For example, new DFDL format properties may be added to the DFDL v2.0 version; however, those properties are not required to be specified unless the version of the schema is specifically set to indicate it is using version 2.0 features. Also, deprecated features should only generate deprecation warnings if the user is requesting version 2.0 of DFDL, and should be silent otherwise.
Proposal:
New attribute on the dfdl:format annotation: DFDLVersion with valid values "1", "2". All values are reserved for future use.
The DFDLVersion, if unspecified, defaults to "1" which means the implementation is of the DFDL v1.0 specification.
If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL may be enforced on the schema, and schema definition errors will result if version 2.0 requirements are not met. DFDL Version 1.0 features that are deprecated will generate warnings.
Example:
<dfdl:format DFDLVersion="2" .../>
Example:
<dfdl:defineFormat name="myV2Format"> <dfdl:format DFDLVersion="2" .../> </dfdl:defineFormat>
The DFDLVersion is not a format property and is not scoped. It can be added to a named format definition which is then referenced in the usual manner for named format definitions.
Backward Compatibility Requirement
DFDL Implementations that provide version 2.0 functionality must also provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends version 1.0.
Mixed Version Schemas
If a schema consists of multiple schema documents, and their respective dfdl:format annotations do not all contain the same value for the DFDLVersion attribute, then the schema is said to be "mixed version". This is allowed.
A named dfdl:format carrying DFDLVersion "2" may not be referenced from a DFDL Schema document having DFDLVersion "1".
The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or lacking the DFDLVersion attribute may be referenced from a DFDL Schema having DFDLVersion="2".
There are really 3 ways that DFDL version 2.0 can differ from version 1.0, and these are that new features are added, old features are deprecated, or old features are removed. This latter is quite undesirable, but will be considered a possibility for purpose of discussion this versioning system.
These will commonly be in combination such as if an old feature is deprecated with the replacement feature being added.
These will now be discussed separately, with the emphasis on how mixed version schemas must work.
New Features Added
Each new required DFDL property introduced for version 2.0 will be defined along with what is called its "version 1.0 default value". When a mixed version schema exists, the DFDL v1.0 parts of it behave as if any required DFDLVersion 2 properties have their version 1.0 default value defined, at top level of the schema (as if defined with that default value in each DFDLVersion 1 schema document's dfdl:format annotation).
Features Removed
Incompatibilities are to be avoided wherever possible, and may not be needed at all. However, there is still the possibility that some feature will want to be entirely removed and replaced by something better, and in that case the versioning scheme needs to be able to support this.
In that situation, the schema definition error for version 2 is only generated when a schema has DFDLVersion 2 specified (in dfdl:format explicitly, or by reference to a named format carrying or referencing DFDLVersion 2). Mixed schemas that contain schema documents entirely using version 1 of DFDL do not generate this schema definition error.
Features deprecated
Deprecation warnings are generated only when a DFDLVersion="2" attribute is present or is incorporated by reference (to a named format). A version 1 schema that is incorporated with a version 2 DFDL schema but is not modified in any way, does not generate deprecation warnings about version 1.0 features that have been deprecated in version 2.0.
(Most likely deprecations would be revising property names or value enums for clarity.)
Implementations may provide mechanisms for suppressing warnings. These are implementation dependent.
Version 1.0 Existing Deprecations and Compatibility
During the process of standardization for DFDL v1.0 a number of properties names were changed. DFDL version 1.0 implementations accept both the old and new property names. (Example: dfdl:separatorSuppressionPolicy, which replaced an older property name.) DFDL schemas that specify DFDLVersion="2" will not accept the older property names, and it is a schema definition error if the old (pre DFDL v1.0) property names are used.
Properly constructed mixed version schemas can still depend on the older property names in the version 1.0 parts of the schema.
------------------------------
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg* <https://www.ogf.org/mailman/listinfo/dfdl-wg>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
These are just experimental properties, we don't have to roll out the full property syntax treatment, we just need to do the minimum to allow their use. I propose the following: - New namespace "http://www.ogf.org/dfdl/dfdl-1.0/experimental" with default prefix "dfdlx" - Must be bound on the root of the schema only - Attribute form not allowed - Element form not allowed - Short form allowed via dfdlx:myNewProp - Scoping rules do not apply There is a problem if we allow scoping. Scoping means that you can not be silent about the value of a property when it is needed by an object. But by definition we do not know the full set of experimental properties, they will change over time, so schemas using the experimental namespace would throw new SDEs out of nowhere if a processor started to support a new experimental feature. We could allow scoping if we changed the rules for experimental properties so they have a built-in default (we have been forced to do that when errata have added new properties to the core 1.0 spec - but there have only been one or two). If so the prefixed property would appear in attribute form on DFDL annotation elements, ie, dfdlx:myNewProp. If we really need to allow element form then that could be done via a new dfdlx:property annotation element. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org>, Bradd Kadlecik <braddk@us.ibm.com> Date: 01/02/2019 14:34 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes This seems reasonable to me. A few questions and suggestions. How are these properties expressed in long form annotations or property-element form annotations? I.e., <dfdl:format ..../> normally the attributes that short form have "dfdl:" would not have any namespace. Would the extension properties that show up inside dfdl:format carry their namespace there also? As in <dfdl:format tddt:pointerEnable="true"/> ? We also would need a way to make ultra long form properties work, e.g., <dfdl:element> <dfdl:property name="tddt:pointerEnable">true</dfdl:property> </dfdl:element> It's odd at first in any XSD-embedded language for a name attribute to take a QName as value. In this case I think it is proper. Really the name attribute isn't creating a name here, it's referencing a name. Assuming all the above notational issues are resolvable,..... Having a agreed upon URI for extension properties makes sense, because we can check if it is bound, and can validate short-form properties in that namespace. We need some way of looking at the schema at the top, and saying "this has extensions". There is a precedent for requiring a namespace binding to be at the root of a schema file, which is the feature where we allow include/import of non-DFDL XSD schemas that define other annotation languages. Such annotation languages can use attributes for example, though DFDL schemas cannot. When there is no binding of the dfdl namespace on the root of a schema, we don't include it for schema compilation purposes, though it is included for schema validation purposes. Given that precedent, we could require a binding of this extension URI at the root of the schema, rather than just allowing it anywhere. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Feb 1, 2019 at 4:34 AM Steve Hanson <smh@uk.ibm.com> wrote: Mike I'd like to make some progress on this if possible? The IBM z/TPF team are a heavy user of IBM DFDL. They are interested in adding offset support and also support for pointers (more on the latter on next WG call). A problem with using DFDLVersion="2" is that it is pre-judging what is going to be in 2.0. I would prefer that new features appeared in DFDL 1.0 schemas as 'experimental' with their DFDL properties in a separate namespace. They would switch to the DFDL namespace only in a 2.0 schema. This gives us flexibility around property naming and behaviour - we might decide to change things when formalising 2.0. For example this is what z/TPF have done for pointers: <xs:element name="myString" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="../len" tddt:indirectKind="pointer" tddt:indirectLength="4" /> It means that when a user migrates to DFDL 2.0 they would need to change their schemas but I think that is preferable to the current proposal where 2.0 bleeds into 1.0. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: dfdl-wg@ogf.org Date: 02/02/2018 22:07 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it. I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features. The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as http://www.xml.com/pub/a/2004/07/21/design.html These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself. The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <smh@uk.ibm.com> wrote: Mike, Your scheme as proposed has a couple of problems. a) It is possible to create a schema that does not use dfdl:format at all. Unlikely but possible. With such a schema there is no way to indicate version. b) What are the version implications for annotations that have nothing to do with dfdl:format? Asserts, discriminators, variables. Most of the language you use refers to versions of schemas. That's not the same as version of a dfdl:format. Any reason you didn't just change the DFDL namespace URL to be http://www.ogf.org/dfdl/dfdl-2.0/, which must be declared before any DFDL feature can be used? I think that was always the intent of the DFDL namespace. The key piece of work is validating any proposal against the scoping rules in section 8 of the spec. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: dfdl-wg@ogf.org Date: 23/01/2018 16:03 Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> There are some features now under discussion which will most likely not be part of the DFDL v1.0 standard, but a subsequent version 2.0 of the standard. Implementations may have these features prior to the existence of that draft version of the DFDL standard; hence, a mechanism for indicating the support, and requirements that implies, on DFDL schemas is needed in order for these v2.0 features to be introduced. The purpose of the versioning mechanism is to provide access to DFDL v2.0 features, while maintaining also backward compatibility with DFDL v1.0 schemas. For example, new DFDL format properties may be added to the DFDL v2.0 version; however, those properties are not required to be specified unless the version of the schema is specifically set to indicate it is using version 2.0 features. Also, deprecated features should only generate deprecation warnings if the user is requesting version 2.0 of DFDL, and should be silent otherwise. Proposal: New attribute on the dfdl:format annotation: DFDLVersion with valid values "1", "2". All values are reserved for future use. The DFDLVersion, if unspecified, defaults to "1" which means the implementation is of the DFDL v1.0 specification. If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL may be enforced on the schema, and schema definition errors will result if version 2.0 requirements are not met. DFDL Version 1.0 features that are deprecated will generate warnings. Example: <dfdl:format DFDLVersion="2" .../> Example: <dfdl:defineFormat name="myV2Format"> <dfdl:format DFDLVersion="2" .../> </dfdl:defineFormat> The DFDLVersion is not a format property and is not scoped. It can be added to a named format definition which is then referenced in the usual manner for named format definitions. Backward Compatibility Requirement DFDL Implementations that provide version 2.0 functionality must also provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends version 1.0. Mixed Version Schemas If a schema consists of multiple schema documents, and their respective dfdl:format annotations do not all contain the same value for the DFDLVersion attribute, then the schema is said to be "mixed version". This is allowed. A named dfdl:format carrying DFDLVersion "2" may not be referenced from a DFDL Schema document having DFDLVersion "1". The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or lacking the DFDLVersion attribute may be referenced from a DFDL Schema having DFDLVersion="2". There are really 3 ways that DFDL version 2.0 can differ from version 1.0, and these are that new features are added, old features are deprecated, or old features are removed. This latter is quite undesirable, but will be considered a possibility for purpose of discussion this versioning system. These will commonly be in combination such as if an old feature is deprecated with the replacement feature being added. These will now be discussed separately, with the emphasis on how mixed version schemas must work. New Features Added Each new required DFDL property introduced for version 2.0 will be defined along with what is called its "version 1.0 default value". When a mixed version schema exists, the DFDL v1.0 parts of it behave as if any required DFDLVersion 2 properties have their version 1.0 default value defined, at top level of the schema (as if defined with that default value in each DFDLVersion 1 schema document's dfdl:format annotation). Features Removed Incompatibilities are to be avoided wherever possible, and may not be needed at all. However, there is still the possibility that some feature will want to be entirely removed and replaced by something better, and in that case the versioning scheme needs to be able to support this. In that situation, the schema definition error for version 2 is only generated when a schema has DFDLVersion 2 specified (in dfdl:format explicitly, or by reference to a named format carrying or referencing DFDLVersion 2). Mixed schemas that contain schema documents entirely using version 1 of DFDL do not generate this schema definition error. Features deprecated Deprecation warnings are generated only when a DFDLVersion="2" attribute is present or is incorporated by reference (to a named format). A version 1 schema that is incorporated with a version 2 DFDL schema but is not modified in any way, does not generate deprecation warnings about version 1.0 features that have been deprecated in version 2.0. (Most likely deprecations would be revising property names or value enums for clarity.) Implementations may provide mechanisms for suppressing warnings. These are implementation dependent. Version 1.0 Existing Deprecations and Compatibility During the process of standardization for DFDL v1.0 a number of properties names were changed. DFDL version 1.0 implementations accept both the old and new property names. (Example: dfdl:separatorSuppressionPolicy, which replaced an older property name.) DFDL schemas that specify DFDLVersion="2" will not accept the older property names, and it is a schema definition error if the old (pre DFDL v1.0) property names are used. Properly constructed mixed version schemas can still depend on the older property names in the version 1.0 parts of the schema. ------------------------------ Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
I think we have to allow scoping rules to apply to new experimental properties if they are regular format properties. Some proposed extensions would be much too hard to use without scoping being available. But some new proposed extensions aren't the kind of thing one puts in scope anyway, so like existing non-scoped properties, new extension properties could be scoped or not-scoped depending on their nature. I think any new experimental property is going to also need a default value for it, for backward compatibility as you suggest. Since these are in namespace dfdlx, we don't have to worry about turning OFF this default. Once a property is accepted into the DFDL spec formally, then implementations probably should have switches for turning on/off defaults for it, as for new properties that have been added since the earliest implementation were already in existence. There are other things needed as extensions that aren't properties however. Ex: we've had requests for %LSP; (and +, * variants) character class entity. This is like %WSP; without the NLs., i.e., intra-line whitespace characters. I expect we want some property-like way to enable/disable use of such an extension feature. (With the default being to disallow.) ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Thu, Apr 4, 2019 at 10:59 AM Steve Hanson <smh@uk.ibm.com> wrote:
These are just experimental properties, we don't have to roll out the full property syntax treatment, we just need to do the minimum to allow their use. I propose the following:
- New namespace "http://www.ogf.org/dfdl/dfdl-1.0/experimental" with default prefix "dfdlx" - Must be bound on the root of the schema only - Attribute form not allowed - Element form not allowed - Short form allowed via dfdlx:myNewProp - Scoping rules do not apply
There is a problem if we allow scoping. Scoping means that you can not be silent about the value of a property when it is needed by an object. But by definition we do not know the full set of experimental properties, they will change over time, so schemas using the experimental namespace would throw new SDEs out of nowhere if a processor started to support a new experimental feature.
We could allow scoping if we changed the rules for experimental properties so they have a built-in default (we have been forced to do that when errata have added new properties to the core 1.0 spec - but there have only been one or two). If so the prefixed property would appear in attribute form on DFDL annotation elements, ie, dfdlx:myNewProp.
If we really need to allow element form then that could be done via a new dfdlx:property annotation element.
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org>, Bradd Kadlecik <braddk@us.ibm.com> Date: 01/02/2019 14:34 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes ------------------------------
This seems reasonable to me.
A few questions and suggestions.
How are these properties expressed in long form annotations or property-element form annotations?
I.e., <dfdl:format ..../> normally the attributes that short form have "dfdl:" would not have any namespace. Would the extension properties that show up inside dfdl:format carry their namespace there also? As in <dfdl:format tddt:pointerEnable="true"/> ?
We also would need a way to make ultra long form properties work, e.g.,
<dfdl:element> <dfdl:property name="tddt:pointerEnable">true</dfdl:property> </dfdl:element>
It's odd at first in any XSD-embedded language for a name attribute to take a QName as value. In this case I think it is proper. Really the name attribute isn't creating a name here, it's referencing a name.
Assuming all the above notational issues are resolvable,.....
Having a agreed upon URI for extension properties makes sense, because we can check if it is bound, and can validate short-form properties in that namespace.
We need some way of looking at the schema at the top, and saying "this has extensions".
There is a precedent for requiring a namespace binding to be at the root of a schema file, which is the feature where we allow include/import of non-DFDL XSD schemas that define other annotation languages. Such annotation languages can use attributes for example, though DFDL schemas cannot. When there is no binding of the dfdl namespace on the root of a schema, we don't include it for schema compilation purposes, though it is included for schema validation purposes.
Given that precedent, we could require a binding of this extension URI at the root of the schema, rather than just allowing it anywhere.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Fri, Feb 1, 2019 at 4:34 AM Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: Mike
I'd like to make some progress on this if possible? The IBM z/TPF team are a heavy user of IBM DFDL. They are interested in adding offset support and also support for pointers (more on the latter on next WG call).
A problem with using DFDLVersion="2" is that it is pre-judging what is going to be in 2.0. I would prefer that new features appeared in DFDL 1.0 schemas as 'experimental' with their DFDL properties in a separate namespace. They would switch to the DFDL namespace only in a 2.0 schema. This gives us flexibility around property naming and behaviour - we might decide to change things when formalising 2.0.
For example this is what z/TPF have done for pointers:
<xs:element name="myString" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="../len" tddt:indirectKind="pointer" tddt:indirectLength="4" />
It means that when a user migrates to DFDL 2.0 they would need to change their schemas but I think that is preferable to the current proposal where 2.0 bleeds into 1.0.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> Date: 02/02/2018 22:07 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it.
I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features.
The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as *http://www.xml.com/pub/a/2004/07/21/design.html * <http://www.xml.com/pub/a/2004/07/21/design.html>
These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself.
The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: Mike,
Your scheme as proposed has a couple of problems. a) It is possible to create a schema that does not use dfdl:format at all. Unlikely but possible. With such a schema there is no way to indicate version. b) What are the version implications for annotations that have nothing to do with dfdl:format? Asserts, discriminators, variables.
Most of the language you use refers to versions of *schemas*. That's not the same as version of a dfdl:format. Any reason you didn't just change the DFDL namespace URL to be *http://www.ogf.org/dfdl/dfdl-2.0/* <http://www.ogf.org/dfdl/dfdl-2.0/>, which must be declared before any DFDL feature can be used? I think that was always the intent of the DFDL namespace.
The key piece of work is validating any proposal against the scoping rules in section 8 of the spec.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:*+44-1962-815848* <+44%201962%20815848> mob:*+44-7717-378890* <+44%207717%20378890> Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> Date: 23/01/2018 16:03 Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
There are some features now under discussion which will most likely not be part of the DFDL v1.0 standard, but a subsequent version 2.0 of the standard.
Implementations may have these features prior to the existence of that draft version of the DFDL standard; hence, a mechanism for indicating the support, and requirements that implies, on DFDL schemas is needed in order for these v2.0 features to be introduced.
The purpose of the versioning mechanism is to provide access to DFDL v2.0 features, while maintaining also backward compatibility with DFDL v1.0 schemas.
For example, new DFDL format properties may be added to the DFDL v2.0 version; however, those properties are not required to be specified unless the version of the schema is specifically set to indicate it is using version 2.0 features. Also, deprecated features should only generate deprecation warnings if the user is requesting version 2.0 of DFDL, and should be silent otherwise.
Proposal:
New attribute on the dfdl:format annotation: DFDLVersion with valid values "1", "2". All values are reserved for future use.
The DFDLVersion, if unspecified, defaults to "1" which means the implementation is of the DFDL v1.0 specification.
If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL may be enforced on the schema, and schema definition errors will result if version 2.0 requirements are not met. DFDL Version 1.0 features that are deprecated will generate warnings.
Example:
<dfdl:format DFDLVersion="2" .../>
Example:
<dfdl:defineFormat name="myV2Format"> <dfdl:format DFDLVersion="2" .../> </dfdl:defineFormat>
The DFDLVersion is not a format property and is not scoped. It can be added to a named format definition which is then referenced in the usual manner for named format definitions.
Backward Compatibility Requirement
DFDL Implementations that provide version 2.0 functionality must also provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends version 1.0.
Mixed Version Schemas
If a schema consists of multiple schema documents, and their respective dfdl:format annotations do not all contain the same value for the DFDLVersion attribute, then the schema is said to be "mixed version". This is allowed.
A named dfdl:format carrying DFDLVersion "2" may not be referenced from a DFDL Schema document having DFDLVersion "1".
The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or lacking the DFDLVersion attribute may be referenced from a DFDL Schema having DFDLVersion="2".
There are really 3 ways that DFDL version 2.0 can differ from version 1.0, and these are that new features are added, old features are deprecated, or old features are removed. This latter is quite undesirable, but will be considered a possibility for purpose of discussion this versioning system.
These will commonly be in combination such as if an old feature is deprecated with the replacement feature being added.
These will now be discussed separately, with the emphasis on how mixed version schemas must work.
New Features Added
Each new required DFDL property introduced for version 2.0 will be defined along with what is called its "version 1.0 default value". When a mixed version schema exists, the DFDL v1.0 parts of it behave as if any required DFDLVersion 2 properties have their version 1.0 default value defined, at top level of the schema (as if defined with that default value in each DFDLVersion 1 schema document's dfdl:format annotation).
Features Removed
Incompatibilities are to be avoided wherever possible, and may not be needed at all. However, there is still the possibility that some feature will want to be entirely removed and replaced by something better, and in that case the versioning scheme needs to be able to support this.
In that situation, the schema definition error for version 2 is only generated when a schema has DFDLVersion 2 specified (in dfdl:format explicitly, or by reference to a named format carrying or referencing DFDLVersion 2). Mixed schemas that contain schema documents entirely using version 1 of DFDL do not generate this schema definition error.
Features deprecated
Deprecation warnings are generated only when a DFDLVersion="2" attribute is present or is incorporated by reference (to a named format). A version 1 schema that is incorporated with a version 2 DFDL schema but is not modified in any way, does not generate deprecation warnings about version 1.0 features that have been deprecated in version 2.0.
(Most likely deprecations would be revising property names or value enums for clarity.)
Implementations may provide mechanisms for suppressing warnings. These are implementation dependent.
Version 1.0 Existing Deprecations and Compatibility
During the process of standardization for DFDL v1.0 a number of properties names were changed. DFDL version 1.0 implementations accept both the old and new property names. (Example: dfdl:separatorSuppressionPolicy, which replaced an older property name.) DFDL schemas that specify DFDLVersion="2" will not accept the older property names, and it is a schema definition error if the old (pre DFDL v1.0) property names are used.
Properly constructed mixed version schemas can still depend on the older property names in the version 1.0 parts of the schema.
------------------------------
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg* <https://www.ogf.org/mailman/listinfo/dfdl-wg>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg* <https://www.ogf.org/mailman/listinfo/dfdl-wg>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
OK so after discussion on WG call the proposal is: - New namespace "http://www.ogf.org/dfdl/dfdl-1.0/experimental" with default prefix "dfdlx" - Must be bound on the root of the schema only - Attribute form allowed - dfdlx:myNewProp - Element form allowed - <dfdl:property name="dfdlx:myNewProp">... - Short form allowed - dfdlx:myNewProp - Scoping rules apply if makes sense for property - A property is needed to enable any experimental feature (presence of the namespace is not sufficient) - An experience document will result from each experimental feature and will guide any inclusion in DFDL 2.0. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: Bradd Kadlecik <braddk@us.ibm.com>, DFDL-WG <dfdl-wg@ogf.org>, Michele Zundo <michele.zundo@esa.int> Date: 04/04/2019 20:58 Subject: Re: [DFDL-WG] Action 301: Versioning mechanism for DFDL v2.0 / experimental features I think we have to allow scoping rules to apply to new experimental properties if they are regular format properties. Some proposed extensions would be much too hard to use without scoping being available. But some new proposed extensions aren't the kind of thing one puts in scope anyway, so like existing non-scoped properties, new extension properties could be scoped or not-scoped depending on their nature. I think any new experimental property is going to also need a default value for it, for backward compatibility as you suggest. Since these are in namespace dfdlx, we don't have to worry about turning OFF this default. Once a property is accepted into the DFDL spec formally, then implementations probably should have switches for turning on/off defaults for it, as for new properties that have been added since the earliest implementation were already in existence. There are other things needed as extensions that aren't properties however. Ex: we've had requests for %LSP; (and +, * variants) character class entity. This is like %WSP; without the NLs., i.e., intra-line whitespace characters. I expect we want some property-like way to enable/disable use of such an extension feature. (With the default being to disallow.) ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Thu, Apr 4, 2019 at 10:59 AM Steve Hanson <smh@uk.ibm.com> wrote: These are just experimental properties, we don't have to roll out the full property syntax treatment, we just need to do the minimum to allow their use. I propose the following: - New namespace "http://www.ogf.org/dfdl/dfdl-1.0/experimental" with default prefix "dfdlx" - Must be bound on the root of the schema only - Attribute form not allowed - Element form not allowed - Short form allowed via dfdlx:myNewProp - Scoping rules do not apply There is a problem if we allow scoping. Scoping means that you can not be silent about the value of a property when it is needed by an object. But by definition we do not know the full set of experimental properties, they will change over time, so schemas using the experimental namespace would throw new SDEs out of nowhere if a processor started to support a new experimental feature. We could allow scoping if we changed the rules for experimental properties so they have a built-in default (we have been forced to do that when errata have added new properties to the core 1.0 spec - but there have only been one or two). If so the prefixed property would appear in attribute form on DFDL annotation elements, ie, dfdlx:myNewProp. If we really need to allow element form then that could be done via a new dfdlx:property annotation element. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org>, Bradd Kadlecik <braddk@us.ibm.com> Date: 01/02/2019 14:34 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes This seems reasonable to me. A few questions and suggestions. How are these properties expressed in long form annotations or property-element form annotations? I.e., <dfdl:format ..../> normally the attributes that short form have "dfdl:" would not have any namespace. Would the extension properties that show up inside dfdl:format carry their namespace there also? As in <dfdl:format tddt:pointerEnable="true"/> ? We also would need a way to make ultra long form properties work, e.g., <dfdl:element> <dfdl:property name="tddt:pointerEnable">true</dfdl:property> </dfdl:element> It's odd at first in any XSD-embedded language for a name attribute to take a QName as value. In this case I think it is proper. Really the name attribute isn't creating a name here, it's referencing a name. Assuming all the above notational issues are resolvable,..... Having a agreed upon URI for extension properties makes sense, because we can check if it is bound, and can validate short-form properties in that namespace. We need some way of looking at the schema at the top, and saying "this has extensions". There is a precedent for requiring a namespace binding to be at the root of a schema file, which is the feature where we allow include/import of non-DFDL XSD schemas that define other annotation languages. Such annotation languages can use attributes for example, though DFDL schemas cannot. When there is no binding of the dfdl namespace on the root of a schema, we don't include it for schema compilation purposes, though it is included for schema validation purposes. Given that precedent, we could require a binding of this extension URI at the root of the schema, rather than just allowing it anywhere. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Feb 1, 2019 at 4:34 AM Steve Hanson <smh@uk.ibm.com> wrote: Mike I'd like to make some progress on this if possible? The IBM z/TPF team are a heavy user of IBM DFDL. They are interested in adding offset support and also support for pointers (more on the latter on next WG call). A problem with using DFDLVersion="2" is that it is pre-judging what is going to be in 2.0. I would prefer that new features appeared in DFDL 1.0 schemas as 'experimental' with their DFDL properties in a separate namespace. They would switch to the DFDL namespace only in a 2.0 schema. This gives us flexibility around property naming and behaviour - we might decide to change things when formalising 2.0. For example this is what z/TPF have done for pointers: <xs:element name="myString" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="../len" tddt:indirectKind="pointer" tddt:indirectLength="4" /> It means that when a user migrates to DFDL 2.0 they would need to change their schemas but I think that is preferable to the current proposal where 2.0 bleeds into 1.0. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: dfdl-wg@ogf.org Date: 02/02/2018 22:07 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it. I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features. The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as http://www.xml.com/pub/a/2004/07/21/design.html These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself. The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <smh@uk.ibm.com> wrote: Mike, Your scheme as proposed has a couple of problems. a) It is possible to create a schema that does not use dfdl:format at all. Unlikely but possible. With such a schema there is no way to indicate version. b) What are the version implications for annotations that have nothing to do with dfdl:format? Asserts, discriminators, variables. Most of the language you use refers to versions of schemas. That's not the same as version of a dfdl:format. Any reason you didn't just change the DFDL namespace URL to be http://www.ogf.org/dfdl/dfdl-2.0/, which must be declared before any DFDL feature can be used? I think that was always the intent of the DFDL namespace. The key piece of work is validating any proposal against the scoping rules in section 8 of the spec. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: dfdl-wg@ogf.org Date: 23/01/2018 16:03 Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> There are some features now under discussion which will most likely not be part of the DFDL v1.0 standard, but a subsequent version 2.0 of the standard. Implementations may have these features prior to the existence of that draft version of the DFDL standard; hence, a mechanism for indicating the support, and requirements that implies, on DFDL schemas is needed in order for these v2.0 features to be introduced. The purpose of the versioning mechanism is to provide access to DFDL v2.0 features, while maintaining also backward compatibility with DFDL v1.0 schemas. For example, new DFDL format properties may be added to the DFDL v2.0 version; however, those properties are not required to be specified unless the version of the schema is specifically set to indicate it is using version 2.0 features. Also, deprecated features should only generate deprecation warnings if the user is requesting version 2.0 of DFDL, and should be silent otherwise. Proposal: New attribute on the dfdl:format annotation: DFDLVersion with valid values "1", "2". All values are reserved for future use. The DFDLVersion, if unspecified, defaults to "1" which means the implementation is of the DFDL v1.0 specification. If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL may be enforced on the schema, and schema definition errors will result if version 2.0 requirements are not met. DFDL Version 1.0 features that are deprecated will generate warnings. Example: <dfdl:format DFDLVersion="2" .../> Example: <dfdl:defineFormat name="myV2Format"> <dfdl:format DFDLVersion="2" .../> </dfdl:defineFormat> The DFDLVersion is not a format property and is not scoped. It can be added to a named format definition which is then referenced in the usual manner for named format definitions. Backward Compatibility Requirement DFDL Implementations that provide version 2.0 functionality must also provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends version 1.0. Mixed Version Schemas If a schema consists of multiple schema documents, and their respective dfdl:format annotations do not all contain the same value for the DFDLVersion attribute, then the schema is said to be "mixed version". This is allowed. A named dfdl:format carrying DFDLVersion "2" may not be referenced from a DFDL Schema document having DFDLVersion "1". The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or lacking the DFDLVersion attribute may be referenced from a DFDL Schema having DFDLVersion="2". There are really 3 ways that DFDL version 2.0 can differ from version 1.0, and these are that new features are added, old features are deprecated, or old features are removed. This latter is quite undesirable, but will be considered a possibility for purpose of discussion this versioning system. These will commonly be in combination such as if an old feature is deprecated with the replacement feature being added. These will now be discussed separately, with the emphasis on how mixed version schemas must work. New Features Added Each new required DFDL property introduced for version 2.0 will be defined along with what is called its "version 1.0 default value". When a mixed version schema exists, the DFDL v1.0 parts of it behave as if any required DFDLVersion 2 properties have their version 1.0 default value defined, at top level of the schema (as if defined with that default value in each DFDLVersion 1 schema document's dfdl:format annotation). Features Removed Incompatibilities are to be avoided wherever possible, and may not be needed at all. However, there is still the possibility that some feature will want to be entirely removed and replaced by something better, and in that case the versioning scheme needs to be able to support this. In that situation, the schema definition error for version 2 is only generated when a schema has DFDLVersion 2 specified (in dfdl:format explicitly, or by reference to a named format carrying or referencing DFDLVersion 2). Mixed schemas that contain schema documents entirely using version 1 of DFDL do not generate this schema definition error. Features deprecated Deprecation warnings are generated only when a DFDLVersion="2" attribute is present or is incorporated by reference (to a named format). A version 1 schema that is incorporated with a version 2 DFDL schema but is not modified in any way, does not generate deprecation warnings about version 1.0 features that have been deprecated in version 2.0. (Most likely deprecations would be revising property names or value enums for clarity.) Implementations may provide mechanisms for suppressing warnings. These are implementation dependent. Version 1.0 Existing Deprecations and Compatibility During the process of standardization for DFDL v1.0 a number of properties names were changed. DFDL version 1.0 implementations accept both the old and new property names. (Example: dfdl:separatorSuppressionPolicy, which replaced an older property name.) DFDL schemas that specify DFDLVersion="2" will not accept the older property names, and it is a schema definition error if the old (pre DFDL v1.0) property names are used. Properly constructed mixed version schemas can still depend on the older property names in the version 1.0 parts of the schema. ------------------------------ Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
I like this proposal. I propose that we include it in DFDL 1.0 (hopefully the last thing we add!) We'll need to create an erratum for it. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Fri, Apr 5, 2019 at 12:42 PM Steve Hanson <smh@uk.ibm.com> wrote:
OK so after discussion on WG call the proposal is:
- New namespace "*http://www.ogf.org/dfdl/dfdl-1.0/experimental* <http://www.ogf.org/dfdl/dfdl-1.0/experimental>" with default prefix "dfdlx" - Must be bound on the root of the schema only - Attribute form allowed - dfdlx:myNewProp - Element form allowed - <dfdl:property name="dfdlx:myNewProp">... - Short form allowed - dfdlx:myNewProp - Scoping rules apply if makes sense for property - A property is needed to enable any experimental feature (presence of the namespace is not sufficient) - An experience document will result from each experimental feature and will guide any inclusion in DFDL 2.0.
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: Bradd Kadlecik <braddk@us.ibm.com>, DFDL-WG <dfdl-wg@ogf.org>, Michele Zundo <michele.zundo@esa.int> Date: 04/04/2019 20:58 Subject: Re: [DFDL-WG] Action 301: Versioning mechanism for DFDL v2.0 / experimental features ------------------------------
I think we have to allow scoping rules to apply to new experimental properties if they are regular format properties. Some proposed extensions would be much too hard to use without scoping being available.
But some new proposed extensions aren't the kind of thing one puts in scope anyway, so like existing non-scoped properties, new extension properties could be scoped or not-scoped depending on their nature.
I think any new experimental property is going to also need a default value for it, for backward compatibility as you suggest.
Since these are in namespace dfdlx, we don't have to worry about turning OFF this default. Once a property is accepted into the DFDL spec formally, then implementations probably should have switches for turning on/off defaults for it, as for new properties that have been added since the earliest implementation were already in existence.
There are other things needed as extensions that aren't properties however. Ex: we've had requests for %LSP; (and +, * variants) character class entity. This is like %WSP; without the NLs., i.e., intra-line whitespace characters. I expect we want some property-like way to enable/disable use of such an extension feature. (With the default being to disallow.)
...mikeb
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Thu, Apr 4, 2019 at 10:59 AM Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: These are just experimental properties, we don't have to roll out the full property syntax treatment, we just need to do the minimum to allow their use. I propose the following:
- New namespace "*http://www.ogf.org/dfdl/dfdl-1.0/experimental* <http://www.ogf.org/dfdl/dfdl-1.0/experimental>" with default prefix "dfdlx" - Must be bound on the root of the schema only - Attribute form not allowed - Element form not allowed - Short form allowed via dfdlx:myNewProp - Scoping rules do not apply
There is a problem if we allow scoping. Scoping means that you can not be silent about the value of a property when it is needed by an object. But by definition we do not know the full set of experimental properties, they will change over time, so schemas using the experimental namespace would throw new SDEs out of nowhere if a processor started to support a new experimental feature.
We could allow scoping if we changed the rules for experimental properties so they have a built-in default (we have been forced to do that when errata have added new properties to the core 1.0 spec - but there have only been one or two). If so the prefixed property would appear in attribute form on DFDL annotation elements, ie, dfdlx:myNewProp.
If we really need to allow element form then that could be done via a new dfdlx:property annotation element.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> Cc: DFDL-WG <*dfdl-wg@ogf.org* <dfdl-wg@ogf.org>>, Bradd Kadlecik < *braddk@us.ibm.com* <braddk@us.ibm.com>> Date: 01/02/2019 14:34 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes ------------------------------
This seems reasonable to me.
A few questions and suggestions.
How are these properties expressed in long form annotations or property-element form annotations?
I.e., <dfdl:format ..../> normally the attributes that short form have "dfdl:" would not have any namespace. Would the extension properties that show up inside dfdl:format carry their namespace there also? As in <dfdl:format tddt:pointerEnable="true"/> ?
We also would need a way to make ultra long form properties work, e.g.,
<dfdl:element> <dfdl:property name="tddt:pointerEnable">true</dfdl:property> </dfdl:element>
It's odd at first in any XSD-embedded language for a name attribute to take a QName as value. In this case I think it is proper. Really the name attribute isn't creating a name here, it's referencing a name.
Assuming all the above notational issues are resolvable,.....
Having a agreed upon URI for extension properties makes sense, because we can check if it is bound, and can validate short-form properties in that namespace.
We need some way of looking at the schema at the top, and saying "this has extensions".
There is a precedent for requiring a namespace binding to be at the root of a schema file, which is the feature where we allow include/import of non-DFDL XSD schemas that define other annotation languages. Such annotation languages can use attributes for example, though DFDL schemas cannot. When there is no binding of the dfdl namespace on the root of a schema, we don't include it for schema compilation purposes, though it is included for schema validation purposes.
Given that precedent, we could require a binding of this extension URI at the root of the schema, rather than just allowing it anywhere.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Fri, Feb 1, 2019 at 4:34 AM Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: Mike
I'd like to make some progress on this if possible? The IBM z/TPF team are a heavy user of IBM DFDL. They are interested in adding offset support and also support for pointers (more on the latter on next WG call).
A problem with using DFDLVersion="2" is that it is pre-judging what is going to be in 2.0. I would prefer that new features appeared in DFDL 1.0 schemas as 'experimental' with their DFDL properties in a separate namespace. They would switch to the DFDL namespace only in a 2.0 schema. This gives us flexibility around property naming and behaviour - we might decide to change things when formalising 2.0.
For example this is what z/TPF have done for pointers:
<xs:element name="myString" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="../len" tddt:indirectKind="pointer" tddt:indirectLength="4" />
It means that when a user migrates to DFDL 2.0 they would need to change their schemas but I think that is preferable to the current proposal where 2.0 bleeds into 1.0.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> Date: 02/02/2018 22:07 Subject: Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it.
I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features.
The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as *http://www.xml.com/pub/a/2004/07/21/design.html * <http://www.xml.com/pub/a/2004/07/21/design.html>
These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself.
The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: Mike,
Your scheme as proposed has a couple of problems. a) It is possible to create a schema that does not use dfdl:format at all. Unlikely but possible. With such a schema there is no way to indicate version. b) What are the version implications for annotations that have nothing to do with dfdl:format? Asserts, discriminators, variables.
Most of the language you use refers to versions of *schemas*. That's not the same as version of a dfdl:format. Any reason you didn't just change the DFDL namespace URL to be *http://www.ogf.org/dfdl/dfdl-2.0/* <http://www.ogf.org/dfdl/dfdl-2.0/>, which must be declared before any DFDL feature can be used? I think that was always the intent of the DFDL namespace.
The key piece of work is validating any proposal against the scoping rules in section 8 of the spec.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:*+44-1962-815848* <+44%201962%20815848> mob:*+44-7717-378890* <+44%207717%20378890> Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> Date: 23/01/2018 16:03 Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
There are some features now under discussion which will most likely not be part of the DFDL v1.0 standard, but a subsequent version 2.0 of the standard.
Implementations may have these features prior to the existence of that draft version of the DFDL standard; hence, a mechanism for indicating the support, and requirements that implies, on DFDL schemas is needed in order for these v2.0 features to be introduced.
The purpose of the versioning mechanism is to provide access to DFDL v2.0 features, while maintaining also backward compatibility with DFDL v1.0 schemas.
For example, new DFDL format properties may be added to the DFDL v2.0 version; however, those properties are not required to be specified unless the version of the schema is specifically set to indicate it is using version 2.0 features. Also, deprecated features should only generate deprecation warnings if the user is requesting version 2.0 of DFDL, and should be silent otherwise.
Proposal:
New attribute on the dfdl:format annotation: DFDLVersion with valid values "1", "2". All values are reserved for future use.
The DFDLVersion, if unspecified, defaults to "1" which means the implementation is of the DFDL v1.0 specification.
If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL may be enforced on the schema, and schema definition errors will result if version 2.0 requirements are not met. DFDL Version 1.0 features that are deprecated will generate warnings.
Example:
<dfdl:format DFDLVersion="2" .../>
Example:
<dfdl:defineFormat name="myV2Format"> <dfdl:format DFDLVersion="2" .../> </dfdl:defineFormat>
The DFDLVersion is not a format property and is not scoped. It can be added to a named format definition which is then referenced in the usual manner for named format definitions.
Backward Compatibility Requirement
DFDL Implementations that provide version 2.0 functionality must also provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends version 1.0.
Mixed Version Schemas
If a schema consists of multiple schema documents, and their respective dfdl:format annotations do not all contain the same value for the DFDLVersion attribute, then the schema is said to be "mixed version". This is allowed.
A named dfdl:format carrying DFDLVersion "2" may not be referenced from a DFDL Schema document having DFDLVersion "1".
The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or lacking the DFDLVersion attribute may be referenced from a DFDL Schema having DFDLVersion="2".
There are really 3 ways that DFDL version 2.0 can differ from version 1.0, and these are that new features are added, old features are deprecated, or old features are removed. This latter is quite undesirable, but will be considered a possibility for purpose of discussion this versioning system.
These will commonly be in combination such as if an old feature is deprecated with the replacement feature being added.
These will now be discussed separately, with the emphasis on how mixed version schemas must work.
New Features Added
Each new required DFDL property introduced for version 2.0 will be defined along with what is called its "version 1.0 default value". When a mixed version schema exists, the DFDL v1.0 parts of it behave as if any required DFDLVersion 2 properties have their version 1.0 default value defined, at top level of the schema (as if defined with that default value in each DFDLVersion 1 schema document's dfdl:format annotation).
Features Removed
Incompatibilities are to be avoided wherever possible, and may not be needed at all. However, there is still the possibility that some feature will want to be entirely removed and replaced by something better, and in that case the versioning scheme needs to be able to support this.
In that situation, the schema definition error for version 2 is only generated when a schema has DFDLVersion 2 specified (in dfdl:format explicitly, or by reference to a named format carrying or referencing DFDLVersion 2). Mixed schemas that contain schema documents entirely using version 1 of DFDL do not generate this schema definition error.
Features deprecated
Deprecation warnings are generated only when a DFDLVersion="2" attribute is present or is incorporated by reference (to a named format). A version 1 schema that is incorporated with a version 2 DFDL schema but is not modified in any way, does not generate deprecation warnings about version 1.0 features that have been deprecated in version 2.0.
(Most likely deprecations would be revising property names or value enums for clarity.)
Implementations may provide mechanisms for suppressing warnings. These are implementation dependent.
Version 1.0 Existing Deprecations and Compatibility
During the process of standardization for DFDL v1.0 a number of properties names were changed. DFDL version 1.0 implementations accept both the old and new property names. (Example: dfdl:separatorSuppressionPolicy, which replaced an older property name.) DFDL schemas that specify DFDLVersion="2" will not accept the older property names, and it is a schema definition error if the old (pre DFDL v1.0) property names are used.
Properly constructed mixed version schemas can still depend on the older property names in the version 1.0 parts of the schema.
------------------------------
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg* <https://www.ogf.org/mailman/listinfo/dfdl-wg>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg* <https://www.ogf.org/mailman/listinfo/dfdl-wg>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
participants (2)
-
Mike Beckerle -
Steve Hanson