Fwd: Clarification: attributes in DFDL schema that are used not for data, but for other non-DFDL annotation elements.
(sent 2nd time due to mail problem.) I like your suggested solution. I'd modify point (a) to say - examine each imported schema. If the schema does not have any xmlns namespace binding for the DFDL namespace URI, then it is explicitly not a DFDL schema, and so would not be treated as one. (E.g., what if somebody really wants to use say "dfd" as a prefix, and so they want to bind the dfdl namespace to "f:" instead of the usual "dfdl:".) The import is effectively skipped by the DFDL processor, though if the DFDL schema was being validated separately (by say, Xerces), then the import would be processed and so the validation of the added annotation language would occur. Rather than points (c) and (d) I agree that requiring such schemas to be imported, but not included works well. References to the namespace would fail (since the DFDL processor would be skipping the import.) The only "hardship" if you can even call it that, is if you wanted to have a DFDL schema that just contains say, reusable type definitions, but doesn't actually have any DFDL annotations in it, then you would have to artificially add a dfdl namespace prefix definition to it. That seems very reasonable to me. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Fri, Feb 23, 2018 at 12:58 PM, Steve Hanson <smh@uk.ibm.com> wrote:
Mike
Had to pull in the DFDLGeneralFormat.dfdl.xsd to get everything resolved, but when I did, this is what IBM DFDL gave:
*CTDS1086E : Included or imported schema not a DFDL schema. 'platform:/resource/Daffodil tests/otherAnnotationLanguage.xsd'.*
I think this is an IBM specific check - I could not find this restriction in a quick review of the DFDL 1.0 spec. I believe it was done because the IBM DFDL validator can't assume how schemas are to be used, and so validates each xsd file in a library independently - which means each one must validate cleanly on its own. We don't use '.dfdl' in the name so can't tell the intent from the name.
Adding the DFDL namespace to the target schema removes that error but as you might expect then gives:
*CTDS1000E : Attribute declarations (local and global) are not allowed in DFDL schemas.* *CTDS1000E : Attribute declarations (local and global) are not allowed in DFDL schemas.* *CTDS1001E : Attribute references are not allowed in DFDL schemas.* *CTDS1008E : Complex types with empty content are not allowed in DFDL schemas. Complex type 'urn:otherAnnotationLanguage#otherAnnotation_._type'.*
Commenting out the xs:include removes all errors. In fact we rely on this behaviour - when IBM DFDL editor imports COBOL, the COBOL compiler adds its own set of annotations.
I think the way to allow your scenario would be to:
a) insist that the target schema did *not* contain an xmlns:dfdl declaration, so is explicitly *not* a DFDL schema b) do not validate such a schema independently c) when processing a DFDL schema that uses it, keep a list of all global objects that the target schema offers c) issue a SDE if an object in the list was referenced by an object in the DFDL schema (outside of an annotation, obviously)
If you insisted that the target schema *must* be in a different namespace to any of the DFDL schemas in the set, you could do the check on namespace and not need to keep a list of objects. Any reference to the namespace by an object (outside of an annotation) is immediately a SDE. So allow such schemas to be imported but do not allow them to be included.
Thoughts?
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 <+44%201962%20815848> mob:+44-7717-378890 <+44%207717%20378890> Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: dfdl-wg@ogf.org Date: 20/02/2018 23:18 Subject: Re: [DFDL-WG] Clarification: attributes in DFDL schema that are used not for data, but for other non-DFDL annotation elements. ------------------------------
Yes, files are attached.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=rJgPyinFaSd9n2H_ZqeiRRId3AD2ehRelanacvNIQYc&s=osdwwVMBgbE2pMiuU7vHGdPGAkprML1I0AxGtqVyrLA&e=> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5Fpolicies.php&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=rJgPyinFaSd9n2H_ZqeiRRId3AD2ehRelanacvNIQYc&s=aOae6-HU7VDdWzcvwekuay2VfJ7tCwybDhy_QSL3Gmc&e=>
On Tue, Feb 20, 2018 at 2:20 PM, Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: Mike, is it possible to post a cut-down example that shows enough to give the error in Daffodil, and I'll see if IBM DFDL gives a similar error.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_dfdl_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=rJgPyinFaSd9n2H_ZqeiRRId3AD2ehRelanacvNIQYc&s=C_3TxayHMwdjplAn6GOpYTYvEvJ9vy1x7YVviVT0HGU&e=> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:*+44-1962-815848* <+44%201962%20815848> mob:*+44-7717-378890* <+44%207717%20378890> Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> Date: 20/02/2018 16:44 Subject: [DFDL-WG] Clarification: attributes in DFDL schema that are used not for data, but for other non-DFDL annotation elements. Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
Interesting issue.
DFDL doesn't allow use of attributes.
But we have a user that has a DFDL schema to which they also want to add other non-DFDL annotations.
Like DFDL itself, this other annotation language has a variety of annotation elements that appear inside xs:appinfo blocks with specific source attribute specified. A DFDL processor should be ignoring these.
Those other annotation elements have attributes (as do DFDL's annotation elements.)
But Daffodil issues a schema definition error on the attribute declarations of this other annotation-language schema. This because it has no way of telling if these attribute declarations are for use *only* in annotations that it will ignore, or if they are used for modeling data.
Question is: Is it acceptable for a DFDL annotation to just say "Nowhere in the extended schema - all files - can there be any attribute declarations.", or do we need to accommodate attributes and in fact every XSD construct that DFDL disallows, so long as they're not being used to model data?
Really there's two questions here. First, does the DFDL specification take (or need to take) a position on this. Second, what's a practical choice for a DFDL implementation like Daffodil to take here.
...mikeb
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=Rxsta0Lr-rXWM16oYenXGrP7xH4Ti_yQTGKV06lYxp4&e=> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5Fpolicies.php&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=uNKBF8PdzGH9PItZ16o0PqxPLYFR4naem91zZaseC1w&e=> -- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org>
*https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=JkCAzi0EGJmPp-0RyMaiKEjpR4gkvCzyc60Q-qrVoaw&e=* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=JkCAzi0EGJmPp-0RyMaiKEjpR4gkvCzyc60Q-qrVoaw&e=>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Of note. We've implemented this in Daffodil. So far it seems to work well. A surprising number of the schemas we test did in fact have files without any namespace declaration for DFDL. In one large schema, there is a master schema file that includes all the (over 100) others, and that master schema had no dfdl namespace definition as it didn't need one. This is a likely case that will be seen in a number of other large schemas. We issue a schema-definition warning whenever we ignore a schema file, so that coupled with the error that inevitably one gets if an important schema file is omitted make this easy to diagnose and fix. ...mike beckerle Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Fri, Mar 2, 2018 at 2:40 PM, Mike Beckerle <mbeckerle.dfdl@gmail.com> wrote:
(sent 2nd time due to mail problem.)
I like your suggested solution.
I'd modify point (a) to say - examine each imported schema. If the schema does not have any xmlns namespace binding for the DFDL namespace URI, then it is explicitly not a DFDL schema, and so would not be treated as one. (E.g., what if somebody really wants to use say "dfd" as a prefix, and so they want to bind the dfdl namespace to "f:" instead of the usual "dfdl:".)
The import is effectively skipped by the DFDL processor, though if the DFDL schema was being validated separately (by say, Xerces), then the import would be processed and so the validation of the added annotation language would occur.
Rather than points (c) and (d) I agree that requiring such schemas to be imported, but not included works well. References to the namespace would fail (since the DFDL processor would be skipping the import.)
The only "hardship" if you can even call it that, is if you wanted to have a DFDL schema that just contains say, reusable type definitions, but doesn't actually have any DFDL annotations in it, then you would have to artificially add a dfdl namespace prefix definition to it. That seems very reasonable to me.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>
On Fri, Feb 23, 2018 at 12:58 PM, Steve Hanson <smh@uk.ibm.com> wrote:
Mike
Had to pull in the DFDLGeneralFormat.dfdl.xsd to get everything resolved, but when I did, this is what IBM DFDL gave:
*CTDS1086E : Included or imported schema not a DFDL schema. 'platform:/resource/Daffodil tests/otherAnnotationLanguage.xsd'.*
I think this is an IBM specific check - I could not find this restriction in a quick review of the DFDL 1.0 spec. I believe it was done because the IBM DFDL validator can't assume how schemas are to be used, and so validates each xsd file in a library independently - which means each one must validate cleanly on its own. We don't use '.dfdl' in the name so can't tell the intent from the name.
Adding the DFDL namespace to the target schema removes that error but as you might expect then gives:
*CTDS1000E : Attribute declarations (local and global) are not allowed in DFDL schemas.* *CTDS1000E : Attribute declarations (local and global) are not allowed in DFDL schemas.* *CTDS1001E : Attribute references are not allowed in DFDL schemas.* *CTDS1008E : Complex types with empty content are not allowed in DFDL schemas. Complex type 'urn:otherAnnotationLanguage#otherAnnotation_._type'.*
Commenting out the xs:include removes all errors. In fact we rely on this behaviour - when IBM DFDL editor imports COBOL, the COBOL compiler adds its own set of annotations.
I think the way to allow your scenario would be to:
a) insist that the target schema did *not* contain an xmlns:dfdl declaration, so is explicitly *not* a DFDL schema b) do not validate such a schema independently c) when processing a DFDL schema that uses it, keep a list of all global objects that the target schema offers c) issue a SDE if an object in the list was referenced by an object in the DFDL schema (outside of an annotation, obviously)
If you insisted that the target schema *must* be in a different namespace to any of the DFDL schemas in the set, you could do the check on namespace and not need to keep a list of objects. Any reference to the namespace by an object (outside of an annotation) is immediately a SDE. So allow such schemas to be imported but do not allow them to be included.
Thoughts?
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 <+44%201962%20815848> mob:+44-7717-378890 <+44%207717%20378890> Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: dfdl-wg@ogf.org Date: 20/02/2018 23:18 Subject: Re: [DFDL-WG] Clarification: attributes in DFDL schema that are used not for data, but for other non-DFDL annotation elements. ------------------------------
Yes, files are attached.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=rJgPyinFaSd9n2H_ZqeiRRId3AD2ehRelanacvNIQYc&s=osdwwVMBgbE2pMiuU7vHGdPGAkprML1I0AxGtqVyrLA&e=> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5Fpolicies.php&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=rJgPyinFaSd9n2H_ZqeiRRId3AD2ehRelanacvNIQYc&s=aOae6-HU7VDdWzcvwekuay2VfJ7tCwybDhy_QSL3Gmc&e=>
On Tue, Feb 20, 2018 at 2:20 PM, Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: Mike, is it possible to post a cut-down example that shows enough to give the error in Daffodil, and I'll see if IBM DFDL gives a similar error.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_dfdl_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=rJgPyinFaSd9n2H_ZqeiRRId3AD2ehRelanacvNIQYc&s=C_3TxayHMwdjplAn6GOpYTYvEvJ9vy1x7YVviVT0HGU&e=> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:*+44-1962-815848* <+44%201962%20815848> mob:*+44-7717-378890* <+44%207717%20378890> Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> Date: 20/02/2018 16:44 Subject: [DFDL-WG] Clarification: attributes in DFDL schema that are used not for data, but for other non-DFDL annotation elements. Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
Interesting issue.
DFDL doesn't allow use of attributes.
But we have a user that has a DFDL schema to which they also want to add other non-DFDL annotations.
Like DFDL itself, this other annotation language has a variety of annotation elements that appear inside xs:appinfo blocks with specific source attribute specified. A DFDL processor should be ignoring these.
Those other annotation elements have attributes (as do DFDL's annotation elements.)
But Daffodil issues a schema definition error on the attribute declarations of this other annotation-language schema. This because it has no way of telling if these attribute declarations are for use *only* in annotations that it will ignore, or if they are used for modeling data.
Question is: Is it acceptable for a DFDL annotation to just say "Nowhere in the extended schema - all files - can there be any attribute declarations.", or do we need to accommodate attributes and in fact every XSD construct that DFDL disallows, so long as they're not being used to model data?
Really there's two questions here. First, does the DFDL specification take (or need to take) a position on this. Second, what's a practical choice for a DFDL implementation like Daffodil to take here.
...mikeb
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=Rxsta0Lr-rXWM16oYenXGrP7xH4Ti_yQTGKV06lYxp4&e=> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5Fpolicies.php&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=uNKBF8PdzGH9PItZ16o0PqxPLYFR4naem91zZaseC1w&e=> -- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org>
*https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=JkCAzi0EGJmPp-0RyMaiKEjpR4gkvCzyc60Q-qrVoaw&e=* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=JkCAzi0EGJmPp-0RyMaiKEjpR4gkvCzyc60Q-qrVoaw&e=>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
participants (1)
-
Mike Beckerle