- dfdl-wg - lists.ogf.org

DFDL4S C++ 1.0 released
by Michele Zundo 07 Mar '18

07 Mar '18

Dear all, we have just publicly released the C++ version of our DFDL4S Java library to read/write binary data using DFDL schema. It implements only a subset of DFDL standard but enough for our use (read/write of CCSDS formatted data). Details and download from https://eop-cfi.esa.int/ <https://eop-cfi.esa.int/>. ESA - European Space Agency Michele Zundo Head of Ground System Definition and Verification Office System Support Division Earth Observation Project Department Directorate of Earth Observation Programmes ESTEC Kepleerlaan 1, 2200AZ Noordwijk, Netherlands michele.zundo(a)esa.int <mailto:michele.zundo@esa.int> | www.esa.int <file:///x-msg/::11:www.esa.int> T. +31 71 565 3924 | F +31 71 565 3171

3 2

Confirmed: OGF DFDL Working Group Call (6 Mar 15:00 GMT)
by Steve Hanson 06 Mar '18

06 Mar '18

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for next OGF DFDL WG Call
by Steve Hanson 06 Mar '18

06 Mar '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13573?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Fwd: Clarification: attributes in DFDL schema that are used not for data, but for other non-DFDL annotation elements.
by Mike Beckerle 05 Mar '18

05 Mar '18

(sent 2nd time due to mail problem.) I like your suggested solution. I'd modify point (a) to say - examine each imported schema. If the schema does not have any xmlns namespace binding for the DFDL namespace URI, then it is explicitly not a DFDL schema, and so would not be treated as one. (E.g., what if somebody really wants to use say "dfd" as a prefix, and so they want to bind the dfdl namespace to "f:" instead of the usual "dfdl:".) The import is effectively skipped by the DFDL processor, though if the DFDL schema was being validated separately (by say, Xerces), then the import would be processed and so the validation of the added annotation language would occur. Rather than points (c) and (d) I agree that requiring such schemas to be imported, but not included works well. References to the namespace would fail (since the DFDL processor would be skipping the import.) The only "hardship" if you can even call it that, is if you wanted to have a DFDL schema that just contains say, reusable type definitions, but doesn't actually have any DFDL annotations in it, then you would have to artificially add a dfdl namespace prefix definition to it. That seems very reasonable to me. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Fri, Feb 23, 2018 at 12:58 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike > > Had to pull in the DFDLGeneralFormat.dfdl.xsd to get everything resolved, > but when I did, this is what IBM DFDL gave: > > *CTDS1086E : Included or imported schema not a DFDL schema. > 'platform:/resource/Daffodil tests/otherAnnotationLanguage.xsd'.* > > I think this is an IBM specific check - I could not find this restriction > in a quick review of the DFDL 1.0 spec. I believe it was done because the > IBM DFDL validator can't assume how schemas are to be used, and so > validates each xsd file in a library independently - which means each one > must validate cleanly on its own. We don't use '.dfdl' in the name so can't > tell the intent from the name. > > Adding the DFDL namespace to the target schema removes that error but as > you might expect then gives: > > *CTDS1000E : Attribute declarations (local and global) are not allowed in > DFDL schemas.* > *CTDS1000E : Attribute declarations (local and global) are not allowed in > DFDL schemas.* > *CTDS1001E : Attribute references are not allowed in DFDL schemas.* > *CTDS1008E : Complex types with empty content are not allowed in DFDL > schemas. Complex type 'urn:otherAnnotationLanguage#otherAnnotation_._type'.* > > Commenting out the xs:include removes all errors. In fact we rely on this > behaviour - when IBM DFDL editor imports COBOL, the COBOL compiler adds its > own set of annotations. > > I think the way to allow your scenario would be to: > > a) insist that the target schema did *not* contain an xmlns:dfdl > declaration, so is explicitly *not* a DFDL schema > b) do not validate such a schema independently > c) when processing a DFDL schema that uses it, keep a list of all global > objects that the target schema offers > c) issue a SDE if an object in the list was referenced by an object in the > DFDL schema (outside of an annotation, obviously) > > If you insisted that the target schema *must* be in a different namespace > to any of the DFDL schemas in the set, you could do the check on namespace > and not need to keep a list of objects. Any reference to the namespace by > an object (outside of an annotation) is immediately a SDE. So allow such > schemas to be imported but do not allow them to be included. > > Thoughts? > > Regards > > Steve Hanson > > IBM Hybrid Integration, Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 <+44%201962%20815848> > mob:+44-7717-378890 <+44%207717%20378890> > Note: I work Tuesday to Friday > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Steve Hanson <smh(a)uk.ibm.com> > Cc: dfdl-wg(a)ogf.org > Date: 20/02/2018 23:18 > Subject: Re: [DFDL-WG] Clarification: attributes in DFDL schema > that are used not for data, but for other non-DFDL annotation elements. > ------------------------------ > > > > Yes, files are attached. > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > *www.tresys.com* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ…> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the *OGF Intellectual Property Policy* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5…> > > > On Tue, Feb 20, 2018 at 2:20 PM, Steve Hanson <*smh(a)uk.ibm.com* > <smh(a)uk.ibm.com>> wrote: > Mike, is it possible to post a cut-down example that shows enough to give > the error in Daffodil, and I'll see if IBM DFDL gives a similar error. > > Regards > > Steve Hanson > IBM Hybrid Integration, Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_dfdl_&d=DwM…> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:*+44-1962-815848* <+44%201962%20815848> > mob:*+44-7717-378890* <+44%207717%20378890> > Note: I work Tuesday to Friday > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com* > <mbeckerle.dfdl(a)gmail.com>> > To: *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> > Date: 20/02/2018 16:44 > Subject: [DFDL-WG] Clarification: attributes in DFDL schema that > are used not for data, but for other non-DFDL annotation elements. > Sent by: "dfdl-wg" <*dfdl-wg-bounces(a)ogf.org* > <dfdl-wg-bounces(a)ogf.org>> > ------------------------------ > > > > > Interesting issue. > > DFDL doesn't allow use of attributes. > > But we have a user that has a DFDL schema to which they also want to add > other non-DFDL annotations. > > Like DFDL itself, this other annotation language has a variety of > annotation elements that appear inside xs:appinfo blocks with specific > source attribute specified. A DFDL processor should be ignoring these. > > Those other annotation elements have attributes (as do DFDL's annotation > elements.) > > But Daffodil issues a schema definition error on the attribute > declarations of this other annotation-language schema. This because it has > no way of telling if these attribute declarations are for use *only* in > annotations that it will ignore, or if they are used for modeling data. > > Question is: Is it acceptable for a DFDL annotation to just say "Nowhere > in the extended schema - all files - can there be any attribute > declarations.", or do we need to accommodate attributes and in fact every > XSD construct that DFDL disallows, so long as they're not being used to > model data? > > Really there's two questions here. First, does the DFDL specification take > (or need to take) a position on this. Second, what's a practical choice for > a DFDL implementation like Daffodil to take here. > > ...mikeb > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > *www.tresys.com* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ…> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the *OGF Intellectual Property Policy* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5…> > -- > dfdl-wg mailing list > *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> > > *https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=sc82YpTJe2zaMkOmRGSHfOxkxJcq0n_c9FjQ8aQefxE&s=JkCAzi0EGJmPp-0RyMaiKEjpR4gkvCzyc60Q-qrVoaw&e=* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li…> > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >

1 1

Mail service test message 2 Mar 2018 - please ignore
by Sill, Alan 02 Mar '18

02 Mar '18

Test mail service after restarting due to certificate renewal. Please ignore (but if you got this, it means the service is working).

1 0

Mail service test message - please ignore
by Sill, Alan 02 Mar '18

02 Mar '18

Test mail service after restarting due to certificate renewal. Please ignore (but if you got this, it means the service is working).

1 0

Clarification: attributes in DFDL schema that are used not for data, but for other non-DFDL annotation elements.
by Mike Beckerle 20 Feb '18

20 Feb '18

Interesting issue. DFDL doesn't allow use of attributes. But we have a user that has a DFDL schema to which they also want to add other non-DFDL annotations. Like DFDL itself, this other annotation language has a variety of annotation elements that appear inside xs:appinfo blocks with specific source attribute specified. A DFDL processor should be ignoring these. Those other annotation elements have attributes (as do DFDL's annotation elements.) But Daffodil issues a schema definition error on the attribute declarations of this other annotation-language schema. This because it has no way of telling if these attribute declarations are for use *only* in annotations that it will ignore, or if they are used for modeling data. Question is: Is it acceptable for a DFDL annotation to just say "Nowhere in the extended schema - all files - can there be any attribute declarations.", or do we need to accommodate attributes and in fact every XSD construct that DFDL disallows, so long as they're not being used to model data? Really there's two questions here. First, does the DFDL specification take (or need to take) a position on this. Second, what's a practical choice for a DFDL implementation like Daffodil to take here. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

OGF DFDL WG Call today 2018-02-20 - cancelled
by Steve Hanson 20 Feb '18

20 Feb '18

Apologies for late notice - a meeting conflict means I can't make the call today Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 20 Feb '18

20 Feb '18

1 0

Review/Feedback wanted on proposal: Data Streaming for base64 and other Layered Transformations
by Mike Beckerle 15 Feb '18

15 Feb '18

We have a great deal of demand for the ability to describe data formats that include layered transformations - that is, regions within the data that need to be algorithmically transformed before parsing and after unparsing. The IETF data formats make extensive use of line folding, base64, etc. Many formats allow compressed payloads. All these are examples of "layering" where a region of the data stream can be identified, algorithmically transformed, and then subsequently further parsed. (The inverse process for unparsing obviously.) Our draft proposal, that we are hoping to implement in Daffodil, is here: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Strea… Any feedback on this is very welcome, and can be addressed to dfdl-wg mailing list. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0