March 2009 - dfdl-wg - lists.ogf.org

Cancelled: OGF DFDL Working Group weekly call
by Steve Hanson 24 Mar '09

24 Mar '09

Mike & Alan unavailable

1 0

UPDATED DFDL: Minutes from OGF DFDL Working Group Call, March-18-2009 -
by Alan Powell 20 Mar '09

20 Mar '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, March-18-2009 Attendees Alan Powell (IBM) Steve Hanson (IBM) Dave Glick (drac) Suman Kalia (IBM) Apologies Mike Beckerle(Oco) <Updated> to correct dfdl:format syntax to set common properties Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, March-18-2009 Attendees Alan Powell (IBM) Steve Hanson (IBM) Dave Glick (drac) Suman Kalia (IBM) Apologies Mike Beckerle(Oco) 1. Specialized Annotations The whole call was taken up discussing the "dfdl:sequence and other specialized annotations" proposal emailed by SH. In summary proposes - the use of specialized annotations on each schema object. eg dfdl:sequence - a separate set of scoped properties for each specialization. <xs:complexType name="myType"> <xs:annotation><xs:appinfo source=?http://www.ogf.org/dfdl/?> <dfdl:complexType> <dfdl:sequence lengthKind="implicit" ... /> <dfdl:element lengthKind="delimited" ... /> </dfdl:complexType> </xs:appinfo></xs:annotation> There was general agreement with the proposal but a lot of discussion about the two issues that SH identified. a. The meaning of short form annotations on complex type. <xs:complexType name="myType" dfdl:lengthKind="implicit" ... > It is not clear which specialized annotation scope the short form properties on complex type should apply to. There was a lot of discussion between 1) disallowing short form annotation on complex type and 2) applying them to all specialized scopes. Disallowing short form is simple but makes complex type different, forces long form which is much more verbose, means common properties, eg encoding, to be specified the same on multiple specialized annotations. This is a particular problem for element and simple type. However applying short form to all specialization introduces a new level of inheritance and consequent complexity. b. You are forced to set common properties in multiple places in defineFormat. <xs:annotation><xs:appinfo source=?http://www.ogf.org/dfdl/?> <dfdl:defineFormat> <dfdl:sequence encoding="UTF-8" ... /> <dfdl:element encoding="UTF-8" ... /> <dfdl:simpleType encoding="UTF-8" ... /> <dfdl:choice encoding="UTF-8" ... /> <dfdl:any encoding="UTF-8" ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> Possible solution is to use dfdl:format to set common properties <xs:annotation><xs:appinfo source=?http://www.ogf.org/dfdl/?> <dfdl:defineFormat> <dfdl:format encoding="UTF-8" ... /> <dfdl:sequence ... /> <dfdl:element ... /> <dfdl:simpleType ... /> <dfdl:choice ... /> <dfdl:any ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> The consensus was that there should be a way to provide common properties so this solution was preferred and you could use the same for complex type <xs:annotation><xs:appinfo source=?http://www.ogf.org/dfdl/?> <dfdl:complexType> <dfdl:format encoding="UTF-8" ... /> <dfdl:sequence ... /> <dfdl:element ... /> <dfdl:simpleType ... /> <dfdl:choice ... /> <dfdl:any ... /> </dfdl:complexType> </xs:appinfo></xs:annotation> It was agreed that this was the preferred solution. During the discussion SK asked what was the precedence of short and long-form properties on the same object. AP subsequently sent SK section 10.3 which states that long-form have precedence over short-form. 2. AOB To be discussed - Escape Schemes - Validation ranges and exclusion lists - Consuming extraneous data after schema has be used. - 'Floating' definitions - Actions Next call 25 March 14:00 UK Meeting closed, 15:00 GMT Actions raised at this meeting No Action Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 023 MB: Review Schema 1.1 29/1: AP and SH to talk to Sandy Gao 04/02 Call arranged for Friday 11/02: Call took place. Identified useful changes. Consolidate with previous list. 04/03: decided to stay on Schema 1.0. 024 String XML type 025 Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 18/02: SH up document concerns 25/02: SH and AP have refined proposal ready for approval. 04/03: SH and AP have further refined proposal. 11/03: discussed. suggested a simplified proposal be evaluated. 18/03: SH and AP had further discussions on simplified proposal 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 031 DG: Review dfdl v033 11/02: Initial comments received 18/02: Will include work items 5 and 12. 11/03: complete 032 DG: Investigate compatibility between DFDL infoset and XDM 033 04/03: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 034 04/03: AP remove redundant properties Closed actions: 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work 026 Remove duration complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF DFDL Working Group Call, March-18-2009
by Alan Powell 20 Mar '09

20 Mar '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, March-18-2009 Attendees Alan Powell (IBM) Steve Hanson (IBM) Dave Glick (drac) Suman Kalia (IBM) Apologies Mike Beckerle(Oco) 1. Specialized Annotations The whole call was taken up discussing the "dfdl:sequence and other specialized annotations" proposal emailed by SH. In summary proposes - the use of specialized annotations on each schema object. eg dfdl:sequence - a separate set of scoped properties for each specialization. <xs:complexType name="myType"> <xs:annotation><xs:appinfo source=?http://www.ogf.org/dfdl/?> <dfdl:complexType> <dfdl:sequence lengthKind="implicit" ... /> <dfdl:element lengthKind="delimited" ... /> </dfdl:complexType> </xs:appinfo></xs:annotation> There was general agreement with the proposal but a lot of discussion about the two issues that SH identified. a. The meaning of short form annotations on complex type. <xs:complexType name="myType" dfdl:lengthKind="implicit" ... > It is not clear which specialized annotation scope the short form properties on complex type should apply to. There was a lot of discussion between 1) disallowing short form annotation on complex type and 2) applying them to all specialized scopes. Disallowing short form is simple but makes complex type different, forces long form which is much more verbose, means common properties, eg encoding, to be specified the same on multiple specialized annotations. This is a particular problem for element and simple type. However applying short form to all specialization introduces a new level of inheritance and consequent complexity. b. You are forced to set common properties in multiple places in defineFormat. <xs:annotation><xs:appinfo source=?http://www.ogf.org/dfdl/?> <dfdl:defineFormat> <dfdl:sequence encoding="UTF-8" ... /> <dfdl:element encoding="UTF-8" ... /> <dfdl:simpleType encoding="UTF-8" ... /> <dfdl:choice encoding="UTF-8" ... /> <dfdl:any encoding="UTF-8" ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> Possible solution is to wrap with dfdl:format <xs:annotation><xs:appinfo source=?http://www.ogf.org/dfdl/?> <dfdl:defineFormat> <dfdl:format encoding="UTF-8" ... > <dfdl:sequence ... /> <dfdl:element ... /> <dfdl:simpleType ... /> <dfdl:choice ... /> <dfdl:any ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> The consensus was that there should be a way to provide common properties so this solution was preferred and you could use the same for complex type <xs:annotation><xs:appinfo source=?http://www.ogf.org/dfdl/?> <dfdl:complexType> <dfdl:format encoding="UTF-8" ... > <dfdl:sequence ... /> <dfdl:element ... /> <dfdl:simpleType ... /> <dfdl:choice ... /> <dfdl:any ... /> </dfdl:complexType> </xs:appinfo></xs:annotation> It was agreed that this was the preferred solution. During the discussion SK asked what was the precedence of short and long-form properties on the same object. AP subsequently sent SK section 10.3 which states that long-form have precedence over short-form. 2. AOB To be discussed - Escape Schemes - Validation ranges and exclusion lists - Consuming extraneous data after schema has be used. - 'Floating' definitions - Actions Next call 25 March 14:00 UK Meeting closed, 15:00 GMT Actions raised at this meeting No Action Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 023 MB: Review Schema 1.1 29/1: AP and SH to talk to Sandy Gao 04/02 Call arranged for Friday 11/02: Call took place. Identified useful changes. Consolidate with previous list. 04/03: decided to stay on Schema 1.0. 024 String XML type 025 Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 18/02: SH up document concerns 25/02: SH and AP have refined proposal ready for approval. 04/03: SH and AP have further refined proposal. 11/03: discussed. suggested a simplified proposal be evaluated. 18/03: SH and AP had further discussions on simplified proposal 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 031 DG: Review dfdl v033 11/02: Initial comments received 18/02: Will include work items 5 and 12. 11/03: complete 032 DG: Investigate compatibility between DFDL infoset and XDM 033 04/03: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 034 04/03: AP remove redundant properties Closed actions: 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work 026 Remove duration complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Precedence of short and long form properties
by Alan Powell 20 Mar '09

20 Mar '09

Suman Correct Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Suman Kalia/Toronto/IBM@IBMCA To: Alan Powell/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 19/03/2009 19:18 Subject: Re: Precedence of short and long form properties Thanks Alan -- In nutshell, long from annotations take precedence over short form when they are defined on the same xsd construct.. i.e. if you lengthKind specified in short form and on dfdl:element on XSD element declaration value specified in dfdl:element will take precedence.. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com From: Alan Powell/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA Cc: dfdl-wg(a)ogf.org% Date: 03/19/2009 02:11 PM Subject: Precedence of short and long form properties Suman One the last work group call you raised a question about precedence of the same property in short and long form that we were unable to answer. The last paragraph of section 10.3 provides the answer 10.3 Annotation Positioning As described in Section ‎5.2, DFDL annotations are positioned at specific annotation points within a DFDL schema. The table below shows the valid annotation points. Annotation Point | Property Scope ----------------------+----------------------------------------------------- Schema declaration | Invalid | | Only top level defining forms (e.g., | dfdl:defineFormat) can appear at top level of the | schema. These definitions are globally available. | Nothing is put into effect about the format of | data by these top-level definitions. ----------------------+----------------------------------------------------- Element declaration | Valid local ----------------------+----------------------------------------------------- Element reference | Valid local ----------------------+----------------------------------------------------- Complex type | Valid scoped over contents definition | ----------------------+----------------------------------------------------- Simple type | Valid local definition | ----------------------+----------------------------------------------------- Sequence | Valid local declaration | ----------------------+----------------------------------------------------- Choice declaration | Valid local ----------------------+----------------------------------------------------- Group reference | Valid local An annotation on a complexType definition applies over the scope of the contents and so is inherited by any contained constructs or construct references. That is, the scope include not only the part of the schema lexically enclosed by the complexType declaration which puts this scope into effect, but these properties are also inherited by these specific instances of any referenced types, referenced elements, or referenced groups from within that scope. This is sometimes called scoping over the full dynamic extent of the scope, not just the lexically appearing scope. When multiple DFDL annotation properties occur at the same annotation point then they are combined with the rule that later format annotations override earlier ones, (later meaning textually later in the schema document) and short-form annotations are interpreted as if they appeared in a long-form annotation that is first before any other long-form annotations. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898

1 0

Re: [DFDL-WG] Precedence of short and long form properties
by Suman Kalia 19 Mar '09

19 Mar '09

Thanks Alan -- In nutshell, long from annotations take precedence over short form when they are defined on the same xsd construct.. i.e. if you lengthKind specified in short form and on dfdl:element on XSD element declaration value specified in dfdl:element will take precedence.. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com From: Alan Powell/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA Cc: dfdl-wg(a)ogf.org Date: 03/19/2009 02:11 PM Subject: Precedence of short and long form properties Suman One the last work group call you raised a question about precedence of the same property in short and long form that we were unable to answer. The last paragraph of section 10.3 provides the answer 10.3 Annotation Positioning As described in Section ‎5.2, DFDL annotations are positioned at specific annotation points within a DFDL schema. The table below shows the valid annotation points. Annotation Point Property Scope Schema declaration Invalid Only top level defining forms (e.g., dfdl:defineFormat) can appear at top level of the schema. These definitions are globally available. Nothing is put into effect about the format of data by these top-level definitions. Element declaration Valid local Element reference Valid local Complex type definition Valid scoped over contents Simple type definition Valid local Sequence declaration Valid local Choice declaration Valid local Group reference Valid local An annotation on a complexType definition applies over the scope of the contents and so is inherited by any contained constructs or construct references. That is, the scope include not only the part of the schema lexically enclosed by the complexType declaration which puts this scope into effect, but these properties are also inherited by these specific instances of any referenced types, referenced elements, or referenced groups from within that scope. This is sometimes called scoping over the full dynamic extent of the scope, not just the lexically appearing scope. When multiple DFDL annotation properties occur at the same annotation point then they are combined with the rule that later format annotations override earlier ones, (later meaning textually later in the schema document) and short-form annotations are interpreted as if they appeared in a long-form annotation that is first before any other long-form annotations. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898

1 0

Precedence of short and long form properties
by Alan Powell 19 Mar '09

19 Mar '09

Suman One the last work group call you raised a question about precedence of the same property in short and long form that we were unable to answer. The last paragraph of section 10.3 provides the answer 10.3 Annotation Positioning As described in Section ‎5.2, DFDL annotations are positioned at specific annotation points within a DFDL schema. The table below shows the valid annotation points. Annotation Point Property Scope Schema declaration Invalid Only top level defining forms (e.g., dfdl:defineFormat) can appear at top level of the schema. These definitions are globally available. Nothing is put into effect about the format of data by these top-level definitions. Element declaration Valid local Element reference Valid local Complex type definition Valid scoped over contents Simple type definition Valid local Sequence declaration Valid local Choice declaration Valid local Group reference Valid local An annotation on a complexType definition applies over the scope of the contents and so is inherited by any contained constructs or construct references. That is, the scope include not only the part of the schema lexically enclosed by the complexType declaration which puts this scope into effect, but these properties are also inherited by these specific instances of any referenced types, referenced elements, or referenced groups from within that scope. This is sometimes called scoping over the full dynamic extent of the scope, not just the lexically appearing scope. When multiple DFDL annotation properties occur at the same annotation point then they are combined with the rule that later format annotations override earlier ones, (later meaning textually later in the schema document) and short-form annotations are interpreted as if they appeared in a long-form annotation that is first before any other long-form annotations. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

details vs. the big picture
by Sampo Syreeni 18 Mar '09

18 Mar '09

Hi. I've been following this list for a while now, because as a data modelling kinda person I'm interested in formats, conversions, metamodels, their mappings, and the like. I'm afraid to say the slow development of DFDL has disappointed me. To me it seems quite a number of implementations of DFDL should have been out there already, yet they aren't. I think the main reasons are that people in this working group are: a) concentrating too much on the details of specific binary formats to be parsed/use cases instead of a generic parsing framework for common binary data as a whole, b) paying too much attention to the XML encoding of the language, instead of its semantics, or its possible alternative encodings, c) spending far too much time meeting, specifying, theoretizing, and downright bickering-in-committee, instead of piece-meal implementing the most useful parts of the standard, d) contrariwise, optimizing the specification prematurely, e.g. with the hesitancy to declare multidimensional arrays as a first class citizen even if there are unresolved issues, e) doing all of the real work over the telephone or in person, so that low budget, practical minded people cannot contribute meaningfully, f) failing to show other people what the worth of this whole effort is, and thus failing to liaise efficiently even with the Grid Forum, and thereby, finally, g) failing to engage the broader circle of developers, managers, curators, nerds, and whathaveyou, who might be interested in this sort of thing. Is it at all possible that this effort could somehow move into the mainstream in the near future? That outside forces, like the FOSS community, could speed up the transition? In all, that some of the above peeves of mine could be rectified? -- Sampo Syreeni, aka decoy - decoy(a)iki.fi, http://decoy.iki.fi/front +358-50-5756111, 025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2

1 0

Agenda for OGF WG call 18 March 2009
by Alan Powell 18 Mar '09

18 Mar '09

Suggested agenda 1. Validation ranges and exclusion lists - vote 2. dfdl format specializations 3. Escape Schemes 4. Actions 5. 'Floating' definitions 6. Consuming extraneous data after schema has be used. 7 AOB Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: dfdl:sequence and other specialized annotations
by Steve Hanson 18 Mar '09

18 Mar '09

Any thoughts on this? Note one correction below. ---------------------------------------------------- We discussed this on the call yesterday as minuted by Alan. I missed xsd:any from the table in the mail below, so that's added. Specialized annotations may help alleviate something that has been noted by folk at IBM when creating DFDL schemas. If using dfdl:format on complex types to set properties, it is usually to default values for the elements of the complex type. But the first thing following the complex type is the model group (sequence, choice) so any properties with the same name have to be explicitly set. That's all working as designed but it is a common mistake to forget to do this. The cause is having properties with the same name on both sequence and element. Having specialized annotations provides a way round this by allowing the introduction of 'parallel' scoping. <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:defineFormat> <dfdl:sequence terminator="$" ... /> <dfdl:element terminator="%" ... /> <dfdl:simpleType ... /> <dfdl:choice ... /> <dfdl:any ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> <xs:complexType name="myType"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:complexType> <dfdl:sequence lengthKind="implicit" ... /> <dfdl:element lengthKind="delimited" ... /> </dfdl:complexType> </xs:appinfo></xs:annotation> <xs:sequence> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:sequence ... /> </xs:appinfo></xs:annotation> <xs:element name="el1" type="xs:string"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:element ... /> </xs:appinfo></xs:annotation> </xs:element> <xs:element name="el2" type="xs:string"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:element ... /> </xs:appinfo></xs:annotation> </xs:element> <xs:element name="el3" type="xs:string"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:element ... /> </xs:appinfo></xs:annotation> </xs:element> </xs:sequence> </xs:complexType> This makes it very clear which properties are going to be used where and allows separate defaults to be set up for different kinds of xsd object. But there are a couple of problems with this. 1) What does this mean? <xs:complexType name="myType" dfdl:lengthKind="implicit" ... > Possible answers: a) We don't allow short form on xs:complexType. b) Short form applies to all DFDL specializations c) Short form applies to one specific DFDL specialization, eg, dfdl:element. Personally I think a) would be a shame and a loss of convenience; c) is too arbitrary; b) looks the most sensible, but opens up another question, namely whether this equivalent is also allowed. <xs:complexType name="myType"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:complexType lengthKind="delimited"> <dfdl:sequence lengthKind="implicit" ... /> <dfdl:element lengthKind="delimited" ... /> </dfdl:complexType> </xs:appinfo></xs:annotation> 2) I am forced to set common properties in multiple places. <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:defineFormat> <dfdl:sequence encoding="UTF-8" ... /> <dfdl:element encoding="UTF-8" ... /> <dfdl:simpleType encoding="UTF-8" ... /> <dfdl:choice encoding="UTF-8" ... /> <dfdl:any encoding="UTF-8" ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> But we could also allow this, and say that dfdl:format applied to all xsd objects. Note that this is consistent with 1) b) above, <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:defineFormat> <dfdl:format encoding="UTF-8" ... /> <dfdl:sequence ... /> <dfdl:element ... /> <dfdl:simpleType ... /> <dfdl:choice ... /> <dfdl:any ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> We should also bear in mind when deciding all this that we do already provide one other facility which provides a form of parallelism. <dfdl:format ref=”reusableDef"... /> So I can already create a set of annotations applicable to a specific kind of xsd object and reuse that via the dfdl:ref attribute. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 12/03/2009 10:00 ----- Steve Hanson/UK/IBM 09/03/2009 16:10 To DFDL Working Group cc Subject Fw: [DFDL-WG] dfdl:sequence and other specialized annotations Although it was minuted in last week's call that we decided to drop specialised annotations, I don't actually recall that was decision on the call. The proposal below is in accordance with the suggestion by Martin, and echoed by Dave, that we have only specialised annotations for xsd objects. Schema object DFDL annotation allowed Notes Schema declaration dfdl:format Only within an enclosing dfd:defineFormat. Element declaration dfdl:element Just those properties allowed on elements Element reference dfdl:element Just those properties allowed on elements. Overrides dfdl:element properties on global element Complex type definition dfdl:complexType Full dfdl:format properties. For scoping purposes only. Simple type definition dfdl:simpleType Just those properties allowed on simple types. Sequence declaration dfdl:sequence Just those properties allowed on sequences. Choice declaration dfdl:choice Just those properties allowed on choices. Group reference dfdl:group Just those properties allowed on sequence/choice. Overrides dfdl:sequence/dfdl:choice properties on global group sequence/choice model group Wildcard element dfdl:any Just those properties allowed on wildcard elementsv An alternative as I previously wrote is to allow dfdl:sequence and dfdl:choice on a group reference. Group reference (to sequence group) dfdl:sequence Just those properties allowed on sequence/choice. Overrides dfdl:sequence properties on global group model sequence group Group reference (to choice group) dfdl:choice Just those properties allowed on sequence/choice. Overrides dfdl:choice properties on global group model choice group Although this does not look as good visually, it is worth bearing in mind section 22.2 of the spec: 22.2 Group Reference A copy of the definition referenced by a group reference is substituted inline for its declaration while combining any DFDL format annotation elements associated with it as given in Section ‎10.5. (TBD: xref to scoping?) Processing then proceeds as for an anonymous group declaration. Note: The use of specialised annotations relies on the removal of something that we had talked about but not yet reflected in the spec - dfdl:format on global xsd:group definitions. The idea of allowing this came about fairly recently when we changed the rules for hidden objects. I was unhappy that embedded hidden xsd objects within DFDL annotations would make them invisible to a schema validator, so dfdl:hidden was changed to be a reference to a global group containing the hidden objects. I suggested that we could also put scoping annotations on global groups to support scoping within the hidden construct, it seemed a good idea at the time, but it was not fully explored. Thinking it through, because the global group effectively disappears when the reference is resolved, then so does its annotations. I'm therefore happy to retract scoping annotations on global groups if it makes specialised annotations work. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia <kalia(a)ca.ibm.com> Cc: dfdl-wg(a)ogf.org Date: 02/03/2009 23:45 Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Suman A DFDL annotation on a group reference overrides a DFDL annotation on the content of the model group (ie, sequence, choice), not a DFDL annotation on the global group. It allows you to override the sequence/choice properties at point of use. The DFDL annotation on a global group is like that on a complex type - it is a scoping construct only. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Suman Kalia <kalia(a)ca.ibm.com> 02/03/2009 22:02 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org, "Dave Glick" <dglick(a)dracorp.com> Subject Re: [DFDL-WG] Reducing the number of DFDL properties. I can see where you are coming from and I agree with most of your proposal except putting dfdl:sequence, dfdl:choice , dfdl:all on group references as it does not give the right optics.. From consistency point of view, the references (group and element references) should have the equivalent/same set of annotations that can appear on the referenced artifacts. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com From: Steve Hanson <smh(a)uk.ibm.com> To: Suman Kalia/Toronto/IBM@IBMCA Cc: dfdl-wg(a)ogf.org, "Dave Glick" <dglick(a)dracorp.com> Date: 03/02/2009 12:45 PM Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. If I recall, one reason for having the specialised elements was so that the content of the annotations could be more tightly validated by the schema-for-dfdl xsd. However, the problem with the specialised annotations is that we do not provide them for all circumstances. For example, the spec allows me to use dfdl:sequence on an xs:sequence but not an xs:group reference to a global xs:group with compostion xs:sequence. Nor does it allow me to use a specialised element on xs:simpleType or xs:restriction. So as it stands, validation would not be fool-proof anyway. I think we must either ditch specialisations, or mandate the use of specialisations on the corresponding schema objects. The latter is more workable now we have changed the scoping rules so that scoping of dfdl properties can only be done from complex type or schema level, as below. But it does mean that dfdl annotations are less flexible. dfdl;defineFormat => dfdl:format xs:complexType => dfdl:format xs:group => dfdl:format xs:sequence or xs:group ref to a global group with xs:sequence => dfdl:sequence xs:choice or xs:group ref to a global group with xs:choice => dfdl:choice xs:element or xs:element ref => dfdl:element xs:any => dfdl:any xs:simpleType, xs:restriction => dfdl:simpleType In other words, you use dfdl:format when you are scoping properties, you must use specialised annotation for specific objects. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Suman Kalia <kalia(a)ca.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 02/03/2009 16:25 To "Dave Glick" <dglick(a)dracorp.com> cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Subject Re: [DFDL-WG] Reducing the number of DFDL properties. I am not sure what kind of confusion or redundancy is caused by these specialized annotations. In the absence of these specialized annotations, you will have to go through plethora of annotations and determine which ones are applicable for sequence , choice, all, elements etc.. The danger is we don't have these levels of abstractions, then number of folks (specifically implementers) would build them anyway and then we will have to contend with incompatible abstractions.. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com From: "Dave Glick" <dglick(a)dracorp.com> To: "Alan Powell" <alan_powell(a)uk.ibm.com>, <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org Date: 03/02/2009 11:16 AM Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Alan/Mike, I agree with this – one of my complaints with v033 was that these specialized annotation elements just added confusion and redundancy. Dave --- David Glick | dglick(a)dracorp.com | 703.299.0700 x212 Data Research and Analysis Corp. | www.dracorp.com From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Alan Powell Sent: Monday, March 02, 2009 11:03 AM To: mbeckerle.dfdl(a)gmail.com Cc: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Mike Thanks Can I also suggest dropping the dfdl:sequence, dfdl:choice, dfdl:element and dfdl:any specialized annotations and just have dfdl:format. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Alan Powell/UK/IBM@IBMGB Cc: <dfdl-wg(a)ogf.org> Date: 28/02/2009 19:38 Subject: RE: Reducing the number of DFDL properties. If a property is redundant, I'm in favor of dropping it. If a property adds generality that we don't have a use case for, I'm in favor of dropping it. I am happy to drop type substitution from v1.0. It's a convenience that can be achieved a different way. E.g., if you really want "float" to mean "float in my particular binary representation", then just put a type definition with DFDL annotations in a different namespace, and when you write your DFDL schema, arrange for the default namespace to pick it up from your namespace, and not xs:float. <xs:element name="myElement" type="float"/>  <xs:element name="anotherElem" type="xs:float"/>  If you want the XSD unadorned "float" type, be explicit and use "xs:float". Voila - no loss of flexibility, equal textual convenience. I think that would satisfy the community that really wanted very compact "slideware acceptable" schemas. This is the same group that wants short-form annotations as well. Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 100 Fifth Ave., 4th Floor, Waltham MA 02451 | mbeckerle.dfdl(a)gmail.com From: Alan Powell [mailto:alan_powell@uk.ibm.com] Sent: Thursday, February 26, 2009 12:22 PM To: mbeckerle.dfdl(a)gmail.com Cc: dfdl-wg(a)ogf.org Subject: Reducing the number of DFDL properties. Mike A number of people at IBM have become concerned at the number of properties in DFDL and have identified a number of 'usability' properties that could be dropped. They feel that we should be simplifying the properties wherever possible and not introducing multiple ways of doing the same function without very good reason. The following are offered for consideration. 1. lengthKind='nullterminated' This is just shorthand for lengthKind=delimited and terminator='%Null'. It was felt this this is not even the most common terminator so why have a special case? 2. trimKind It is felt that there aren't any cases when you would want to pad but not trim and vice versa so make padKind control both. 3. typeSubstitution. Is this needed in DFDL v1? Can you consider these before the call next week Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: xsd:any support - propose dfdl:initiatorLength property
by Steve Hanson 18 Mar '09

18 Mar '09

It also occurred to me that, if we follow through with our 'variable markup' piece of work, we could remove these properties altogether and use variable markup instead. Eg: Use dfdl:initiatorType property which referred to a simple type by name that carried DFDL properties that described how to extract the initiator. Not as convenient, but xsd:any is unlikely to be used that frequently. One thing we have not defined is how the element name for the infoset is deduced from the initiator. This is very straightforward with variable markup. It's just the value returned by the dfdl:initiatorType. If using separate properties we have to say what happens to white space etc. Neither have we explicitly said what types of DFDL representations are allowed for xsd:any. I think we had been assuming xsd:string only, but I don't see why it can't be any simple type? Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/03/2009 12:01 ----- Steve Hanson/UK/IBM 16/03/2009 10:45 To DFDL Working Group cc Subject xsd:any support - propose dfdl:initiatorLength property Here's what we say about xsd:any (wildcard support) in DFDL spec. Any Element Wildcard The "any" element wildcard provides support for initiator-tagged data formats. The dfdl:initiatorSeparator property must be defined or it is a schema definition error. The 'processContents' attribute is not allowed in DFDL 1.0 and it's use is a schema definition error. This kind of wildcard is used to absorb content which while syntactically compatible is not logically modeled as elements of the sequence group. These declarations match a triplet of strings. The first is a variable-length string that matches any initiator tag. The second is a fixed value string which must match the initiator-separator property value. The third is a variable-length string that matches any content up until the end of the "element" being matched. The length of data matching an any-element wildcard is specified by the usual length properties: dfdl:lengthKind, dfdl:lengthUnits, dfdl:length, dfdl:terminator. dfdl:initiator is not used by Any Element wildcards and is ignored if specified in scope. We insist on the presence of a known initiator separator to separate initiator from data. We should also allow initiator length, to support models where there is no separator. Exactly one of initiator separator and initiator length must be specified. Initiator length is not a valid DFDL property on any other xsd object. This is supported today by IBM's MRM parser. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0