March 2009 - dfdl-wg - lists.ogf.org

xsd:any support - propose dfdl:initiatorLength property
by Steve Hanson 16 Mar '09

16 Mar '09

Here's what we say about xsd:any (wildcard support) in DFDL spec. Any Element Wildcard The "any" element wildcard provides support for initiator-tagged data formats. The dfdl:initiatorSeparator property must be defined or it is a schema definition error. The 'processContents' attribute is not allowed in DFDL 1.0 and it's use is a schema definition error. This kind of wildcard is used to absorb content which while syntactically compatible is not logically modeled as elements of the sequence group. These declarations match a triplet of strings. The first is a variable-length string that matches any initiator tag. The second is a fixed value string which must match the initiator-separator property value. The third is a variable-length string that matches any content up until the end of the "element" being matched. The length of data matching an any-element wildcard is specified by the usual length properties: dfdl:lengthKind, dfdl:lengthUnits, dfdl:length, dfdl:terminator. dfdl:initiator is not used by Any Element wildcards and is ignored if specified in scope. We insist on the presence of a known initiator separator to separate initiator from data. We should also allow initiator length, to support models where there is no separator. Exactly one of initiator separator and initiator length must be specified. Initiator length is not a valid DFDL property on any other xsd object. This is supported today by IBM's MRM parser. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: dfdl:sequence and other specialized annotations
by Steve Hanson 12 Mar '09

12 Mar '09

We discussed this on the call yesterday as minuted by Alan. I missed xsd:any from the table in the mail below, so that's added. Specialized annotations may help alleviate something that has been noted by folk at IBM when creating DFDL schemas. If using dfdl:format on complex types to set properties, it is usually to default values for the elements of the complex type. But the first thing following the complex type is the model group (sequence, choice) so any properties with the same name have to be explicitly set. That's all working as designed but it is a common mistake to forget to do this. The cause is having properties with the same name on both sequence and element. Having specialized annotations provides a way round this by allowing the introduction of 'parallel' scoping. <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:defineFormat> <dfdl:sequence terminator="$" ... /> <dfdl:element terminator="%" ... /> <dfdl:simpleType ... /> <dfdl:choice ... /> <dfdl:any ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> <xs:complexType name="myType"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:complexType> <dfdl:sequence lengthKind="implicit" ... /> <dfdl:element lengthKind="delimited" ... /> </dfdl:complexType> </xs:appinfo></xs:annotation> <xs:sequence> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:sequence ... /> </xs:appinfo></xs:annotation> <xs:element name="el1" type="xs:string"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:element ... /> </xs:appinfo></xs:annotation> </xs:element> <xs:element name="el2" type="xs:string"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:element ... /> </xs:appinfo></xs:annotation> </xs:element> <xs:element name="el3" type="xs:string"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:element ... /> </xs:appinfo></xs:annotation> </xs:element> </xs:sequence> </xs:complexType> This makes it very clear which properties are going to be used where and allows separate defaults to be set up for different kinds of xsd object. But there are a couple of problems with this. 1) What does this mean? <xs:complexType name="myType" dfdl:lengthKind="implicit" ... > Possible answers: a) We don't allow short form on xs:complexType. b) Short form applies to all DFDL specializations c) Short form applies to one specific DFDL specialization, eg, dfdl:element. Personally I think a) would be a shame and a loss of convenience; c) is too arbitrary; b) looks the most sensible, but opens up another question, namely whether this equivalent is also allowed. <xs:complexType name="myType"> <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:complexType lengthKind="delimited"> <dfdl:sequence lengthKind="implicit" ... /> <dfdl:element lengthKind="delimited" ... /> </dfdl:complexType> </xs:appinfo></xs:annotation> 2) I am forced to set common properties in multiple places. <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:defineFormat> <dfdl:sequence encoding="UTF-8" ... /> <dfdl:element encoding="UTF-8" ... /> <dfdl:simpleType encoding="UTF-8" ... /> <dfdl:choice encoding="UTF-8" ... /> <dfdl:any encoding="UTF-8" ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> But we could also allow this, and say that dfdl:format applied to all xsd objects. Note that this is consistent with 1c) above, <xs:annotation><xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:defineFormat> <dfdl:format encoding="UTF-8" ... /> <dfdl:sequence ... /> <dfdl:element ... /> <dfdl:simpleType ... /> <dfdl:choice ... /> <dfdl:any ... /> </dfdl:defineFormat> </xs:appinfo></xs:annotation> We should also bear in mind when deciding all this that we do already provide one other facility which provides a form of parallelism. <dfdl:format ref=”reusableDef"... /> So I can already create a set of annotations applicable to a specific kind of xsd object and reuse that via the dfdl:ref attribute. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 12/03/2009 10:00 ----- Steve Hanson/UK/IBM 09/03/2009 16:10 To DFDL Working Group cc Subject Fw: [DFDL-WG] dfdl:sequence and other specialized annotations Although it was minuted in last week's call that we decided to drop specialised annotations, I don't actually recall that was decision on the call. The proposal below is in accordance with the suggestion by Martin, and echoed by Dave, that we have only specialised annotations for xsd objects. Schema object DFDL annotation allowed Notes Schema declaration dfdl:format Only within an enclosing dfd:defineFormat. Element declaration dfdl:element Just those properties allowed on elements Element reference dfdl:element Just those properties allowed on elements. Overrides dfdl:element properties on global element Complex type definition dfdl:complexType Full dfdl:format properties. For scoping purposes only. Simple type definition dfdl:simpleType Just those properties allowed on simple types. Sequence declaration dfdl:sequence Just those properties allowed on sequences. Choice declaration dfdl:choice Just those properties allowed on choices. Group reference dfdl:group Just those properties allowed on sequence/choice. Overrides dfdl:sequence/dfdl:choice properties on global group sequence/choice model group Wildcard element dfdl:any Just those properties allowed on wildcard elementsv An alternative as I previously wrote is to allow dfdl:sequence and dfdl:choice on a group reference. Group reference (to sequence group) dfdl:sequence Just those properties allowed on sequence/choice. Overrides dfdl:sequence properties on global group model sequence group Group reference (to choice group) dfdl:choice Just those properties allowed on sequence/choice. Overrides dfdl:choice properties on global group model choice group Although this does not look as good visually, it is worth bearing in mind section 22.2 of the spec: 22.2 Group Reference A copy of the definition referenced by a group reference is substituted inline for its declaration while combining any DFDL format annotation elements associated with it as given in Section ‎10.5. (TBD: xref to scoping?) Processing then proceeds as for an anonymous group declaration. Note: The use of specialised annotations relies on the removal of something that we had talked about but not yet reflected in the spec - dfdl:format on global xsd:group definitions. The idea of allowing this came about fairly recently when we changed the rules for hidden objects. I was unhappy that embedded hidden xsd objects within DFDL annotations would make them invisible to a schema validator, so dfdl:hidden was changed to be a reference to a global group containing the hidden objects. I suggested that we could also put scoping annotations on global groups to support scoping within the hidden construct, it seemed a good idea at the time, but it was not fully explored. Thinking it through, because the global group effectively disappears when the reference is resolved, then so does its annotations. I'm therefore happy to retract scoping annotations on global groups if it makes specialised annotations work. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia <kalia(a)ca.ibm.com> Cc: dfdl-wg(a)ogf.org Date: 02/03/2009 23:45 Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Suman A DFDL annotation on a group reference overrides a DFDL annotation on the content of the model group (ie, sequence, choice), not a DFDL annotation on the global group. It allows you to override the sequence/choice properties at point of use. The DFDL annotation on a global group is like that on a complex type - it is a scoping construct only. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Suman Kalia <kalia(a)ca.ibm.com> 02/03/2009 22:02 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org, "Dave Glick" <dglick(a)dracorp.com> Subject Re: [DFDL-WG] Reducing the number of DFDL properties. I can see where you are coming from and I agree with most of your proposal except putting dfdl:sequence, dfdl:choice , dfdl:all on group references as it does not give the right optics.. From consistency point of view, the references (group and element references) should have the equivalent/same set of annotations that can appear on the referenced artifacts. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com From: Steve Hanson <smh(a)uk.ibm.com> To: Suman Kalia/Toronto/IBM@IBMCA Cc: dfdl-wg(a)ogf.org, "Dave Glick" <dglick(a)dracorp.com> Date: 03/02/2009 12:45 PM Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. If I recall, one reason for having the specialised elements was so that the content of the annotations could be more tightly validated by the schema-for-dfdl xsd. However, the problem with the specialised annotations is that we do not provide them for all circumstances. For example, the spec allows me to use dfdl:sequence on an xs:sequence but not an xs:group reference to a global xs:group with compostion xs:sequence. Nor does it allow me to use a specialised element on xs:simpleType or xs:restriction. So as it stands, validation would not be fool-proof anyway. I think we must either ditch specialisations, or mandate the use of specialisations on the corresponding schema objects. The latter is more workable now we have changed the scoping rules so that scoping of dfdl properties can only be done from complex type or schema level, as below. But it does mean that dfdl annotations are less flexible. dfdl;defineFormat => dfdl:format xs:complexType => dfdl:format xs:group => dfdl:format xs:sequence or xs:group ref to a global group with xs:sequence => dfdl:sequence xs:choice or xs:group ref to a global group with xs:choice => dfdl:choice xs:element or xs:element ref => dfdl:element xs:any => dfdl:any xs:simpleType, xs:restriction => dfdl:simpleType In other words, you use dfdl:format when you are scoping properties, you must use specialised annotation for specific objects. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Suman Kalia <kalia(a)ca.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 02/03/2009 16:25 To "Dave Glick" <dglick(a)dracorp.com> cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Subject Re: [DFDL-WG] Reducing the number of DFDL properties. I am not sure what kind of confusion or redundancy is caused by these specialized annotations. In the absence of these specialized annotations, you will have to go through plethora of annotations and determine which ones are applicable for sequence , choice, all, elements etc.. The danger is we don't have these levels of abstractions, then number of folks (specifically implementers) would build them anyway and then we will have to contend with incompatible abstractions.. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com From: "Dave Glick" <dglick(a)dracorp.com> To: "Alan Powell" <alan_powell(a)uk.ibm.com>, <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org Date: 03/02/2009 11:16 AM Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Alan/Mike, I agree with this – one of my complaints with v033 was that these specialized annotation elements just added confusion and redundancy. Dave --- David Glick | dglick(a)dracorp.com | 703.299.0700 x212 Data Research and Analysis Corp. | www.dracorp.com From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Alan Powell Sent: Monday, March 02, 2009 11:03 AM To: mbeckerle.dfdl(a)gmail.com Cc: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Mike Thanks Can I also suggest dropping the dfdl:sequence, dfdl:choice, dfdl:element and dfdl:any specialized annotations and just have dfdl:format. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Alan Powell/UK/IBM@IBMGB Cc: <dfdl-wg(a)ogf.org> Date: 28/02/2009 19:38 Subject: RE: Reducing the number of DFDL properties. If a property is redundant, I'm in favor of dropping it. If a property adds generality that we don't have a use case for, I'm in favor of dropping it. I am happy to drop type substitution from v1.0. It's a convenience that can be achieved a different way. E.g., if you really want "float" to mean "float in my particular binary representation", then just put a type definition with DFDL annotations in a different namespace, and when you write your DFDL schema, arrange for the default namespace to pick it up from your namespace, and not xs:float. <xs:element name="myElement" type="float"/>  <xs:element name="anotherElem" type="xs:float"/>  If you want the XSD unadorned "float" type, be explicit and use "xs:float". Voila - no loss of flexibility, equal textual convenience. I think that would satisfy the community that really wanted very compact "slideware acceptable" schemas. This is the same group that wants short-form annotations as well. Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 100 Fifth Ave., 4th Floor, Waltham MA 02451 | mbeckerle.dfdl(a)gmail.com From: Alan Powell [mailto:alan_powell@uk.ibm.com] Sent: Thursday, February 26, 2009 12:22 PM To: mbeckerle.dfdl(a)gmail.com Cc: dfdl-wg(a)ogf.org Subject: Reducing the number of DFDL properties. Mike A number of people at IBM have become concerned at the number of properties in DFDL and have identified a number of 'usability' properties that could be dropped. They feel that we should be simplifying the properties wherever possible and not introducing multiple ways of doing the same function without very good reason. The following are offered for consideration. 1. lengthKind='nullterminated' This is just shorthand for lengthKind=delimited and terminator='%Null'. It was felt this this is not even the most common terminator so why have a special case? 2. trimKind It is felt that there aren't any cases when you would want to pad but not trim and vice versa so make padKind control both. 3. typeSubstitution. Is this needed in DFDL v1? Can you consider these before the call next week Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, March-11-2009
by Alan Powell 11 Mar '09

11 Mar '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, March-11-2009 Attendees Alan Powell (IBM) Steve Hanson (IBM) Dave Glick (drac) Suman Kalia (IBM) Apologies Mike Beckerle(Oco) Tim Kimber (IBM) 1. Specialized Annotations Discussed SH proposal for using specialized annotations. Generally thought to be reasonable if details can be worked out. Additional suggestion for dfdl:format and dfdl:complextype to allow specialized annotaions as children rather than them being the union of all properties. Led to perhaps allowing scoped properties to be tied to specialized annotation. eg dfdl:sequence lengthKind and dfdl:element: lengthKind. Identified problem with short form properties on complexType, which specialized annotation should they apply to? SH to update proposal. 2. Escape Schemes IBM have done work to complete the current proposal (006) to provide a general mechanism that covers all cases and have concluded that it ends up to be much to complicated. They have suggested a simplified version that covers that major cases only. Summary. : allow a single reference on field to an escape scheme. The escapeScheme defines one of 3 different escape alternatives - escapeCharacter and escapeforescapeCharacter - escape surround/bracket with escapeforescapeCharacter - escape surround/bracket with alternate escape surround/bracket Also suggested adding property to list 'Other characters that need to be escaped' as a catchall mechanism AP document simplified proposal 3. Validation ranges and exclusion lists. General agreement that validation outside that provided by schema validation should be avoided. Discussed SH emailed proposal to use unions to define validation ranges. Need to decide if benefit of validation ranges has enough value to justify the complication of allowing limited union support. Be prepared to decide on the next call. 4. AOB To be discussed - Consuming extraneous data after schema has be used. - 'Floating' definitions - Actions Next call 18 March 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 023 MB: Review Schema 1.1 29/1: AP and SH to talk to Sandy Gao 04/02 Call arranged for Friday 11/02: Call took place. Identified useful changes. Consolidate with previous list. 04/03: decided to stay on Schema 1.0. 024 String XML type 025 Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 18/02: SH up document concerns 25/02: SH and AP have refined proposal ready for approval. 04/03: SH and AP have further refined proposal. 11/03: discussed. suggested a simplified proposal be evaluated. 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 031 DG: Review dfdl v033 11/02: Initial comments received 18/02: Will include work items 5 and 12. 11/03: complete 032 DG: Investigate compatibility between DFDL infoset and XDM 033 04/03: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 034 04/03: AP remove redundant properties Closed actions: 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work 026 Remove duration complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Can't attend call today
by DFDL 11 Mar '09

11 Mar '09

Pesky customers ...mikeb

1 0

Numbers with multiple ranges
by Steve Hanson 11 Mar '09

11 Mar '09

One of the existing non-XML parsers from IBM is capable of modelling an integer or decimal with non-contiguous ranges, eg, 0-100, 200-300, 400-500. It is also possible to model this using XML Schema, using a user-defined simple type which is a union of simple type restrictions. However it is not possible to model this in DFDL because unions are not supported. A choice could be used, but that would give different named elements in the resultant infoset depending on the value, which is not ideal. I believe this is use case that we should consider for DFDL 1.0. We could decide to allow unions, but with a constraint that the members of the union had to be restrictions of the same built-in type, and that clashing DFDL properties on the members was an error. I believe the existing scoping rules for visiting types would still apply ok. No annotation appears on the xs:union only on the xs:simpleType as currently. The net effect is that our parsing rules are the same as currently, but we gain validation capability. I am not in favour of more widespread support for unions. I don't have a compelling use case for union of different logical types, eg, xs:dateTime and xs:string. Your thoughts are welcome. !-- integer range 0 - 100 <xsd:simpleType name="intRange1"> <xsd:restriction base="xsd:int"> <xsd:maxInclusive value="100"/> <xsd:minInclusive value="0"/> </xsd:restriction> </xsd:simpleType> !-- integer range 200 - 300 <xsd:simpleType name="intRange2"> <xsd:restriction base="xsd:int"> <xsd:maxInclusive value="300"/> <xsd:minInclusive value="200"/> </xsd:restriction> </xsd:simpleType> !-- integer range 400-500 <xsd:simpleType name="intRange3"> <xsd:restriction base="xsd:int"> <xsd:maxInclusive value="500"/> <xsd:minInclusive value="400"/> </xsd:restriction> </xsd:simpleType> !-- Union of above types <xsd:simpleType name="intRange"> <xsd:union memberTypes="intRange1 intRange2 intRange3"/> </xsd:simpleType> !-- Union of anonymous types being restrictions of above types <xsd:simpleType name="intRangeLocal"> <xsd:union> <xsd:simpleType> <xsd:restriction base="intRange1"/> </xsd:simpleType> <xsd:simpleType> <xsd:restriction base="intRange2"/> </xsd:simpleType> <xsd:simpleType> <xsd:restriction base="intRange3"/> </xsd:simpleType> </xsd:union> </xsd:simpleType> !-- Union of anonymous types being local restrictions <xsd:simpleType name="intRangeLocalRestrictions"> <xsd:union> <xsd:simpleType> <xsd:restriction base="xsd:int"> <xsd:minInclusive value="0"/> <xsd:maxInclusive value="100"/> </xsd:restriction> </xsd:simpleType> <xsd:simpleType> <xsd:restriction base="xsd:int"> <xsd:minInclusive value="200"/> <xsd:maxInclusive value="300"/> </xsd:restriction> </xsd:simpleType> <xsd:simpleType> <xsd:restriction base="xsd:int"> <xsd:minInclusive value="400"/> <xsd:maxInclusive value="500"/> </xsd:restriction> </xsd:simpleType> </xsd:union> </xsd:simpleType> Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 6

Errors in Running VirtualXML Example
by Jyotishman Pathak 10 Mar '09

10 Mar '09

Hello all, I have been trying to play with virtualxml sample examples, and in particular, the example about Binary Data (cionverting the byte sequence as XML data). Now, if I try to run the example usign the "some-binary.data" file provided with the VirtualXML release, everything works fine. However, if I try to generate this data file using the C program ("some-binary.c") provided with the release, I get the following exception: Exception in thread "main" java.lang.RuntimeException: com.ibm.xml.xq.virtual.dfdl.DFDLException: XX0000: unknown (Could not complete contents.) at com.ibm.xml.xq.virtual.dfdl.DFDLInputMediator.buildChild(DFDLInputMediator.java:473) at com.ibm.xml.xq.virtual.dfdl.DFDLInputMediator.buildNext(DFDLInputMediator.java:154) at com.ibm.xml.xq.dm.cache.CacheManager.getCachedFollowingSibling(CacheManager.java:403) at com.ibm.xml.xq.dm.cache.dom.DOMCachedNode.getCachedFollowingSibling(DOMCachedNode.java:235) at com.ibm.xml.xq.dm.cache.CacheFocus.toNext(CacheFocus.java:420) at com.ibm.xml.xq.dm.util.Copier.copySequence2Content(Copier.java:209) at com.ibm.xml.xq.dm.util.Copier.copyContents(Copier.java:279) at com.ibm.xml.xq.dm.util.Copier.copySequence2Content(Copier.java:214) at com.ibm.xml.xq.dm.util.Copier.copyContents(Copier.java:279) at com.ibm.xml.xq.dm.util.Copier.copySequence2Content(Copier.java:206) at com.ibm.xml.xq.dm.util.Copier.copy(Copier.java:119) at com.ibm.xml.xq.dm.util.Copier.copy(Copier.java:71) at com.ibm.xml.xq.processor.Run.output(Run.java:442) at com.ibm.xml.xq.processor.Run.run(Run.java:408) at com.ibm.xml.xq.virtual.Virtual.main(Virtual.java:52) Caused by: com.ibm.xml.xq.virtual.dfdl.DFDLException: XX0000: unknown (Could not complete contents.) at com.ibm.xml.xq.virtual.dfdl.DFDLInputMediator.buildChild(DFDLInputMediator.java:469) ... 14 more I have tried executing this example in both windows and linux platforms, but with no success. Could you please assist me? Thanks, - Jyoti

2 1

Agenda for OGF WG call 11 March 2009
by Alan Powell 10 Mar '09

10 Mar '09

Suggested agenda 1. dfdl format specializations 2. Escape Schemes 3. Consuming extraneous data after schema has be used. 4. 'Floating' definitions 5. Validation ranges and exclusion lists 6 Actions 7 AOB Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: dfdl:sequence and other specialized annotations
by Steve Hanson 09 Mar '09

09 Mar '09

Although it was minuted in last week's call that we decided to drop specialised annotations, I don't actually recall that was decision on the call. The proposal below is in accordance with the suggestion by Martin, and echoed by Dave, that we have only specialised annotations for xsd objects. Schema object DFDL annotation allowed Notes Schema declaration dfdl:format Only within an enclosing dfd:defineFormat. Element declaration dfdl:element Just those properties allowed on elements Element reference dfdl:element Just those properties allowed on elements. Overrides dfdl:element properties on global element Complex type definition dfdl:complexType Full dfdl:format properties. For scoping purposes only. Simple type definition dfdl:simpleType Just those properties allowed on simple types. Sequence declaration dfdl:sequence Just those properties allowed on sequences. Choice declaration dfdl:choice Just those properties allowed on choices. Group reference dfdl:group Just those properties allowed on sequence/choice. Overrides dfdl:sequence/dfdl:choice properties on global group sequence/choice model group An alternative as I previously wrote is to allow dfdl:sequence and dfdl:choice on a group reference. Group reference (to sequence group) dfdl:sequence Just those properties allowed on sequence/choice. Overrides dfdl:sequence properties on global group model sequence group Group reference (to choice group) dfdl:choice Just those properties allowed on sequence/choice. Overrides dfdl:choice properties on global group model choice group Although this does not look as good visually, it is worth bearing in mind section 22.2 of the spec: 22.2 Group Reference A copy of the definition referenced by a group reference is substituted inline for its declaration while combining any DFDL format annotation elements associated with it as given in Section ‎10.5. (TBD: xref to scoping?) Processing then proceeds as for an anonymous group declaration. Note: The use of specialised annotations relies on the removal of something that we had talked about but not yet reflected in the spec - dfdl:format on global xsd:group definitions. The idea of allowing this came about fairly recently when we changed the rules for hidden objects. I was unhappy that embedded hidden xsd objects within DFDL annotations would make them invisible to a schema validator, so dfdl:hidden was changed to be a reference to a global group containing the hidden objects. I suggested that we could also put scoping annotations on global groups to support scoping within the hidden construct, it seemed a good idea at the time, but it was not fully explored. Thinking it through, because the global group effectively disappears when the reference is resolved, then so does its annotations. I'm therefore happy to retract scoping annotations on global groups if it makes specialised annotations work. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia <kalia(a)ca.ibm.com> Cc: dfdl-wg(a)ogf.org Date: 02/03/2009 23:45 Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Suman A DFDL annotation on a group reference overrides a DFDL annotation on the content of the model group (ie, sequence, choice), not a DFDL annotation on the global group. It allows you to override the sequence/choice properties at point of use. The DFDL annotation on a global group is like that on a complex type - it is a scoping construct only. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Suman Kalia <kalia(a)ca.ibm.com> 02/03/2009 22:02 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org, "Dave Glick" <dglick(a)dracorp.com> Subject Re: [DFDL-WG] Reducing the number of DFDL properties. I can see where you are coming from and I agree with most of your proposal except putting dfdl:sequence, dfdl:choice , dfdl:all on group references as it does not give the right optics.. From consistency point of view, the references (group and element references) should have the equivalent/same set of annotations that can appear on the referenced artifacts. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com From: Steve Hanson <smh(a)uk.ibm.com> To: Suman Kalia/Toronto/IBM@IBMCA Cc: dfdl-wg(a)ogf.org, "Dave Glick" <dglick(a)dracorp.com> Date: 03/02/2009 12:45 PM Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. If I recall, one reason for having the specialised elements was so that the content of the annotations could be more tightly validated by the schema-for-dfdl xsd. However, the problem with the specialised annotations is that we do not provide them for all circumstances. For example, the spec allows me to use dfdl:sequence on an xs:sequence but not an xs:group reference to a global xs:group with compostion xs:sequence. Nor does it allow me to use a specialised element on xs:simpleType or xs:restriction. So as it stands, validation would not be fool-proof anyway. I think we must either ditch specialisations, or mandate the use of specialisations on the corresponding schema objects. The latter is more workable now we have changed the scoping rules so that scoping of dfdl properties can only be done from complex type or schema level, as below. But it does mean that dfdl annotations are less flexible. dfdl;defineFormat => dfdl:format xs:complexType => dfdl:format xs:group => dfdl:format xs:sequence or xs:group ref to a global group with xs:sequence => dfdl:sequence xs:choice or xs:group ref to a global group with xs:choice => dfdl:choice xs:element or xs:element ref => dfdl:element xs:any => dfdl:any xs:simpleType, xs:restriction => dfdl:simpleType In other words, you use dfdl:format when you are scoping properties, you must use specialised annotation for specific objects. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Suman Kalia <kalia(a)ca.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 02/03/2009 16:25 To "Dave Glick" <dglick(a)dracorp.com> cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Subject Re: [DFDL-WG] Reducing the number of DFDL properties. I am not sure what kind of confusion or redundancy is caused by these specialized annotations. In the absence of these specialized annotations, you will have to go through plethora of annotations and determine which ones are applicable for sequence , choice, all, elements etc.. The danger is we don't have these levels of abstractions, then number of folks (specifically implementers) would build them anyway and then we will have to contend with incompatible abstractions.. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com From: "Dave Glick" <dglick(a)dracorp.com> To: "Alan Powell" <alan_powell(a)uk.ibm.com>, <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org Date: 03/02/2009 11:16 AM Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Alan/Mike, I agree with this – one of my complaints with v033 was that these specialized annotation elements just added confusion and redundancy. Dave --- David Glick | dglick(a)dracorp.com | 703.299.0700 x212 Data Research and Analysis Corp. | www.dracorp.com From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Alan Powell Sent: Monday, March 02, 2009 11:03 AM To: mbeckerle.dfdl(a)gmail.com Cc: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] Reducing the number of DFDL properties. Mike Thanks Can I also suggest dropping the dfdl:sequence, dfdl:choice, dfdl:element and dfdl:any specialized annotations and just have dfdl:format. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Alan Powell/UK/IBM@IBMGB Cc: <dfdl-wg(a)ogf.org> Date: 28/02/2009 19:38 Subject: RE: Reducing the number of DFDL properties. If a property is redundant, I'm in favor of dropping it. If a property adds generality that we don't have a use case for, I'm in favor of dropping it. I am happy to drop type substitution from v1.0. It's a convenience that can be achieved a different way. E.g., if you really want "float" to mean "float in my particular binary representation", then just put a type definition with DFDL annotations in a different namespace, and when you write your DFDL schema, arrange for the default namespace to pick it up from your namespace, and not xs:float. <xs:element name="myElement" type="float"/>  <xs:element name="anotherElem" type="xs:float"/>  If you want the XSD unadorned "float" type, be explicit and use "xs:float". Voila - no loss of flexibility, equal textual convenience. I think that would satisfy the community that really wanted very compact "slideware acceptable" schemas. This is the same group that wants short-form annotations as well. Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 100 Fifth Ave., 4th Floor, Waltham MA 02451 | mbeckerle.dfdl(a)gmail.com From: Alan Powell [mailto:alan_powell@uk.ibm.com] Sent: Thursday, February 26, 2009 12:22 PM To: mbeckerle.dfdl(a)gmail.com Cc: dfdl-wg(a)ogf.org Subject: Reducing the number of DFDL properties. Mike A number of people at IBM have become concerned at the number of properties in DFDL and have identified a number of 'usability' properties that could be dropped. They feel that we should be simplifying the properties wherever possible and not introducing multiple ways of doing the same function without very good reason. The following are offered for consideration. 1. lengthKind='nullterminated' This is just shorthand for lengthKind=delimited and terminator='%Null'. It was felt this this is not even the most common terminator so why have a special case? 2. trimKind It is felt that there aren't any cases when you would want to pad but not trim and vice versa so make padKind control both. 3. typeSubstitution. Is this needed in DFDL v1? Can you consider these before the call next week Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL Working Group weekly call - summer time
by Steve Hanson 09 Mar '09

09 Mar '09

Please be aware that in North America, summer time starts this week which is 3 weeks earlier than in UK. The call is therefore at 10am Eastern time for next 3 weeks. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, March-04-2009
by Alan Powell 05 Mar '09

05 Mar '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, March-04-2009 Attendees Mike Beckerle(Oco) Alan Powell (IBM) Steve Hanson (IBM) Tim Kimber (IBM) Apologies Dave Glick (drac) 1. XML Schema 1.1 It was agreed that the lack of support for Schema 1.1 in tools, libraries etc would be an inhibitor to DFDL implementations so it was agreed to remain on XML schema 1.0 . SH highlighted the main isuues with staying on XSD 1.0 1) xs:all improvements - relaxing min/max occurs, allowing wildcards, I don't see an alternative to our current proposal of dfdl:sequenceKind. 2) weakened wildcard Either live with this restriction, or say that the DFDL parser will implement the 1.1 rule where an optional element takes precedent over a wildcard and say that if you use this feature of DFDL, you should avoid XML schema tools that implement this UPA check. (It is my understanding that many tools that validate XML Schema for correctness do not include the UPA checks). These issues need to be resolved. It was noted thet Schema 1.1 assert only allow reference to children of the item with the assert and that it would be desirable if DFDL was compatible. As there are dfdl use cases were reference to earlier fields is required AP to investigate if assigning earlier fields to variables in sufficient. 2. DFDL;ASSERT and DFDL:DISCRIMINATOR Discussion to clarify the semantics of assert and discriminator. Decided that dfdl:discriminator should only be used to resolve choices. TK to investigate if there are other use cases where discriminator would be useful. dfdl:assert is to ensure that a document is well formed which, while possibly applying similar tests, is separate from validation. Confirmed that assert cannot be turned off. AP to document semantics. 3. Reducing number of properties General agreement to reduce properties when they are redundant and don't support a common use case. LengthKind=NullTerminated, typeSubstitution, and trimkind.will be removed. dfdl:format specialization such as dfd;:sequence were also discussed. It had been pointed out in emails that there are two stable options - Only dfdl:format with no specializations - Only specializations and no general dfdl:format. Supporting both dfdl:format and specialization was felt to be undesirable. SH point out that the second option was not practical for all constructs as complex type definitions which put properties in scope would require all possible properties. I was agreed that for dfdl v1 the specializations would be dropped. 4. AOB To be discussed - Escape Schemes - Consuming extraneous data after schema has be used. - 'Floating' definitions - Validation ranges and exclusion lists - Actions Next call 11 March 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action 033 04/03: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 034 04/03: AP remove redundant properties Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 023 MB: Review Schema 1.1 29/1: AP and SH to talk to Sandy Gao 04/02 Call arranged for Friday 11/02: Call took place. Identified useful changes. Consolidate with previous list. 04/03: decided to stay on Schema 1.0. 024 String XML type 025 Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 18/02: SH up document concerns 25/02: SH and AP have refined proposal ready for approval. 04/03: SH and AP have further refined proposal. 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 031 DG: Review dfdl v033 11/02: Initial comments received 18/02: Will include work items 5 and 12. 032 DG: Investigate compatibility between DFDL infoset and XDM 033 04/03: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 034 04/03: AP remove redundant properties Closed actions: 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work 026 Remove duration complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0