- dfdl-wg - lists.ogf.org

Minutes for OGF DFDL Working Group Call, May-06-2009
by Alan Powell 06 May '09

06 May '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, May-06-2009 Meeting opened, 14:00 UK Attendees Steve Hanson (IBM) Mike Beckerle (Oco) Suman Kalia (IBM) Alan Powell (IBM) Apologies Dave Glick (drac) Agenda: 1. Go through actions. Actions updated below 2. LengthKind on Sequences and choices. Agreed with proposal to remove lengthKind from sequences and choices. Discussed whether other properties such as initiator and terminator should also be moved but decided it would introduce too many elements and make the resultant tree deep. Will keep open until SH completes property precedence and will then look at all sequence and choice properties 3. Discuss UnorderedInitated email from SH Agreed that it was useful to indicate that children of sequences are initiated to enable more helpful error messagies to be generated. Agreed to generalize to all sequences not just unordered. 4. Infoset codepage and encoding Decided simplest solution is to pick UTF-16 5. AOB Next version (034) will be available for review on Friday Next call 13 May 14:00 UK Meeting closed, 15:10 UK Actions raised at this meeting No Action Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 08/04: Not discussed 22/04: AP to complete asap once the ICU strict/lax behaviour is understood. 29/04: No progress 06/05: No progress 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 08/04: Not discussed 22/04: SH to complete last remaining issue, which is the behaviour when logical type is signed/unsigned and the physical type is unsigned/signed. 29/04: SH had identified a problem with definition values and types in the infoset and will email proposal. DG to be asked to accelerate action 032 to see if helps 06/05: No progress 026 SH: Envelopes and Payloads 08/04: Not discussed explicity, but recursive use of DFDL is tied up with this 22/04: Two aspects. Firstly compositional - do sufficient mechanisms exist to model an envelope with a payload that varies. Secondly markup syntax - this might be defined in the envelope. The second of these is very much tied up with the variable markup action 028, so will be considered there. SH to verify the composition aspect. 29/04: SH and AP working on proposal. related to Action 028 06/05: No progress 027 SH: Property precedence tables 08/04: Not discussed 22/04: Two things missing from the existing precedence trees. Firstly, does not show alternates (eg, initiator v initiatorkind). Secondly, need a tree per concrete DFDL object (eg, element). SH to update. 29/04: No progress 06/05: SH is updating tables which will be ready for next call 028 SH: Variable markup 08/04: Discussed briefly at end of call, IBM to see whether there any use cases that require recursive use of DFDL. 15/04: Use case was distributed and will be discussed on next call. 22/04: The use case in question is EDI where the terminating markup for the payload segments is defined in the ISA envelope segment. The markup is modelled as an element of simple type where the allowable markup values are defined as enums on the type. But we need to handle two cases - firstly where the envelope is present, so the value used by the payload is taken from the envelope. Secondly where only the payload is present. Here we need a way of scanning for all the enum values, and adopting the one we actually find, when parsing. And using a default when unparsing. SH to explore use of a DFDL variable, where the variable has a default, but also has a type that is the same as the markup element - that way we get to use the enums without defining everything twice. 29/04: SH and AP working on proposal. 06/05: No progress 029 MB: valueCalc (output length calculation) 08/04: Not discussed 22/04: Action allocated to MB, this is to complete the work started at the Hursley WG F2F meeting. 29/04: No progress 06/05: MB will have update for next call 032 DG: Investigate compatibility between DFDL infoset and XDM 08/04: No update 22/04: No update 29/04: No update 06/05: DG indicates will have update next week 033 AP/TK: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 08/04: In progress within IBM 22/04: Waiting for TK to return from leave to complete. 29/04: TK has sent examples shown need for discriminators beyond choice. Agreed. MB to respond to TK 06/05: Discussed suggestion of adding type indicator to discriminator. MB to provide examples. 036 SH: Provide use case for floating component in a sequence 08/04: Raised 15/04: Use case sent and discussed. SH to do further investigation 22/04: IBM feedback from WTX team is that alternate suggested ways of modelling the EDI floating NTE segment have significant usability issues. The DFDL principle is that for a problem that can be expressed as two-layered, then two DFDL models are needed. The EDI NTE segment does not fall into this though, as its use is on a per sequence basis. Ongoing. 29/04: Agreed that need to be in V1. SH to make a proposal 06/05: No progress 037 All: Approach for XML Schema 1.0 UPA checks. 22/04: Several non-XML models, when expressed in their most obvious DFDL Schema form, would fail XML Schema 1.0 Unique Particle Attribution checks that police model ambiguity. And even re-jigging the model sometimes fails to fix this. Note this is equally applicable to XMl Schema 1.1 and 1.0. While the DFDL parser/unparser can happily resolve the ambiguities, the issue is one of definition. If an XSD editor that implements UPA checks is used to create DFDL Schema, then errors will be flagged. DFDL may have to adopt the position that: a)DFDL parser/unparser will not implement some/all UPA checks (exact checks tbd) b) XML Schema editors that implement UPA checks will not be suitable for all DFDL models c) If DFDL annotations are removed, the resulting pure XSD will not always be valid (ie, the equivalent XML is ambiguous and can't be modelled by XML Schema 1.0) Ongoing in case another solution can be found. 29/04: Will ask DG and S Gao for opinion before closing 06/05: Discussed S Gao email and suggestions. Decided need to review all XML UPA rules and decide which apply to dfdl. 038 MB: Submit response to OMG RFI for non-XML standardization 22/04: First step is for MB to mail the OGF Data Area chair to say that we want to submit 29/04: MB has been in contact with OMG and will sunbit dfdl. 06/05: MB has prepared response to OMG. Will send DFDL sepc v033 039 SKK: Approach for creating Schema-For-DFDL xsds. 22/04: Resolve issue around multiple declarations needed for DFDL properties, perhaps using MB's meta approach 29/04: Don't like qualified attributes in long form. SKK to check there are no code gen implications, eg EMF. 06/05: SKK will send update by Friday 040 SH: LengthKind on complex objects. 29/04: All send comment before next call 06/05: See minutes. Agreed to remove lengthKind from sequence and choice. 041 AP: UnorderedInitiated 29/04: All: Review for next call 06/05: See minutes: Agreed to generalize to all sequences 042 MB: Complete variable specification. To include how properties such as encoding can be set externally. Must be a known variable name. 06/05: No progress Closed actions: 024 <No owner> String XML type 08/04: Not discussed 22/04: Need to allocate owner. Work is to describe the semantics of using dfdl:representation="xml" to model a well-formed XML fragment in an overall non-XML document described by a DFDL schema. 29/04: As no resource availbel to progress this action agreed to defer from V1. Will close next week if no objections 06/05: closed Work items: No Item target version status 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 034 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 (A033) 034 in progress 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) not started 025 Augmented infoset and unparsing (Alan) 034 added but needs work 026 Remove duration 034 027 Calendar schemes 034 028 Validation ranges (A035) 034 029 Decimals (A020) - document unsigned type behaviour - packedDecimalSignCodes behaviour depends on NumberCheckPolicy 034 030 Remove redundant properties, fix examples. (A036) 034 031 Specialized annotations 034 032 Floating components 033 Specialized annotations 034 Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Minutes for OGF DFDL Working Group Call, April-29-2009
by Alan Powell 05 May '09

05 May '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, April-29-2009 Meeting opened, 14:00 UK Attendees Steve Hanson (IBM) Mike Beckerle (Oco) Suman Kalia (IBM) Alan Powell (IBM) Apologies Dave Glick (drac) Agenda: 1. Go through actions. Actions updated below 2. LengthKind on Sequences and choices. LengthKind on sequences and choices and their parent element has proved confusing to new users of DFDL. It is proposed that lengthKind is removed from groups and only allow it to be set on parent element. See email from SH Not discussed. Action raised. Please review SH email and make comments before next call 3. Discuss UnorderedInitated email from SH Not discussed. Action raised. Please review SH email and make comments before next call 4. Infoset codepage and encoding The spec does not say what codepage and encoding is used for string fields. 5. AOB Next version (034) 6. Next call 6 May 14:00 UK Meeting closed, 15:10 UK Actions raised at this meeting No Action 040 SH: LengthKind on complex objects. 29/04: All send comment before next call 041 AP: UnorderedInitiated 29/04: All: Review for next call 042 MB: Complete variable specification. To include how properties such as encoding can be set externally. Must be a known variable name. Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 08/04: Not discussed 22/04: AP to complete asap once the ICU strict/lax behaviour is understood. 29/04: No progress 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 08/04: Not discussed 22/04: SH to complete last remaining issue, which is the behaviour when logical type is signed/unsigned and the physical type is unsigned/signed. 29/04: SH had identified a problem with definition values and types in the infoset and will email proposal. DG to be asked to accelerate action 032 to see if helps 024 <No owner> String XML type 08/04: Not discussed 22/04: Need to allocate owner. Work is to describe the semantics of using dfdl:representation="xml" to model a well-formed XML fragment in an overall non-XML document described by a DFDL schema. 29/04: As no resource availbel to progress this action agreed to defer from V1. Will close next week if no objections 026 SH: Envelopes and Payloads 08/04: Not discussed explicity, but recursive use of DFDL is tied up with this 22/04: Two aspects. Firstly compositional - do sufficient mechanisms exist to model an envelope with a payload that varies. Secondly markup syntax - this might be defined in the envelope. The second of these is very much tied up with the variable markup action 028, so will be considered there. SH to verify the composition aspect. 29/04: SH and AP working on proposal. related to Action 028 027 SH: Property precedence tables 08/04: Not discussed 22/04: Two things missing from the existing precedence trees. Firstly, does not show alternates (eg, initiator v initiatorkind). Secondly, need a tree per concrete DFDL object (eg, element). SH to update. 29/04: No progress 028 SH: Variable markup 08/04: Discussed briefly at end of call, IBM to see whether there any use cases that require recursive use of DFDL. 15/04: Use case was distributed and will be discussed on next call. 22/04: The use case in question is EDI where the terminating markup for the payload segments is defined in the ISA envelope segment. The markup is modelled as an element of simple type where the allowable markup values are defined as enums on the type. But we need to handle two cases - firstly where the envelope is present, so the value used by the payload is taken from the envelope. Secondly where only the payload is present. Here we need a way of scanning for all the enum values, and adopting the one we actually find, when parsing. And using a default when unparsing. SH to explore use of a DFDL variable, where the variable has a default, but also has a type that is the same as the markup element - that way we get to use the enums without defining everything twice. 29/04: SH and AP working on proposal. 029 MB: valueCalc (output length calculation) 08/04: Not discussed 22/04: Action allocated to MB, this is to complete the work started at the Hursley WG F2F meeting. 29/04: No progress 032 DG: Investigate compatibility between DFDL infoset and XDM 08/04: No update 22/04: No update 29/04: No update 033 AP/TK: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 08/04: In progress within IBM 22/04: Waiting for TK to return from leave to complete. 29/04: TK has sent examples shown need for discriminators beyond choice. Agreed. MB to respond to TK 036 SH: Provide use case for floating component in a sequence 08/04: Raised 15/04: Use case sent and discussed. SH to do further investigation 22/04: IBM feedback from WTX team is that alternate suggested ways of modelling the EDI floating NTE segment have significant usability issues. The DFDL principle is that for a problem that can be expressed as two-layered, then two DFDL models are needed. The EDI NTE segment does not fall into this though, as its use is on a per sequence basis. Ongoing. 29/04: Agreed that need to be in V1. SH to make a proposal 037 All: Approach for XML Schema 1.0 UPA checks. 22/04: Several non-XML models, when expressed in their most obvious DFDL Schema form, would fail XML Schema 1.0 Unique Particle Attribution checks that police model ambiguity. And even re-jigging the model sometimes fails to fix this. Note this is equally applicable to XMl Schema 1.1 and 1.0. While the DFDL parser/unparser can happily resolve the ambiguities, the issue is one of definition. If an XSD editor that implements UPA checks is used to create DFDL Schema, then errors will be flagged. DFDL may have to adopt the position that: a)DFDL parser/unparser will not implement some/all UPA checks (exact checks tbd) b) XML Schema editors that implement UPA checks will not be suitable for all DFDL models c) If DFDL annotations are removed, the resulting pure XSD will not always be valid (ie, the equivalent XML is ambiguous and can't be modelled by XML Schema 1.0) Ongoing in case another solution can be found. 29/04: Will ask DG and S Gao for oppinion before closing 038 MB: Submit response to OMG RFI for non-XML standardization 22/04: First step is for MB to mail the OGF Data Area chair to say that we want to submit 29/04: MB has been in contact with OMG and will sunbit dfdl. 039 SKK: Approach for creating Schema-For-DFDL xsds. 22/04: Resolve issue around multiple declarations needed for DFDL properties, perhaps using MB's meta approach 29/04: Don't like qualified attributes in long form. SKK to check there are no code gen implications, eg EMF. Closed actions: 025 AP: Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 18/02: SH up document concerns 25/02: SH and AP have refined proposal ready for approval. 04/03: SH and AP have further refined proposal. 11/03: discussed. suggested a simplified proposal be evaluated. 18/03: SH and AP had further discussions on simplified proposal 08/04: See minutes, review in detail for next call 15/04: See minutes, review for next call 22/04: MB mailed answers to the mailing list in response to AP's last few questions. Following agreed: 1.Should data containing the escapeEscapeCharacater cause escaping to be used if if so how should it be escaped. EEC alone isn't an active character. it has to be followed by the EC to be interpreted at all. That said, if the pair EEC EC appears in the data, then yes, we must escape the EC, with another EEC. 2.Should we only look for escapeStartString at the beginning of the data Yes, we will be restrictive/conservative for v1.0 3.Property names (everyone has their own favourite so lets just pick one.) Only changes areescapeBlockStart and escapeBlockEnd. AP to incorporate the agreed scheme into draft 0.34. 29/04: closed. Moved to workitems 034 AP: Remove redundant properties, correct old examples 08/04: No update 22/04: In progress as part of draft 0.34. 29/04: closed. Moved to work item 035 AP: Add validation ranges to spec, update specialized annotations in spec. 08/04: Raised. For draft 0.34 22/04: In progress as part of draft 0.34. 29/04: closed. Moved to work item Work items: No Item target version status 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 034 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 (A033) 034 in progress 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) not started 025 Augmented infoset and unparsing (Alan) 034 added but needs work 026 Remove duration 034 027 Calendar schemes 034 028 Validation ranges (A035) 034 029 Decimals (A020) - document unsigned type behaviour - packedDecimalSignCodes behaviour depends on NumberCheckPolicy 034 030 Remove redundant properties, fix examples. (A036) 034 031 Specialized annotations 034 032 Floating components 033 Specialized annotations 034 Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 5

Fw: UnorderedInitiated
by Steve Hanson 05 May '09

05 May '09

It has been raised that the dfdl:unorderedInitiated property is redundant, because we have speculative parsing, and because the DFDL parser can optimise by analysis without being explicitly told. This is true but saying that all children of a sequence have an initiator is also a way for the modeler to ensure that his model is correct and that all children are correctly specified. I propose that we decide whether to: a) Drop the property b) Rename the property dfdl:initiated and have it apply to any sequence, ordered or unordered There's also a question in the spec that we should resolve. (TBD: do we allow sequences with initiators to be children of an unordered sequence, or do we require the children of an unordered sequence to be elements? Conservative would be to require elements.) To which Mike had commented: Suggest sequences where children have initiators cannot be directly nested. You must use an element in this case. I would prefer that children of an unordered sequence are required to be elements in general for v1.0. I'd like to close on this as well. Let's discuss on today's call. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 29/04/2009 10:38 ----- Steve Hanson/UK/IBM 28/04/2009 15:41 To Tim Kimber/UK/IBM cc Alan Powell/UK/IBM@IBMGB Subject Re: UnorderedInitiated Here's the table from draft 0.33: sequenceKind separatorPolicy Implications unorderedInitiated ignored (suppress behavior implied) Initiators are used to identify which elements are present and which are missing. All children must have dfdl:initiator strings that are distinct and not the empty string. (Schema definition error otherwise.) nilValueInitiatorPolicy and defaultValueInitiatorPolicy must both be 'required' (Schema definition error otherwise) Setting unorderedInitiated forces extra checks to take place to ensure that an initiator would be present under all circumstances. I think this provides a direct equivalent to TDS Tagged. I'd prefer that speculative parsing was the sole mechanism used to resolve the child elements. In practice, most unordered sequences will have initiators on their children anyway. There's also this comment in the spec: (TBD: do we allow sequences with initiators to be children of an unordered sequence, or do we require the children of an unordered sequence to be elements? Conservative would be to require elements.) To which Mike had commented: Suggest sequences where children have initiators cannot be directly nested. You must use an element in this case. I would prefer that children of an unordered sequence are required to be elements in general for v1.0. Which corresponds to schema's rule for xs:all and the MRM's rule for unorderedSet, so I'm happy with that. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Tim Kimber/UK/IBM 15/04/2009 23:26 To Alan Powell/UK/IBM@IBMGB cc Steve Hanson/UK/IBM@IBMGB Subject Re: UnorderedInitiated Hi Alan, Understood. In that case, please can you add this to the list of unresolved issues in the specification. We should take pains not to include redundant properties in DFDL v1.0, because each property has a cost in terms of implementation, testing and documentation. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Alan Powell/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 15/04/2009 09:37 Subject: Re: UnorderedInitiated Tim The problem with committees is that you can't just decide by yourself to drop something. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Tim Kimber/UK/IBM To: Alan Powell/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB, Dragan Besevic/Boca Raton/IBM@IBMUS, Sunil Dandamudi/Boca Raton/IBM@IBMUS, Lorenzito Jimenez/Boca Raton/IBM@IBMUS Date: 14/04/2009 21:19 Subject: Re: UnorderedInitiated Alan, Thanks for the clarification. If that is the only reason for the property then it should be removed in v0.34. The PIF generator can easily perform that kind of optimisation ( if necessary ). This is not the only optimisation which the PIF generator might want to perform, and the specification does not supply extra properties for those other cases. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Alan Powell/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 14/04/2009 17:23 Subject: Re: UnorderedInitiated Tim There is a history. You are correct that speculative parsing should take care of it all. However originally unordered elements had to be initiated. When that was relaxed it was felt that it was worth giving the parser a chance to optimize to initiated case. Designed by committee. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Alan Powell/UK/IBM@IBMGB Date: 08/04/2009 22:54 Subject: UnorderedInitiated I can't see why this is required, but I'm probably missing something. Wouldn't the normal speculative parsing rules will make use of the initiator if it is present. Or does unorderedInitiated cause the initiator to behave as a discriminator? If so, I would prefer to leave that to the user. We don't do it for any other point of uncertainty ( there's no equivalent choiceInitiated setting ). regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Action 033 : Assertions and discriminators
by Tim Kimber 05 May '09

05 May '09

Hi all, I have this action: 033 04/03: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. I believe the rules should be: 1. A point of uncertainty is any of * an element which has minOccurs != maxOccurs * a choice * a sequence with sequenceKind="unordered" 2. Nested within the scope of a point of uncertainty, there might be other points of uncertainty. 3. A discriminator which evaluates to true resolves the nearest in-scope point of uncertainty. 4. An assertion which evaluates to false causes a processing error 5. Any processing error ( from an assertion failure or otherwise ) will cause the parser to backtrack to the nearest unresolved point of uncertainty and try the next available branch, if any. If there are no more branches available, the parser will backtrack to the next nearest unresolved point of uncertainty. 6. A processing error which reaches the root tag is reported to the host application. 7. Assertions and discriminators are allowed on any point of uncertainty ( not only on the branches of a choice ) Rationale: If we only allow a discriminator on a choice branch, then it will be difficult to model this common style of message Tagged header, minOccurs="1", maxOccurs="1" Untagged body, maxOccurs="unbounded" Tagged trailer, minOccurs="1", maxOccurs="1" An example with 3 occurrences of the body would be: HE,headerfield1,headerfield2,headerfield3 John Smith, 100, bodyfield3 John Brown, 200, bodyfield3 Elton John, 30Z, bodyfield3 TR,trailerfield1,trailerfield2,trailerfield3 And the DFDL schema would look something like this ( excuse the almost inevitable errors, this is just for completeness ): ... <xs:element name="message"> <xs:complexType dfdl:lengthKind="implicit"> <xs:sequence dfdl:separator="\r\n"> <xs:element name="header" initiator="HE,"> <xs:complexType> <xs:sequence dfdl:separator=","> <xs:element name="header1" type="xs:string"> <xs:element name="header2" type="xs:string"> <xs:element name="header3" type="xs:string"> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="body" maxOccurs="unbounded"> <xs:complexType> <xs:sequence dfdl:separator=","> <xs:element name="body1" type="xs:string"> <xs:element name="body2" type="xs:int"> <xs:element name="body3" type="xs:string"> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="trailer" initiator="TR,"> <xs:complexType> <xs:sequence dfdl:separator=","> <xs:element name="trailer1" type="xs:string"> <xs:element name="trailer2" type="xs:string"> <xs:element name="trailer3" type="xs:string"> </xs:sequence> </xs:complexType> </xs:element> </xs:sequence> </xs:complexType> </xs:element> The '30Z' value for the final occurrence of element Body2 is incorrect. It is not a valid integer, and will trigger a processing error. Without a discriminator, this failure will cause the parser to backtrack to the optional field and try the next element ( the trailer element ). The initiator will not be found, and the reported error will be "Initiator 'TR,' not found for element 'trailer'". The user would almost certainly prefer "Invalid value '30Z' for element 'body2'. Value could not be converted to simple type 'xs:int'" For this example, the discriminator would need to detect unambiguously that it really was dealing with a Body element and not a Trailer element. Due to the message style ( which is quite common ) the only way to do this is to detect that it is *not* a Trailer. I cannot think of an elegant way to do that using the facilities in v0.33 of the specification. I have raised this with Alan and Steve. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

DFDL lengthKind on complex objects
by Steve Hanson 05 May '09

05 May '09

I have a long-standing concern about the usability of dfdl:lengthKind, which others in IBM are encountering when modeling real life formats such as EDI. My main concern is below. For example, what's the semantic of setting different values on the element and the sequence? <xs:element name="container" dfdl:lengthKind="implicit"> <xs:complexType> <xs:sequence dfdl:separator="@" dfdl:lengthKind="implicit"> <xs:element name="one" type="xs:string" dfdl:lengthKind="delimited" /> <xs:element name="two" type="xs:string" dfdl:lengthKind="delimited" /> <xs:element name="three" type="xs:string" dfdl:lengthKind="delimited" /> </xs:sequence> </xs:complexType> </xs:element> It gets even more noticeable if I set a scoping dfdl:lengthKind on the complex type. I propose that we limit dfdl:lengthKind to elements only. It means that the length of a xs:sequence or xs:choice is always and implicitly given by its chidren, and if you want to provide an explicit length or a length prefix you must use a complex element to wrap the sequence or choice. We have looked at the implications on dfdl:choiceKind for choices, and dfdl:occursKind on arrays, and the proposal works happily in those scenarios. There's an analogy here with not alowing sequences and choices to repeat, only elements. It also simplifies the grammar, in the sense that any excess fill characters in a 'box' are always considered part of the element when parsing. I'd like to discuss this on today's call. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Agenda for OGF WG call 1 April March 2009
by Alan Powell 29 Apr '09

29 Apr '09

Agenda: 1. Go through actions. 2. LengthKind on Sequences and choices. LengthKind on sequences and choices and their parent element has proved confusing to new users of DFDL. It is proposed that lengthKind is removed from groups and only allow it to be set on parent element. See email from SH 3. Discuss UnorderedInitated email from SH 4. Infoset codepage and encoding The spec does not say what codepage and encoding is used for string fields. 5. AOB Next version (034) Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Go Meta - RE: DFDL Schema - Mulitple forms of Annotation syntax
by Mike Beckerle 29 Apr '09

29 Apr '09

These difficulties, all seem to arise from the intellectual objective to leverage XML Schema validation against DFDL schema annotations maximally. Given the 3 syntaxes, this is a bit painful if done directly. I suggest the following approach. Write a "Meta XML Schema" for DFDL annotations. This XML document would contain exactly one list of all the DFDL annotations. E.g., something like:  <dfdl:defineProperty name="initiator" allowedOn="complexType sequence element" type="string" expressionAllowed="yes"/> <dfdl:defineProperty name="terminator" allowedOn="complexType sequence element" type="string" expressionAllowed="yes"/> ... This document could be turned into the set of redundant annotations on a "real" XML Schema for DFDL annotations by a XSL stylesheet translation. This would actually be a pretty easy XSL stylesheet. Each dfdl:defineProperty in the meta schema would end up putting content into the big enum, a local and global attribute definition, and anywhere else needed in the "real" XML Schema for DFDL annotations. I would suggest that doc strings might also be good to put in: e.g., <dfdl:defineProperty name="initiator" allowedOn="complexType sequence element" type="string" expressionAllowed="yes"> Defines a string used to recognize the beginning of a construct..blah blah blah </dfdl:defineProperty> This could then be used to generate online doc also. I always use this sort of "go meta" approach whenever I have redundant, but isomorphic things happening in code. I always create a spec and generate the redundant instances from that. This helps tremendously with QA, because whole classes of inconsistency errors are ruled out. ...mike 1. Shot form syntax <xs:element name="foo" type="xs:int" maxOccurs="unbounded" dfdl:representation="text" dfdl:initiator="{" dfdl:terminator="}"/> Note: It requires all properties to be defined as global attributes in schema as each attribute has to be uniquely identifiable and accessible within the DFDL namespace 2. Long form syntax ( used for dfdl:format and specialized annotations such as dfdl:element etc) <xs:element name="foo" type="xs:int" maxOccurs="unbounded"> <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format representation="text" initiator="{" terminator="}"/> </xs:appinfo></xs:annotation> </xs:element> Note: Attributes specified here are not qualified with dfdl namespace prefix. It means the attributes have to be defined locally within the complex type are not accessible directly (like in the short form syntax) 3. Long form property representation to ease syntactic expression difficulties with data in attribute values: <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format> <dfdl:property name='encoding'>utf-8</dfdl:property> <dfdl:property name='separator'>\n</dfdl:property> <dfdl:property name='initiator'><[CDATA[<!-- ]]></dfdl:property> </dfdl:format> </xs:appinfo> </xs:annotation> Note: The name attribute will be enumeration of base name of all DFDL properties (ie without namespace prefix). On the specialized annotations (ie dfdl:element) we will also need to have a repeating child element definition for dfdl:property. This annotation will appear on specialized annotation element for dfdl attributes whose value cannot be stored as XML attribute. PS: Alan - when you update the spec for specialized annotation please mention this use case and put an example also. To support syntax form 1 & 2, it requires defining all attribute definition for each DFDL property multiple times ie one global attribute and a local attribute in the complex type of each long form annotation element. Using Attribute groups we can address the latter issue ie define related local attributes once in the attribute group and include the attribute group in one or more complex type. But it still requires defining the attribute definition for each DFDL property atleast twice (ie one global and one local attribute definition). This duplication can potentially lead to inconsistency and inadvertently updating one definition and not the other. One option that we can consider to remove this redundancy is to define only the global attribute definition for each DFDL property and include it through attribute reference in each of the complex type of dfdl:format and specialized annotations. The downside of this approach is that attributes will appear qualified in the dfdl:format and specialized annotation elements. Given below is how the dfdl:format annotation will appear in the user defined schema where each attribute is prefixed with dfdl (hilited in green). <xs:element name="foo" type="xs:int" maxOccurs="unbounded"> <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format dfdl:representation="text" dfdl:initiator="{" dfdl:terminator="}"/> </xs:appinfo></xs:annotation> </xs:element> >From technical perspective (e.g. schema integrity , avoiding duplication definitions etc.), this is better but it does have usability issue. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools <http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h tml> http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.ht ml Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com

3 2

Minutes: OGF DFDL Working Group Call, April-22-2009
by Steve Hanson 23 Apr '09

23 Apr '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, April-22-2009 Attendees Steve Hanson (IBM) Mike Beckerle (Oco) Apologies Suman Kalia (IBM) Alan Powell (IBM) Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

Re: [DFDL-WG] Simplified Escape Scheme V3
by Alan Powell 22 Apr '09

22 Apr '09

Steve Thanks for the comment. Wording changes accepted. Can you discuss on this weeks WG call. I cannot attend unfortunately. We need to agree 1.Should data containing the escapeEscapeCharacater cause escaping to be used if if so how should it be escaped. 2.Should we only look for escapeStartString at the beginning of the data 3.Property names (everyone has their own favourite so lets just pick one.) Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Steve Hanson/UK/IBM To: Alan Powell/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 19/04/2009 12:24 Subject: Re: [DFDL-WG] Simplified Escape Scheme V3 Alan Comments: - I think escapeBlockStart and escapeBlockEnd are better names, that way you can immediately see they are for use with escapeBlock. - escapeKind. Clarification to escapeBlock parsing behaviour. "On parsing the escapeStartString is removed from the beginning of the data and escapeEndString is removed from end of the data and any escapeEscapeCharacters are removed when they precede any other occurences of the escapeEndString in the data." - extraEscapedCharacters. Clarification: "A space separated list of single characters that must be escaped in addition to in-scope markup" - generateEscape. The behaviour when escapeKind = escapeCharacter and value is 'always' is not defined. I would prefer that: a) The descriptions of 'whenNeeded' behaviour are moved into the escapeKind property to keep all the rules in one place. b) generateEscape is renamed generateEscapeBlock and only applies to escapeKind = escapeBlock, as that is only when it has an effect. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Alan Powell/UK/IBM@IBMGB Sent by: dfdl-wg-bounces(a)ogf.org 17/04/2009 15:22 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Simplified Escape Scheme V3 Attached is the latest version of escape schemes. It includes Steve and Mike's comments (although not renaming properties), removed escapeBlock2 and added uses cases in section 5 which you might like to start with. The uses cases confirm that the syntax works with some minor clarifications but highlights two questions: 1. Should data containing the escapeEscapeCharacater cause escaping to be used if if so how should it be escaped. 2. Should we only look for escapeStartString at the beginning of the data. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "ggf-dfdl-simplified-escape-scheme-v3.doc" deleted by Alan Powell/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

DFDL WG Call 2009-04-22: Agenda
by Steve Hanson 22 Apr '09

22 Apr '09

I propose a change to the way things are discussed on the call. To prevent build-up of actions, the first thing discussed on the call are the current set of actions, followed by other agenda items. The mailed agenda will contain the current actions in pdf form. Agenda: 1. Go through actions - see pdf 2. OGF 26 in Chapel Hill, NC 3. OMG non-XML/XML Information Transformation Standardization (RFI) - see below. 4. Short and long form properties require all dfdl properties to be defined at least twice in the schema for DFDL --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- OMG non-XML/XML Information Transformation Standardization (RFI) The Object Management Group (OMG)[1] is pursuing the standardization of non-XML/XML information transformation. The Middleware and Related Services (MARS) Platform Task Force (PTF)[2] Information Exchange Framework (IEF) Working Group[3] has released an Information Transformation Request for Information (RFI)[4] to collect concepts, participation and scope for creating and publishing the respective Request for Proposal (RFP). Responses to the RFI should consist of one electronic copy in machine-readable format (typically ASCII, MS Word, or WordPerfect format) sent to omg-documents(a)omg.org, and one paper copy sent to the OMG postal address below: Object Management Group, Inc. 140 Kendrick Street Building A Suite 300 Needham, MA 02494 USA Attn: Information Transformation Services RFI At the last meeting the RFI was extended to May 25th 2009, 4 weeks before the San Jose, Costa Rica meeting. However companies can request an extension to provide time to response. Progress of responses can be found at http://www.omg.org/techprocess/meetings/schedule/Information_Transformation… . The benefits in responding to this RFI include improving the systems-of-systems integration community life-cycle-costs and provide an opportunity to scope the standardization process around your specific solution such as the WebSphere ESB Mediation and Transformation Technologies. The Information Transformation RFI is attached with a summary below. Your input and participation is greatly appreciated to establish a standard in the long needed non-XML/XML information transformation methodology. There are many existing implementations within the industry that provide great potential to be standardized in whole or by parts. In today?s world, many information transformation solutions exist. Within the XML information technology domain, solutions have been standardized by the W3C and other standards bodies such as Organization for the Advancement of Structured Information Standards (OASIS). Even though XML is only ten years old, there has been much success in developing and standardizing XML information transformation technologies, but on the other hand, non-XML transformation methodologies for the most part have been ignored. This may be a side effect of an assumption that all information exchange instances across all vertical markets can use XML. Though this has proven to be wrong, application server, ESB and database providers have recognized the need to ease non-XML information transformation development by providing GUI and declarative methods. The problem is that these methods are proprietary, and often use closed architecture techniques. This presents a risk of vendor lock-in, which application developers and system integrators seek to avoid. This risk is even greater within standardized frameworks such as ESB. While ESB relevant wire, definition, and protocol standards include J2CA, JMS, SOAP, HTTP, WSDL, JDBC, XML1XSD, XSLT, XPATH, and several WS-* protocols, including WS-Addressing, WS-Security and WS-Reliable Messaging, vendors are including proprietary, non-standard extensions to handle required functions that are not standardized, like non-XML transformation. The elevated risk is the consequence of ESB developers unknowingly using everything within the ESB product assuming everything is standards and open architecture based. It is well known that ESB's can no longer exclusively focus on Web Services or message driven (JMS) use cases. They must also provide the ability to "service enable" non-XML based endpoints such as mainframes, flat files, Extract Transform Load (ETL) procedures and enterprise applications. To this end, the IEF WG is pursuing a non-XML transformation standardization effort that will give these middleware vendors an option to implement hence avoiding vendor lock-in. Even better, perhaps the same technology can be used across middleware capabilities such as application servers, ESB and databases. The goal is also to establish a standard approach for resolving semantics across interfaces. While much work has been accomplished using ontologies and the semantic Web with respect to information discovery, how these technologies can be used to automate system interoperability is a new field. Today, it is very typical to expend a knowledge engineering effort to resolve mappings between two interfaces. Once resolved, an XSLT is generated and the knowledge that provided basis for the rule is lost. The question is what can be done to retain this knowledge so it can be applied to future integration efforts . The intent of this RFI is to gather information about non-XML information transformation methodologies and standards; how they can coexist with XML standards, application server, ESB and database products; and how should the knowledge that defined the transformations be modeled. It is expected to get responses from middleware vendors providing their candidate solutions, as well as application developers and integrators providing their insight to the problems with regard to what issues need to be addressed. The information sought by this RFI will be used to move towards the goal of interoperable non-XML and XML that uses a standard data model format and standard service interfaces within an Information Exchange Framework. The approach is completely open in that the standards may simply point to an existing standard, or aggregation of standards; use standards by reference; or reuse standards, or any combination of these approaches. Very respectfully, Gary Sikora Progeny Systems Corporation gsikora(a)progeny.net 703-795-4684 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0