May 2009 - dfdl-wg - lists.ogf.org

Fw: Action 020 completion
by Steve Hanson 13 May '09

13 May '09

Hi Dave We should also bear the problem below in mind when thinking about DFDL Infoset & XDM. XDM assumes that an element with a concrete type-name has a typed-value conforming to the type-name, ie, it has been 'validated'. If this is not the case then the type-name is set to xs:untyped or xs:untypedAtomic (extra types added to XDM for this purpose). In DFDL Infoset we had been assuming that the [dataType] would be set to that implied by the DFDL xsd, regardless of whether validation succeeded or not - though there are issues with this as explained below. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/05/2009 12:51 ----- Steve Hanson/UK/IBM 29/04/2009 15:54 To Alan Powell, Dave Glick, Mike Beckerle (Work) cc Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM Subject Fw: Action 020 completion Hi Dave We discussed this on the call and agreed that the unsigned types are just range restrictions so treating negative numbers in the data as parse errors instead of validation errors seemed inconsistent. Various options discussed: 1) Remove the unsigned types altogether. - Means we'd need an extra property to describe binary integer representation, as we could no longer infer the rep from the logical type - Loses type information for applications where the fact that data is unsigned is important. - Means DFDL modelers would have to create their own duplicate restrictions for common C etc data types. 2) Change [dataType] to point to the XML Schema primitive type instead of the XML Schema built-in type. - Means that the value and the type would be xs:decimal which is too general 3) Change [dataValue] to say "The value in the value space of the underlying XML Schema primitive type forthe [datatype] member or special value nil" - Allows the infoset to carry integer data that is invalid due to range regardless of value. - Means that the value would be a decimal even though the data type was (say) xs:unsignedLong, ie, the datatype and datavalue are no longer in step unless validated 4) Option 2) with the modification that the primitive type for all integer types was xs:integer and not xs:decimal. 5) Option 3) with the modification that the primitive type for all integer types was xs:integer and not xs:decimal. We agreed not to close on this until you had reported back on your action 032. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 29/04/2009 15:32 ----- Steve Hanson/UK/IBM 27/04/2009 13:42 To Alan Powell/UK/IBM cc dglick(a)dracorp.com, mbeckerle(a)oco-inc.com Subject Re: Action 020 completion Thanks Alan I've added correct property names, see below. But I've omitted floats deliberately for clarity, the logical type is always signed and physical type is always signed, so there's no issue. However, there is a problem with what I have stated, as you pointed out on Friday. On parsing I am effectively validating the input data, but on unparsing I am not assuming the data has been validated. This is not consistent and needs correcting. But as I looked into this, I realised we have a problem with how we have described the DFDL infoset. The spec says "There is no requirement for DFDL-described data to be valid in order to have a DFDL information set.", which is in accordance with our agreed position on validation being optional. But further on it also says: [datatype] String. The name of the XML Schema 1.0 built-in simple type to which the value corresponds. DFDL supports a subset of these types listed in the specification at section 4.1. [dataValue] The value in the value space of the [datatype] member or special value nil. This says to me that the DFDL parser must have done enough validation to ascertain that the value matched the underlying built-in type. For example, I have a user-defined simple type that adds a max/min range of +100-+200 to an xs:unsignedInt. If the input data has value 99, the value will be accepted into the infoset, but will not validate if validation is switched on. If the input data is a packed decimal with value -1, the value will not be accepted into the infoset. Given that xs:unsignedInt is itself just a range restriction of xs:integer (via xs:nonNegativeInteger), this seems a bit arbitrary. Dave - given your action item looking at DFDL Infoset versus XDM, I'd be interested in your opinion here. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Alan Powell/UK/IBM 24/04/2009 15:13 To Steve Hanson/UK/IBM@IBMGB cc dglick(a)dracorp.com, mbeckerle(a)oco-inc.com Subject Re: Action 020 completion Steve Looks OK But can you use the correct property name eg binaryNumberRepresentation and for completeness add binaryFloatRepresentation (even though it may be obvious) Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Steve Hanson/UK/IBM To: mbeckerle(a)oco-inc.com, Alan Powell/UK/IBM, dglick(a)dracorp.com Date: 23/04/2009 12:50 Subject: Action 020 completion Here's my proposal for the behaviour when a logical type is signed and the physical data has no sign (either because it is not capable of carrying a sign, or it carries an unsigned indicator), and when the logical type is unsigned and the physical data has a sign. The principle is to be flexible and only to give errors when things are clearly mis-matched Logical type textNumberRepresentation= text (4) textNumberRepresentation= zoned (2) (6) binaryNumberRepresentation= packed (5) binaryNumberRepresentation= bcd (1) binaryNumberRepresentation= binary Signed (decimal, integer) Parse: OK Unparse: OK Parse: Unsigned data => +ve Unparse: Data always punched with sign Parse: Unsigned data => +ve Unparse: Data signed as per +ve/-ve nibble specifiers, unsigned nibble specifier never used Parse: Data always +ve Unparse: -ve data is error N/A Signed (long, int, short, byte) Ditto Ditto Ditto Ditto Parse: Data assumed 2's complement binary Unparse: Data output as 2's complement binary Unsigned (unsigned long, unsigned int, unsigned short, unsigned byte) (3) Parse: -ve data is error Unparse: -ve data is error Parse: +ve data => OK, -ve data is error Unparse: Sign never punched, -ve data is error Parse: +ve data => OK, -ve data is error Unparse: Unsigned nibble specifier always used, -ve data is error Parse: OK Unparse: -ve data is error Parse: Data assumed unsigned binary Unparse: Data output as unsigned binary (1) Can not physically carry a sign (2) Some systems output unsigned for +ve, but accept +ve on input (eg, IBM iSeries) (3) Assumes that on unparsing, the infoset could still present a -ve value (4) The -ve sign is indicated by numberPattern property (5) The exact sign nibbles are given by the packedDecimalSignCodes property (6) The punching style to use is given by the numberZonedSignStyle property Mail back any comments before next week's call. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/04/2009 12:04 ----- Steve Hanson/UK/IBM 18/02/2009 16:55 To Mike Beckerle (Work) cc Alan Powell/UK/IBM, dglick(a)dracorp.com Subject DFDL: Packed & zoned decimals - more thoughts (was Action 020) Hi Mike While we are on the subject of how to handle signs, the spec does not fully define what happens for a number if the logical type is unsigned. We need to say what is expected in the physical data and what happens if the data contains a sign. For example, we say that for an unsigned integer, if the rep is binary then we treat the data as 'unsigned binary' and not twos complement. And we say that BCD is only allowed for unsigned logical types. That is good. But we don't do the same for packed, text, zoned. I think we need to say that no explicit sign is expected in the data (eg, packed should have only F or 0, no A,B,C,D) and if it does: Alternatives: i) Error ii) Positive sign discarded, negative sign gives error iii) Sign discarded iv) As per i) if 'strict' set, as per ii) if 'lax' set v) As per i) if 'strict' set, as per iii) if 'lax' set Personally I vote for i) Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/02/2009 16:41 ----- Steve Hanson/UK/IBM 12/02/2009 13:18 To <mbeckerle.dfdl(a)gmail.com> cc dfdl-wg(a)ogf.org Subject RE: [DFDL-WG] Fw: DFDL OGF WG call - Action 020 Hi Mike I think it's a simplification too far. Many people especially those with a mainframe or COBOL background know what a zoned decimal is. The wikipedia entry for binary coded decimal explicitly covers the BCD, packed & zoned 'variants'. MRM and WTX both explicitly support zoned too. And it's easier to say that the 'decimalSignStyle' property applies to zoned decimals than to say it applies to any patterns that happen to have a P in them. On balance I would keep zoned as a representation. So we need to decide whether zoned is only allowed for a signed decimal. There's no harm in allowing it for unsigned, just some redundancy, and it makes validation of the pattern against the rep easier (if something is zoned it can only have a subset of pattern chars). Btw we don't need leading overpunched sign, only trailing - see my case for this below. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> 12/02/2009 12:51 Please respond to <mbeckerle.dfdl(a)gmail.com> To Steve Hanson/UK/IBM@IBMGB, <dfdl-wg(a)ogf.org> cc Subject RE: [DFDL-WG] Fw: DFDL OGF WG call - Action 020 This does suggest another simplification. Zoned is so close to text....Suppose we scrap the concept of "zoned" altogether, and just add a character to our number pattern language to allow one to specify a overpunched sign digit. E.g., "+00000" is text "P0000" same with overpunched leading sign. "00000+" text "0000P" same with overpunched trailing sign The decimal point would normally be implied in these, (I still like having a cobol-style "V" to position this instead of separate properties stating the position - one of the few good features about cobol is the number patterns. I still think we could quite easily pre-process the "V" out of these strings and then hand the rest through to an ICU library as an implementation - however the "P" probably does need to be a change in that library.) Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451 | mbeckerle.dfdl(a)gmail.com From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Steve Hanson Sent: Wednesday, February 11, 2009 1:34 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] Fw: DFDL OGF WG call - Action 020 It was noted on the call this week that there is an alternative to my zoned decimal overpunching proposal i) below. I said: - If it is an unsigned type then DFDL expects the rightmost byte to have a zone nibble when parsing, and outputs a zone nibble when unparsing. - If it is a signed type then DFDL expects it to have a sign nibble when parsing, and outputs a sign nibble when unparsing. But my unsigned type behaviour could be achieved by specifying a rep of text instead of zoned. If that is the case, the alternative is to only allow zoned rep for signed decimal logical types. Thoughts? Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/02/2009 10:17 ----- Steve Hanson/UK/IBM 28/01/2009 13:54 To DFDL Working Group cc Subject DFDL OGF WG call - Action 020Link Action 020: 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position a) Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy Add new property to section 15.4 Properties Specific to Number with Binary representation. binaryNumberCheckPolicy Enum Values are ?strict? and ?lax?. Indicates how lenient to be when parsing binary numbers. If ?lax? then the parser tolerates all valid alternatives where such alternatives exist. Specifically, for binaryNumberRepresentation = 'packed' the sign nibble for positive, negative, unsigned and zero is allowed to be any of the valid respective values. On unparsing, the specified value is always used. Also suggest changing some of the other property names in 15.4: "decimalVirtualPoint" -> "binaryDecimalVirtualPoint" "packedDecimalSignCodes" -> "binaryPackedSignCodes" And changing binaryNumberRepresentation enumeration: "BCD" -> "bcd" b) Zoned decimals: How to decide to overpunch and sign position Spec assumes that overpunching of the rightmost character always takes place. IBM architecture allows no overpunching (ie, Fx instead of Cx/Dx) - this is supported by IBM MRM & WTX parsers. Additionally IBM MRM parser allows separate sign byte, and sign byte on left. Let's deal with these separately: i) No overpunching. The IBM architecture allows the rightmost byte to have a zone (Fx) or a sign (Cx/Dx) as the left nibble. I don't see why we can't base what to expect when parsing, and output when unparsing, on the logical xsd type. - If it is an unsigned type then DFDL expects the rightmost byte to have a zone nibble when parsing, and outputs a zone nibble when unparsing. - If it is a signed type then DFDL expects it to have a sign nibble when parsing, and outputs a sign nibble when unparsing. For analogy with DFDL packed decimals, it seems at first glance that we should also extend the numberCheckPolicy 'lax' setting to treat a zone nibble as a +ve sign nibble for a signed type. However, IBM iSeries always outputs Fx to mean +ve but accepts both Fx & Cx on input. It is perhaps better therefore that DFDL always tolerates Fx when parsing a signed zoned decimal, otherwise iSeries users would always have to set numberCheckPolicy to 'lax', which might have other implications in the future. ii) Separate sign byte. I don't believe the IBM architecture allows this. I don't think DFDL needs to support it. MRM has this, but I think it's because early on MRM did not explicitly support text decimals as such, just COBOL variations, and it was easier just to call them all zoned. iii) Sign byte on left. I don't believe the IBM architecture allows this. I don't think DFDL needs to support it. MRM has this, but for the same reason as ii) Conclusion: No new DFDL properties needed, but words need adding to explain zoned parse/unparse behaviour better. Also suggest changing property names: "zonedDecimalSignStyle" -> "numberZonedSignStyle" "zeroNumberRep" -> "numberZeroRep" Should also make clear that any explicit negative pattern in numberPattern will be ignored if the xsd type is unsigned. (We could make this an error but it precludes creation of a textNumberFormat that works with both signed and unsigned types, plus pattern "##0.0" implictly is equivalent to "##0.0;(##0.0)" ). Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Alan Powell/UK/IBM@IBMGB Sent by: dfdl-wg-bounces(a)ogf.org 23/01/2009 13:36 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] DFDL: Minutes from OGF WG call, 21 January 2009 Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 14:00 GMT, 21 January 2009 Attendees Alan Powell (IBM) Mike Beckerle(Oco) Apologies Steve Hanson (IBM) 1. XSD 1.1 Deferred to next call 2. Calendar formats Discussed updated (v4) supplement emailed by AP Agreed millisec/secSinceEpoc cannot be implied by length of logical data so need seperate enumerations. Observed that these options were really combination of 3 properties binary, length and sec/millisec. Suggested renaming to binarySeconds and binaryMilliseconds Packed calendars: decided that need to be able to specify at least the packedDecimalSignCodes property rather than assuming a default so reference will be added to calendar description Locale needs to be specified for numberformats and calendarFormats (didn't identify any other areas) as it modifies the behaviour of ICU. Decided to add locale to numberFormat and CalendarFormat 3. Escape Schemes Agreed need for multiple escape delimiter pairs but not nested. Need an escape for escape character even though in most cases this will be the same character, eg /n //, There are some formats that have a different escape, eg /n &/. Only need single escape characters and one level of escape characters. Discussed how to deal with comments of the form /* comment */ where the escape delimiters are also the initiator and terminator of the field. Semantic needed is 'only look for field terminator not any parent terminator or any other syntax elements'. May fall out naturally from the speculative parsing rules. Need further discussion. 4. AOB Next call 28 January 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action 031 Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 023 MB: Review Schema 1.1 024 String XML type 025 Escape schemes 21/1: discussed requirements 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 030 AP: confirm with WTX that can drop duration 21/6: WTX confirm that they do not have a duration type so do not need it in dfdl. Will drop from spec. Closed Closed actions: 030 AP: confirm with WTX that can drop duration 21/6: WTX confirm that they do not have a duration type so do not need it in dfdl. Will drop from spec. Closed 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF WG call 13 May 2009
by Alan Powell 13 May '09

13 May '09

Agenda: 1. Go through actions. 2. Check Work items - add incorporate DG comments on 033 3. Annotations on Top Level element 4. Scoping rules for assert/discriminator and other annotations 5. Comments latest version (034) 6. AOB Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 08/04: Not discussed 22/04: AP to complete asap once the ICU strict/lax behaviour is understood. 29/04: No progress 06/05: No progress 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 08/04: Not discussed 22/04: SH to complete last remaining issue, which is the behaviour when logical type is signed/unsigned and the physical type is unsigned/signed. 29/04: SH had identified a problem with definition values and types in the infoset and will email proposal. DG to be asked to accelerate action 032 to see if helps 06/05: No progress 026 SH: Envelopes and Payloads 08/04: Not discussed explicity, but recursive use of DFDL is tied up with this 22/04: Two aspects. Firstly compositional - do sufficient mechanisms exist to model an envelope with a payload that varies. Secondly markup syntax - this might be defined in the envelope. The second of these is very much tied up with the variable markup action 028, so will be considered there. SH to verify the composition aspect. 29/04: SH and AP working on proposal. related to Action 028 06/05: No progress 027 SH: Property precedence tables 08/04: Not discussed 22/04: Two things missing from the existing precedence trees. Firstly, does not show alternates (eg, initiator v initiatorkind). Secondly, need a tree per concrete DFDL object (eg, element). SH to update. 29/04: No progress 06/05: SH is updating tables which will be ready for next call 028 SH: Variable markup 08/04: Discussed briefly at end of call, IBM to see whether there any use cases that require recursive use of DFDL. 15/04: Use case was distributed and will be discussed on next call. 22/04: The use case in question is EDI where the terminating markup for the payload segments is defined in the ISA envelope segment. The markup is modelled as an element of simple type where the allowable markup values are defined as enums on the type. But we need to handle two cases - firstly where the envelope is present, so the value used by the payload is taken from the envelope. Secondly where only the payload is present. Here we need a way of scanning for all the enum values, and adopting the one we actually find, when parsing. And using a default when unparsing. SH to explore use of a DFDL variable, where the variable has a default, but also has a type that is the same as the markup element - that way we get to use the enums without defining everything twice. 29/04: SH and AP working on proposal. 06/05: No progress 029 MB: valueCalc (output length calculation) 08/04: Not discussed 22/04: Action allocated to MB, this is to complete the work started at the Hursley WG F2F meeting. 29/04: No progress 06/05: MB will have update for next call 032 DG: Investigate compatibility between DFDL infoset and XDM 08/04: No update 22/04: No update 29/04: No update 06/05: DG indicates will have update next week 033 AP/TK: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 08/04: In progress within IBM 22/04: Waiting for TK to return from leave to complete. 29/04: TK has sent examples shown need for discriminators beyond choice. Agreed. MB to respond to TK 06/05: Discussed suggestion of adding type indicator to discriminator. MB to provide examples. 036 SH: Provide use case for floating component in a sequence 08/04: Raised 15/04: Use case sent and discussed. SH to do further investigation 22/04: IBM feedback from WTX team is that alternate suggested ways of modelling the EDI floating NTE segment have significant usability issues. The DFDL principle is that for a problem that can be expressed as two-layered, then two DFDL models are needed. The EDI NTE segment does not fall into this though, as its use is on a per sequence basis. Ongoing. 29/04: Agreed that need to be in V1. SH to make a proposal 06/05: No progress 037 All: Approach for XML Schema 1.0 UPA checks. 22/04: Several non-XML models, when expressed in their most obvious DFDL Schema form, would fail XML Schema 1.0 Unique Particle Attribution checks that police model ambiguity. And even re-jigging the model sometimes fails to fix this. Note this is equally applicable to XMl Schema 1.1 and 1.0. While the DFDL parser/unparser can happily resolve the ambiguities, the issue is one of definition. If an XSD editor that implements UPA checks is used to create DFDL Schema, then errors will be flagged. DFDL may have to adopt the position that: a)DFDL parser/unparser will not implement some/all UPA checks (exact checks tbd) b) XML Schema editors that implement UPA checks will not be suitable for all DFDL models c) If DFDL annotations are removed, the resulting pure XSD will not always be valid (ie, the equivalent XML is ambiguous and can't be modelled by XML Schema 1.0) Ongoing in case another solution can be found. 29/04: Will ask DG and S Gao for opinion before closing 06/05: Discussed S Gao email and suggestions. Decided need to review all XML UPA rules and decide which apply to dfdl. 038 MB: Submit response to OMG RFI for non-XML standardization 22/04: First step is for MB to mail the OGF Data Area chair to say that we want to submit 29/04: MB has been in contact with OMG and will sunbit dfdl. 06/05: MB has prepared response to OMG. Will send DFDL sepc v033 039 SKK: Approach for creating Schema-For-DFDL xsds. 22/04: Resolve issue around multiple declarations needed for DFDL properties, perhaps using MB's meta approach 29/04: Don't like qualified attributes in long form. SKK to check there are no code gen implications, eg EMF. 06/05: SKK will send update by Friday 040 SH: LengthKind on complex objects. 29/04: All send comment before next call 06/05: See minutes. Agreed to remove lengthKind from sequence and choice. 041 AP: UnorderedInitiated 29/04: All: Review for next call 06/05: See minutes: Agreed to generalize to all sequences 042 MB: Complete variable specification. To include how properties such as encoding can be set externally. Must be a known variable name. 06/05: No progress Work items: No Item target version status 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 034 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 (A033) 034 in progress 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) not started 025 Augmented infoset and unparsing (Alan) 034 added but needs work 026 Remove duration 034 027 Calendar schemes 034 028 Validation ranges (A035) 034 029 Decimals (A020) - document unsigned type behaviour - packedDecimalSignCodes behaviour depends on NumberCheckPolicy 034 030 Remove redundant properties, fix examples. (A036) 034 031 Specialized annotations 034 032 Floating components 033 Specialized annotations 034 Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF WG call 1 April March 2009
by Alan Powell 13 May '09

13 May '09

Agenda: 1. Go through actions. 2. Check Work items - add incorporate DG comments on 033 3. Annotations on Top Level element 4. Scoping rules for assert/discriminator and other annotations 5. Comments latest version (034) 6. AOB Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 08/04: Not discussed 22/04: AP to complete asap once the ICU strict/lax behaviour is understood. 29/04: No progress 06/05: No progress 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 08/04: Not discussed 22/04: SH to complete last remaining issue, which is the behaviour when logical type is signed/unsigned and the physical type is unsigned/signed. 29/04: SH had identified a problem with definition values and types in the infoset and will email proposal. DG to be asked to accelerate action 032 to see if helps 06/05: No progress 026 SH: Envelopes and Payloads 08/04: Not discussed explicity, but recursive use of DFDL is tied up with this 22/04: Two aspects. Firstly compositional - do sufficient mechanisms exist to model an envelope with a payload that varies. Secondly markup syntax - this might be defined in the envelope. The second of these is very much tied up with the variable markup action 028, so will be considered there. SH to verify the composition aspect. 29/04: SH and AP working on proposal. related to Action 028 06/05: No progress 027 SH: Property precedence tables 08/04: Not discussed 22/04: Two things missing from the existing precedence trees. Firstly, does not show alternates (eg, initiator v initiatorkind). Secondly, need a tree per concrete DFDL object (eg, element). SH to update. 29/04: No progress 06/05: SH is updating tables which will be ready for next call 028 SH: Variable markup 08/04: Discussed briefly at end of call, IBM to see whether there any use cases that require recursive use of DFDL. 15/04: Use case was distributed and will be discussed on next call. 22/04: The use case in question is EDI where the terminating markup for the payload segments is defined in the ISA envelope segment. The markup is modelled as an element of simple type where the allowable markup values are defined as enums on the type. But we need to handle two cases - firstly where the envelope is present, so the value used by the payload is taken from the envelope. Secondly where only the payload is present. Here we need a way of scanning for all the enum values, and adopting the one we actually find, when parsing. And using a default when unparsing. SH to explore use of a DFDL variable, where the variable has a default, but also has a type that is the same as the markup element - that way we get to use the enums without defining everything twice. 29/04: SH and AP working on proposal. 06/05: No progress 029 MB: valueCalc (output length calculation) 08/04: Not discussed 22/04: Action allocated to MB, this is to complete the work started at the Hursley WG F2F meeting. 29/04: No progress 06/05: MB will have update for next call 032 DG: Investigate compatibility between DFDL infoset and XDM 08/04: No update 22/04: No update 29/04: No update 06/05: DG indicates will have update next week 033 AP/TK: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 08/04: In progress within IBM 22/04: Waiting for TK to return from leave to complete. 29/04: TK has sent examples shown need for discriminators beyond choice. Agreed. MB to respond to TK 06/05: Discussed suggestion of adding type indicator to discriminator. MB to provide examples. 036 SH: Provide use case for floating component in a sequence 08/04: Raised 15/04: Use case sent and discussed. SH to do further investigation 22/04: IBM feedback from WTX team is that alternate suggested ways of modelling the EDI floating NTE segment have significant usability issues. The DFDL principle is that for a problem that can be expressed as two-layered, then two DFDL models are needed. The EDI NTE segment does not fall into this though, as its use is on a per sequence basis. Ongoing. 29/04: Agreed that need to be in V1. SH to make a proposal 06/05: No progress 037 All: Approach for XML Schema 1.0 UPA checks. 22/04: Several non-XML models, when expressed in their most obvious DFDL Schema form, would fail XML Schema 1.0 Unique Particle Attribution checks that police model ambiguity. And even re-jigging the model sometimes fails to fix this. Note this is equally applicable to XMl Schema 1.1 and 1.0. While the DFDL parser/unparser can happily resolve the ambiguities, the issue is one of definition. If an XSD editor that implements UPA checks is used to create DFDL Schema, then errors will be flagged. DFDL may have to adopt the position that: a)DFDL parser/unparser will not implement some/all UPA checks (exact checks tbd) b) XML Schema editors that implement UPA checks will not be suitable for all DFDL models c) If DFDL annotations are removed, the resulting pure XSD will not always be valid (ie, the equivalent XML is ambiguous and can't be modelled by XML Schema 1.0) Ongoing in case another solution can be found. 29/04: Will ask DG and S Gao for opinion before closing 06/05: Discussed S Gao email and suggestions. Decided need to review all XML UPA rules and decide which apply to dfdl. 038 MB: Submit response to OMG RFI for non-XML standardization 22/04: First step is for MB to mail the OGF Data Area chair to say that we want to submit 29/04: MB has been in contact with OMG and will sunbit dfdl. 06/05: MB has prepared response to OMG. Will send DFDL sepc v033 039 SKK: Approach for creating Schema-For-DFDL xsds. 22/04: Resolve issue around multiple declarations needed for DFDL properties, perhaps using MB's meta approach 29/04: Don't like qualified attributes in long form. SKK to check there are no code gen implications, eg EMF. 06/05: SKK will send update by Friday 040 SH: LengthKind on complex objects. 29/04: All send comment before next call 06/05: See minutes. Agreed to remove lengthKind from sequence and choice. 041 AP: UnorderedInitiated 29/04: All: Review for next call 06/05: See minutes: Agreed to generalize to all sequences 042 MB: Complete variable specification. To include how properties such as encoding can be set externally. Must be a known variable name. 06/05: No progress Work items: No Item target version status 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 034 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 (A033) 034 in progress 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) not started 025 Augmented infoset and unparsing (Alan) 034 added but needs work 026 Remove duration 034 027 Calendar schemes 034 028 Validation ranges (A035) 034 029 Decimals (A020) - document unsigned type behaviour - packedDecimalSignCodes behaviour depends on NumberCheckPolicy 034 030 Remove redundant properties, fix examples. (A036) 034 031 Specialized annotations 034 032 Floating components 033 Specialized annotations 034 Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF WG call 6 May 2009
by Alan Powell 13 May '09

13 May '09

Agenda: 1. Go through actions. 2. LengthKind on Sequences and choices. LengthKind on sequences and choices and their parent element has proved confusing to new users of DFDL. It is proposed that lengthKind is removed from groups and only allow it to be set on parent element. See email from SH 3. Discuss UnorderedInitated email from SH 4. Infoset codepage and encoding The spec does not say what codepage and encoding is used for string fields. 5. AOB Next version (034) Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 08/04: Not discussed 22/04: AP to complete asap once the ICU strict/lax behaviour is understood. 29/04: No progress 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 08/04: Not discussed 22/04: SH to complete last remaining issue, which is the behaviour when logical type is signed/unsigned and the physical type is unsigned/signed. 29/04: SH had identified a problem with definition values and types in the infoset and will email proposal. DG to be asked to accelerate action 032 to see if helps 024 <No owner> String XML type 08/04: Not discussed 22/04: Need to allocate owner. Work is to describe the semantics of using dfdl:representation="xml" to model a well-formed XML fragment in an overall non-XML document described by a DFDL schema. 29/04: As no resource availbel to progress this action agreed to defer from V1. Will close next week if no objections 026 SH: Envelopes and Payloads 08/04: Not discussed explicity, but recursive use of DFDL is tied up with this 22/04: Two aspects. Firstly compositional - do sufficient mechanisms exist to model an envelope with a payload that varies. Secondly markup syntax - this might be defined in the envelope. The second of these is very much tied up with the variable markup action 028, so will be considered there. SH to verify the composition aspect. 29/04: SH and AP working on proposal. related to Action 028 027 SH: Property precedence tables 08/04: Not discussed 22/04: Two things missing from the existing precedence trees. Firstly, does not show alternates (eg, initiator v initiatorkind). Secondly, need a tree per concrete DFDL object (eg, element). SH to update. 29/04: No progress 028 SH: Variable markup 08/04: Discussed briefly at end of call, IBM to see whether there any use cases that require recursive use of DFDL. 15/04: Use case was distributed and will be discussed on next call. 22/04: The use case in question is EDI where the terminating markup for the payload segments is defined in the ISA envelope segment. The markup is modelled as an element of simple type where the allowable markup values are defined as enums on the type. But we need to handle two cases - firstly where the envelope is present, so the value used by the payload is taken from the envelope. Secondly where only the payload is present. Here we need a way of scanning for all the enum values, and adopting the one we actually find, when parsing. And using a default when unparsing. SH to explore use of a DFDL variable, where the variable has a default, but also has a type that is the same as the markup element - that way we get to use the enums without defining everything twice. 29/04: SH and AP working on proposal. 029 MB: valueCalc (output length calculation) 08/04: Not discussed 22/04: Action allocated to MB, this is to complete the work started at the Hursley WG F2F meeting. 29/04: No progress 032 DG: Investigate compatibility between DFDL infoset and XDM 08/04: No update 22/04: No update 29/04: No update 033 AP/TK: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 08/04: In progress within IBM 22/04: Waiting for TK to return from leave to complete. 29/04: TK has sent examples shown need for discriminators beyond choice. Agreed. MB to respond to TK 036 SH: Provide use case for floating component in a sequence 08/04: Raised 15/04: Use case sent and discussed. SH to do further investigation 22/04: IBM feedback from WTX team is that alternate suggested ways of modelling the EDI floating NTE segment have significant usability issues. The DFDL principle is that for a problem that can be expressed as two-layered, then two DFDL models are needed. The EDI NTE segment does not fall into this though, as its use is on a per sequence basis. Ongoing. 29/04: Agreed that need to be in V1. SH to make a proposal 037 All: Approach for XML Schema 1.0 UPA checks. 22/04: Several non-XML models, when expressed in their most obvious DFDL Schema form, would fail XML Schema 1.0 Unique Particle Attribution checks that police model ambiguity. And even re-jigging the model sometimes fails to fix this. Note this is equally applicable to XMl Schema 1.1 and 1.0. While the DFDL parser/unparser can happily resolve the ambiguities, the issue is one of definition. If an XSD editor that implements UPA checks is used to create DFDL Schema, then errors will be flagged. DFDL may have to adopt the position that: a)DFDL parser/unparser will not implement some/all UPA checks (exact checks tbd) b) XML Schema editors that implement UPA checks will not be suitable for all DFDL models c) If DFDL annotations are removed, the resulting pure XSD will not always be valid (ie, the equivalent XML is ambiguous and can't be modelled by XML Schema 1.0) Ongoing in case another solution can be found. 29/04: Will ask DG and S Gao for oppinion before closing 038 MB: Submit response to OMG RFI for non-XML standardization 22/04: First step is for MB to mail the OGF Data Area chair to say that we want to submit 29/04: MB has been in contact with OMG and will sunbit dfdl. 039 SKK: Approach for creating Schema-For-DFDL xsds. 22/04: Resolve issue around multiple declarations needed for DFDL properties, perhaps using MB's meta approach 29/04: Don't like qualified attributes in long form. SKK to check there are no code gen implications, eg EMF. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 7

Re: [DFDL-WG] DFDL: Action 027 - rework of property precedence - plus issues arising
by Steve Hanson 12 May '09

12 May '09

Hi Alan Thanks for the feedback, I will take a look at each point. For point 3 though, encoding and byteOrder do apply for sequence, choice and any, because there could be markup involved. Whether inputValueCalc and outputValueCalc do is a different matter and I will think about those. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Alan Powell/UK/IBM 11/05/2009 16:57 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Subject Re: [DFDL-WG] DFDL: Action 027 - rework of property precedence - plus issues arising Steve Comments I don't think the tables answer the precedence of padding, escaping and encoding that we had when discussing escape schemes. On parsing it should be remove padding, remove escape characters, apply encoding. On unparsing it is the reverse but the tables look the same. I think the difficulty is that escape scheme is used for identification and extraction but also during conversion. I don't think the core properties (inputvaluecalc, etc) apply to sequence, choice or any. xxxPadKind is checked before xxxPadCharacter, xxpadxxx Calendar-binary. binaryCalendarFormatRef is only used when binCalRep = packed or bcd Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 08/05/2009 17:24 Subject: [DFDL-WG] DFDL: Action 027 - rework of property precedence - plus issues arising I've created a separate property precedence for each schema object that can carry non-scoping DFDL properties (attached for review). The following issues were noted: 1) Missing property dfdl:textBooleanJustification - similar properties exist for string, number and calendar types. 2) What is the rule when the same DFDL properties occur on a xs:simpleType and a xs:element that uses that type? Does this work a) like element/group references (ie, properties combined with element winning) or b) like complex element and its sequence (ie, element and simpleType are considered separate objects)? I don't think section 10 covers this case. 3) Should we allow the DFDL nil & default control properties on a simple type? xs:nillable and xs:default are element only attributes in xsd. Spec currently allows this. 4) Should we allow DFDL occurs properties on global elements? Whether something repeats is a particle thing. Spec currently allows this. (IBM's WTX and MRM don't allow this). 5) Missing work item to get BiDi properties into shape and incorporated into spec. Should these be grouped, like escape scheme, calendar scheme, etc? Do they apply to calendar and number types? 6) Should dfdl:integerBooleanXXXRep be renamed dfdl:binaryBooleanXXXRep ? 7) We might want to reconsider the name of the new flag dfdl:initiated - it could be read that the xs:sequence itself is initiated rather than its children. 8) Should dfdl:initiated also apply to xs:choice? 9) Draft 33 property precedence had dfdl:outputLengthCalc - but that is not in the spec anywhere else? Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "Property Precedence 034.doc" deleted by Alan Powell/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL Schema and instance documents ( Action Item: 039 )
by Suman Kalia 10 May '09

10 May '09

I have created schema based on version 0.33 of spec and other discussions to the best of my recollection.. Alan - I think we need to update the DFDL property description for Any element (page 130) . The document is missing DFDL properties for ALL group, In the schema I have assumed ALL group to have the same set of properties as the Sequence group. It is quite possible that I might have missed some property or had a typo in the property name. We should schedule review of the schema next week. As you would notice in the instance documents, the attribute instances in the special annotations are unqualified as per the spec.. Also there is a global attribute definition for each of the DFDL property defined in the spec.. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com

2 1

Minutes for OGF DFDL Working Group Call, May-06-2009
by Alan Powell 06 May '09

06 May '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, May-06-2009 Meeting opened, 14:00 UK Attendees Steve Hanson (IBM) Mike Beckerle (Oco) Suman Kalia (IBM) Alan Powell (IBM) Apologies Dave Glick (drac) Agenda: 1. Go through actions. Actions updated below 2. LengthKind on Sequences and choices. Agreed with proposal to remove lengthKind from sequences and choices. Discussed whether other properties such as initiator and terminator should also be moved but decided it would introduce too many elements and make the resultant tree deep. Will keep open until SH completes property precedence and will then look at all sequence and choice properties 3. Discuss UnorderedInitated email from SH Agreed that it was useful to indicate that children of sequences are initiated to enable more helpful error messagies to be generated. Agreed to generalize to all sequences not just unordered. 4. Infoset codepage and encoding Decided simplest solution is to pick UTF-16 5. AOB Next version (034) will be available for review on Friday Next call 13 May 14:00 UK Meeting closed, 15:10 UK Actions raised at this meeting No Action Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 08/04: Not discussed 22/04: AP to complete asap once the ICU strict/lax behaviour is understood. 29/04: No progress 06/05: No progress 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 08/04: Not discussed 22/04: SH to complete last remaining issue, which is the behaviour when logical type is signed/unsigned and the physical type is unsigned/signed. 29/04: SH had identified a problem with definition values and types in the infoset and will email proposal. DG to be asked to accelerate action 032 to see if helps 06/05: No progress 026 SH: Envelopes and Payloads 08/04: Not discussed explicity, but recursive use of DFDL is tied up with this 22/04: Two aspects. Firstly compositional - do sufficient mechanisms exist to model an envelope with a payload that varies. Secondly markup syntax - this might be defined in the envelope. The second of these is very much tied up with the variable markup action 028, so will be considered there. SH to verify the composition aspect. 29/04: SH and AP working on proposal. related to Action 028 06/05: No progress 027 SH: Property precedence tables 08/04: Not discussed 22/04: Two things missing from the existing precedence trees. Firstly, does not show alternates (eg, initiator v initiatorkind). Secondly, need a tree per concrete DFDL object (eg, element). SH to update. 29/04: No progress 06/05: SH is updating tables which will be ready for next call 028 SH: Variable markup 08/04: Discussed briefly at end of call, IBM to see whether there any use cases that require recursive use of DFDL. 15/04: Use case was distributed and will be discussed on next call. 22/04: The use case in question is EDI where the terminating markup for the payload segments is defined in the ISA envelope segment. The markup is modelled as an element of simple type where the allowable markup values are defined as enums on the type. But we need to handle two cases - firstly where the envelope is present, so the value used by the payload is taken from the envelope. Secondly where only the payload is present. Here we need a way of scanning for all the enum values, and adopting the one we actually find, when parsing. And using a default when unparsing. SH to explore use of a DFDL variable, where the variable has a default, but also has a type that is the same as the markup element - that way we get to use the enums without defining everything twice. 29/04: SH and AP working on proposal. 06/05: No progress 029 MB: valueCalc (output length calculation) 08/04: Not discussed 22/04: Action allocated to MB, this is to complete the work started at the Hursley WG F2F meeting. 29/04: No progress 06/05: MB will have update for next call 032 DG: Investigate compatibility between DFDL infoset and XDM 08/04: No update 22/04: No update 29/04: No update 06/05: DG indicates will have update next week 033 AP/TK: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 08/04: In progress within IBM 22/04: Waiting for TK to return from leave to complete. 29/04: TK has sent examples shown need for discriminators beyond choice. Agreed. MB to respond to TK 06/05: Discussed suggestion of adding type indicator to discriminator. MB to provide examples. 036 SH: Provide use case for floating component in a sequence 08/04: Raised 15/04: Use case sent and discussed. SH to do further investigation 22/04: IBM feedback from WTX team is that alternate suggested ways of modelling the EDI floating NTE segment have significant usability issues. The DFDL principle is that for a problem that can be expressed as two-layered, then two DFDL models are needed. The EDI NTE segment does not fall into this though, as its use is on a per sequence basis. Ongoing. 29/04: Agreed that need to be in V1. SH to make a proposal 06/05: No progress 037 All: Approach for XML Schema 1.0 UPA checks. 22/04: Several non-XML models, when expressed in their most obvious DFDL Schema form, would fail XML Schema 1.0 Unique Particle Attribution checks that police model ambiguity. And even re-jigging the model sometimes fails to fix this. Note this is equally applicable to XMl Schema 1.1 and 1.0. While the DFDL parser/unparser can happily resolve the ambiguities, the issue is one of definition. If an XSD editor that implements UPA checks is used to create DFDL Schema, then errors will be flagged. DFDL may have to adopt the position that: a)DFDL parser/unparser will not implement some/all UPA checks (exact checks tbd) b) XML Schema editors that implement UPA checks will not be suitable for all DFDL models c) If DFDL annotations are removed, the resulting pure XSD will not always be valid (ie, the equivalent XML is ambiguous and can't be modelled by XML Schema 1.0) Ongoing in case another solution can be found. 29/04: Will ask DG and S Gao for opinion before closing 06/05: Discussed S Gao email and suggestions. Decided need to review all XML UPA rules and decide which apply to dfdl. 038 MB: Submit response to OMG RFI for non-XML standardization 22/04: First step is for MB to mail the OGF Data Area chair to say that we want to submit 29/04: MB has been in contact with OMG and will sunbit dfdl. 06/05: MB has prepared response to OMG. Will send DFDL sepc v033 039 SKK: Approach for creating Schema-For-DFDL xsds. 22/04: Resolve issue around multiple declarations needed for DFDL properties, perhaps using MB's meta approach 29/04: Don't like qualified attributes in long form. SKK to check there are no code gen implications, eg EMF. 06/05: SKK will send update by Friday 040 SH: LengthKind on complex objects. 29/04: All send comment before next call 06/05: See minutes. Agreed to remove lengthKind from sequence and choice. 041 AP: UnorderedInitiated 29/04: All: Review for next call 06/05: See minutes: Agreed to generalize to all sequences 042 MB: Complete variable specification. To include how properties such as encoding can be set externally. Must be a known variable name. 06/05: No progress Closed actions: 024 <No owner> String XML type 08/04: Not discussed 22/04: Need to allocate owner. Work is to describe the semantics of using dfdl:representation="xml" to model a well-formed XML fragment in an overall non-XML document described by a DFDL schema. 29/04: As no resource availbel to progress this action agreed to defer from V1. Will close next week if no objections 06/05: closed Work items: No Item target version status 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 034 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 (A033) 034 in progress 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) not started 025 Augmented infoset and unparsing (Alan) 034 added but needs work 026 Remove duration 034 027 Calendar schemes 034 028 Validation ranges (A035) 034 029 Decimals (A020) - document unsigned type behaviour - packedDecimalSignCodes behaviour depends on NumberCheckPolicy 034 030 Remove redundant properties, fix examples. (A036) 034 031 Specialized annotations 034 032 Floating components 033 Specialized annotations 034 Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Minutes for OGF DFDL Working Group Call, April-29-2009
by Alan Powell 05 May '09

05 May '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, April-29-2009 Meeting opened, 14:00 UK Attendees Steve Hanson (IBM) Mike Beckerle (Oco) Suman Kalia (IBM) Alan Powell (IBM) Apologies Dave Glick (drac) Agenda: 1. Go through actions. Actions updated below 2. LengthKind on Sequences and choices. LengthKind on sequences and choices and their parent element has proved confusing to new users of DFDL. It is proposed that lengthKind is removed from groups and only allow it to be set on parent element. See email from SH Not discussed. Action raised. Please review SH email and make comments before next call 3. Discuss UnorderedInitated email from SH Not discussed. Action raised. Please review SH email and make comments before next call 4. Infoset codepage and encoding The spec does not say what codepage and encoding is used for string fields. 5. AOB Next version (034) 6. Next call 6 May 14:00 UK Meeting closed, 15:10 UK Actions raised at this meeting No Action 040 SH: LengthKind on complex objects. 29/04: All send comment before next call 041 AP: UnorderedInitiated 29/04: All: Review for next call 042 MB: Complete variable specification. To include how properties such as encoding can be set externally. Must be a known variable name. Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 08/04: Not discussed 22/04: AP to complete asap once the ICU strict/lax behaviour is understood. 29/04: No progress 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 08/04: Not discussed 22/04: SH to complete last remaining issue, which is the behaviour when logical type is signed/unsigned and the physical type is unsigned/signed. 29/04: SH had identified a problem with definition values and types in the infoset and will email proposal. DG to be asked to accelerate action 032 to see if helps 024 <No owner> String XML type 08/04: Not discussed 22/04: Need to allocate owner. Work is to describe the semantics of using dfdl:representation="xml" to model a well-formed XML fragment in an overall non-XML document described by a DFDL schema. 29/04: As no resource availbel to progress this action agreed to defer from V1. Will close next week if no objections 026 SH: Envelopes and Payloads 08/04: Not discussed explicity, but recursive use of DFDL is tied up with this 22/04: Two aspects. Firstly compositional - do sufficient mechanisms exist to model an envelope with a payload that varies. Secondly markup syntax - this might be defined in the envelope. The second of these is very much tied up with the variable markup action 028, so will be considered there. SH to verify the composition aspect. 29/04: SH and AP working on proposal. related to Action 028 027 SH: Property precedence tables 08/04: Not discussed 22/04: Two things missing from the existing precedence trees. Firstly, does not show alternates (eg, initiator v initiatorkind). Secondly, need a tree per concrete DFDL object (eg, element). SH to update. 29/04: No progress 028 SH: Variable markup 08/04: Discussed briefly at end of call, IBM to see whether there any use cases that require recursive use of DFDL. 15/04: Use case was distributed and will be discussed on next call. 22/04: The use case in question is EDI where the terminating markup for the payload segments is defined in the ISA envelope segment. The markup is modelled as an element of simple type where the allowable markup values are defined as enums on the type. But we need to handle two cases - firstly where the envelope is present, so the value used by the payload is taken from the envelope. Secondly where only the payload is present. Here we need a way of scanning for all the enum values, and adopting the one we actually find, when parsing. And using a default when unparsing. SH to explore use of a DFDL variable, where the variable has a default, but also has a type that is the same as the markup element - that way we get to use the enums without defining everything twice. 29/04: SH and AP working on proposal. 029 MB: valueCalc (output length calculation) 08/04: Not discussed 22/04: Action allocated to MB, this is to complete the work started at the Hursley WG F2F meeting. 29/04: No progress 032 DG: Investigate compatibility between DFDL infoset and XDM 08/04: No update 22/04: No update 29/04: No update 033 AP/TK: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. 08/04: In progress within IBM 22/04: Waiting for TK to return from leave to complete. 29/04: TK has sent examples shown need for discriminators beyond choice. Agreed. MB to respond to TK 036 SH: Provide use case for floating component in a sequence 08/04: Raised 15/04: Use case sent and discussed. SH to do further investigation 22/04: IBM feedback from WTX team is that alternate suggested ways of modelling the EDI floating NTE segment have significant usability issues. The DFDL principle is that for a problem that can be expressed as two-layered, then two DFDL models are needed. The EDI NTE segment does not fall into this though, as its use is on a per sequence basis. Ongoing. 29/04: Agreed that need to be in V1. SH to make a proposal 037 All: Approach for XML Schema 1.0 UPA checks. 22/04: Several non-XML models, when expressed in their most obvious DFDL Schema form, would fail XML Schema 1.0 Unique Particle Attribution checks that police model ambiguity. And even re-jigging the model sometimes fails to fix this. Note this is equally applicable to XMl Schema 1.1 and 1.0. While the DFDL parser/unparser can happily resolve the ambiguities, the issue is one of definition. If an XSD editor that implements UPA checks is used to create DFDL Schema, then errors will be flagged. DFDL may have to adopt the position that: a)DFDL parser/unparser will not implement some/all UPA checks (exact checks tbd) b) XML Schema editors that implement UPA checks will not be suitable for all DFDL models c) If DFDL annotations are removed, the resulting pure XSD will not always be valid (ie, the equivalent XML is ambiguous and can't be modelled by XML Schema 1.0) Ongoing in case another solution can be found. 29/04: Will ask DG and S Gao for oppinion before closing 038 MB: Submit response to OMG RFI for non-XML standardization 22/04: First step is for MB to mail the OGF Data Area chair to say that we want to submit 29/04: MB has been in contact with OMG and will sunbit dfdl. 039 SKK: Approach for creating Schema-For-DFDL xsds. 22/04: Resolve issue around multiple declarations needed for DFDL properties, perhaps using MB's meta approach 29/04: Don't like qualified attributes in long form. SKK to check there are no code gen implications, eg EMF. Closed actions: 025 AP: Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 18/02: SH up document concerns 25/02: SH and AP have refined proposal ready for approval. 04/03: SH and AP have further refined proposal. 11/03: discussed. suggested a simplified proposal be evaluated. 18/03: SH and AP had further discussions on simplified proposal 08/04: See minutes, review in detail for next call 15/04: See minutes, review for next call 22/04: MB mailed answers to the mailing list in response to AP's last few questions. Following agreed: 1.Should data containing the escapeEscapeCharacater cause escaping to be used if if so how should it be escaped. EEC alone isn't an active character. it has to be followed by the EC to be interpreted at all. That said, if the pair EEC EC appears in the data, then yes, we must escape the EC, with another EEC. 2.Should we only look for escapeStartString at the beginning of the data Yes, we will be restrictive/conservative for v1.0 3.Property names (everyone has their own favourite so lets just pick one.) Only changes areescapeBlockStart and escapeBlockEnd. AP to incorporate the agreed scheme into draft 0.34. 29/04: closed. Moved to workitems 034 AP: Remove redundant properties, correct old examples 08/04: No update 22/04: In progress as part of draft 0.34. 29/04: closed. Moved to work item 035 AP: Add validation ranges to spec, update specialized annotations in spec. 08/04: Raised. For draft 0.34 22/04: In progress as part of draft 0.34. 29/04: closed. Moved to work item Work items: No Item target version status 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 034 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 (A033) 034 in progress 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) not started 025 Augmented infoset and unparsing (Alan) 034 added but needs work 026 Remove duration 034 027 Calendar schemes 034 028 Validation ranges (A035) 034 029 Decimals (A020) - document unsigned type behaviour - packedDecimalSignCodes behaviour depends on NumberCheckPolicy 034 030 Remove redundant properties, fix examples. (A036) 034 031 Specialized annotations 034 032 Floating components 033 Specialized annotations 034 Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 5

Fw: UnorderedInitiated
by Steve Hanson 05 May '09

05 May '09

It has been raised that the dfdl:unorderedInitiated property is redundant, because we have speculative parsing, and because the DFDL parser can optimise by analysis without being explicitly told. This is true but saying that all children of a sequence have an initiator is also a way for the modeler to ensure that his model is correct and that all children are correctly specified. I propose that we decide whether to: a) Drop the property b) Rename the property dfdl:initiated and have it apply to any sequence, ordered or unordered There's also a question in the spec that we should resolve. (TBD: do we allow sequences with initiators to be children of an unordered sequence, or do we require the children of an unordered sequence to be elements? Conservative would be to require elements.) To which Mike had commented: Suggest sequences where children have initiators cannot be directly nested. You must use an element in this case. I would prefer that children of an unordered sequence are required to be elements in general for v1.0. I'd like to close on this as well. Let's discuss on today's call. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 29/04/2009 10:38 ----- Steve Hanson/UK/IBM 28/04/2009 15:41 To Tim Kimber/UK/IBM cc Alan Powell/UK/IBM@IBMGB Subject Re: UnorderedInitiated Here's the table from draft 0.33: sequenceKind separatorPolicy Implications unorderedInitiated ignored (suppress behavior implied) Initiators are used to identify which elements are present and which are missing. All children must have dfdl:initiator strings that are distinct and not the empty string. (Schema definition error otherwise.) nilValueInitiatorPolicy and defaultValueInitiatorPolicy must both be 'required' (Schema definition error otherwise) Setting unorderedInitiated forces extra checks to take place to ensure that an initiator would be present under all circumstances. I think this provides a direct equivalent to TDS Tagged. I'd prefer that speculative parsing was the sole mechanism used to resolve the child elements. In practice, most unordered sequences will have initiators on their children anyway. There's also this comment in the spec: (TBD: do we allow sequences with initiators to be children of an unordered sequence, or do we require the children of an unordered sequence to be elements? Conservative would be to require elements.) To which Mike had commented: Suggest sequences where children have initiators cannot be directly nested. You must use an element in this case. I would prefer that children of an unordered sequence are required to be elements in general for v1.0. Which corresponds to schema's rule for xs:all and the MRM's rule for unorderedSet, so I'm happy with that. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Tim Kimber/UK/IBM 15/04/2009 23:26 To Alan Powell/UK/IBM@IBMGB cc Steve Hanson/UK/IBM@IBMGB Subject Re: UnorderedInitiated Hi Alan, Understood. In that case, please can you add this to the list of unresolved issues in the specification. We should take pains not to include redundant properties in DFDL v1.0, because each property has a cost in terms of implementation, testing and documentation. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Alan Powell/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 15/04/2009 09:37 Subject: Re: UnorderedInitiated Tim The problem with committees is that you can't just decide by yourself to drop something. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Tim Kimber/UK/IBM To: Alan Powell/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB, Dragan Besevic/Boca Raton/IBM@IBMUS, Sunil Dandamudi/Boca Raton/IBM@IBMUS, Lorenzito Jimenez/Boca Raton/IBM@IBMUS Date: 14/04/2009 21:19 Subject: Re: UnorderedInitiated Alan, Thanks for the clarification. If that is the only reason for the property then it should be removed in v0.34. The PIF generator can easily perform that kind of optimisation ( if necessary ). This is not the only optimisation which the PIF generator might want to perform, and the specification does not supply extra properties for those other cases. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Alan Powell/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 14/04/2009 17:23 Subject: Re: UnorderedInitiated Tim There is a history. You are correct that speculative parsing should take care of it all. However originally unordered elements had to be initiated. When that was relaxed it was felt that it was worth giving the parser a chance to optimize to initiated case. Designed by committee. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Alan Powell/UK/IBM@IBMGB Date: 08/04/2009 22:54 Subject: UnorderedInitiated I can't see why this is required, but I'm probably missing something. Wouldn't the normal speculative parsing rules will make use of the initiator if it is present. Or does unorderedInitiated cause the initiator to behave as a discriminator? If so, I would prefer to leave that to the user. We don't do it for any other point of uncertainty ( there's no equivalent choiceInitiated setting ). regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Action 033 : Assertions and discriminators
by Tim Kimber 05 May '09

05 May '09

Hi all, I have this action: 033 04/03: Assert/Discriminator semantics. AP to document. TK to check uses of discriminator besides choice. I believe the rules should be: 1. A point of uncertainty is any of * an element which has minOccurs != maxOccurs * a choice * a sequence with sequenceKind="unordered" 2. Nested within the scope of a point of uncertainty, there might be other points of uncertainty. 3. A discriminator which evaluates to true resolves the nearest in-scope point of uncertainty. 4. An assertion which evaluates to false causes a processing error 5. Any processing error ( from an assertion failure or otherwise ) will cause the parser to backtrack to the nearest unresolved point of uncertainty and try the next available branch, if any. If there are no more branches available, the parser will backtrack to the next nearest unresolved point of uncertainty. 6. A processing error which reaches the root tag is reported to the host application. 7. Assertions and discriminators are allowed on any point of uncertainty ( not only on the branches of a choice ) Rationale: If we only allow a discriminator on a choice branch, then it will be difficult to model this common style of message Tagged header, minOccurs="1", maxOccurs="1" Untagged body, maxOccurs="unbounded" Tagged trailer, minOccurs="1", maxOccurs="1" An example with 3 occurrences of the body would be: HE,headerfield1,headerfield2,headerfield3 John Smith, 100, bodyfield3 John Brown, 200, bodyfield3 Elton John, 30Z, bodyfield3 TR,trailerfield1,trailerfield2,trailerfield3 And the DFDL schema would look something like this ( excuse the almost inevitable errors, this is just for completeness ): ... <xs:element name="message"> <xs:complexType dfdl:lengthKind="implicit"> <xs:sequence dfdl:separator="\r\n"> <xs:element name="header" initiator="HE,"> <xs:complexType> <xs:sequence dfdl:separator=","> <xs:element name="header1" type="xs:string"> <xs:element name="header2" type="xs:string"> <xs:element name="header3" type="xs:string"> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="body" maxOccurs="unbounded"> <xs:complexType> <xs:sequence dfdl:separator=","> <xs:element name="body1" type="xs:string"> <xs:element name="body2" type="xs:int"> <xs:element name="body3" type="xs:string"> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="trailer" initiator="TR,"> <xs:complexType> <xs:sequence dfdl:separator=","> <xs:element name="trailer1" type="xs:string"> <xs:element name="trailer2" type="xs:string"> <xs:element name="trailer3" type="xs:string"> </xs:sequence> </xs:complexType> </xs:element> </xs:sequence> </xs:complexType> </xs:element> The '30Z' value for the final occurrence of element Body2 is incorrect. It is not a valid integer, and will trigger a processing error. Without a discriminator, this failure will cause the parser to backtrack to the optional field and try the next element ( the trailer element ). The initiator will not be found, and the reported error will be "Initiator 'TR,' not found for element 'trailer'". The user would almost certainly prefer "Invalid value '30Z' for element 'body2'. Value could not be converted to simple type 'xs:int'" For this example, the discriminator would need to detect unambiguously that it really was dealing with a Body element and not a Trailer element. Due to the message style ( which is quite common ) the only way to do this is to detect that it is *not* a Trailer. I cannot think of an elegant way to do that using the facilities in v0.33 of the specification. I have raised this with Alan and Steve. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1