- dfdl-wg - lists.ogf.org

DFDL: Minutes from OGF WG call, 10 December 2008
by Alan Powell 10 Dec '08

10 Dec '08

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 13:00 GMT, 10 December 2008 Attendees Steve Hanson (IBM) Alan Powell (IBM) Appolgies Mike Beckerle(Oco) 1. Padding to minlength Discussed AP proposal and subsequent emails. SH comments "Alan - I see what you have done - you've added textStringPad but also made it the controlling property - ie, you look at textStringPad first. (This matches what IBM WTX does). That's ok - as long as you don't have textStringPad controlling trimming. So here's a proposal that reflects that. Note that I have made textPadKind applicable to strings, numbers and calendars (like textTrimKind) - I think it's the justification direction and the pad char that vary per type, not whether to pad or not. I've also dropped the stuff about a length prefixed string being trimmed to a max length - we can re-instate in the appropriate place if needed - was that to do with ensuring you couldn't pass a string whose length could not be accommodated in the prefix type (because that's not how it was worded)." Refined proposal Property Name Description textTrimKind Enum Valid values ?none?, ?padChar? Indicates whether to trim data on input. 'none': No trimming takes place 'padChar': If lengthKind results in a fixed length then the padding character can be trimmed instead, as controlled by textStringJustification. Annotation: dfdl:element (simple type ?string?) textPadKind Enum Valid values ?none?, ?fixed?, ?minimum' Indicates whether to pad data on output. ?none?: The string is expected to match the length, it is an error otherwise. 'fixed': If lengthKind results in a fixed length then the data can be padded to the fixed length, as controlled by textStringJustification. 'minimum': If lengthKind results in a fixed length then the data can be padded to the length given by dfdl:outputMinLength property, as controlled by textStringJustification. Property Name Description textStringJustification Enum Valid values ?left?, ?right?, ?center? Output: 'left': Justifies to the left and adds padding chars to the string contents if the string is too short, to the length determined by the textPadKind property. 'right': Justifies to the right and adds padding chars to the string contents if the string is too short, to the length determined by the textPadKind property. ?center?: Adds equal padding chars left and right of the string contents if the string is too short, to the length determined by the textPadKind property. It adds one extra padding char before if needed. Input: 'left': Trims any padding characters from the right of the string, according to textTrimKind property. 'right': Trims any padding characters from the left of the string, according to textTrimKind property. ?center? Trims any padding characters from the left and right of the string, according to textTrimKind property. [OMG/CAM property justification] Annotation: dfdl:element (simple type ?string?) Similar for textNumberJustification and textCalendarJustification 2. Implicit Lengths Decided to disallow zoned for Float and Double (MB to confirm) Use numberPattern for all zoned lengths instead of fixed length implied by type Simple type text (+ xml) binary String Length/maxlength n/a Float Text: numberPattern (length of positive subpattern) 32 Double Text: numberPattern 64 Decimal/Integer Text: numberPattern Zoned:numberPattern Pkd/bcd: (TotalDigits+1)/2 Long, UnsignedLong Text: numberPattern Zoned:numberPattern binary: 64 Pkd/bcd: (TotalDigits+1)/2 Int, Unsignedint Text: numberPattern Zoned:numberPattern binary: 32 Pkd/bcd: (TotalDigits+1)/2 Short, Unsignedshort Text: numberPattern Zoned:numberPattern binary: 16 Pkd/bcd: (TotalDigits+1)/2 Byte, Unsignedbyte Text: numberPattern Zoned:numberPattern binary: 8 Pkd/bcd: (TotalDigits+1)/2 DateTime Calendarpattern Date Calendarpattern Time Calendarpattern Duration Calendarpattern Boolean Length of textBooleanTrue 32 HexBinary n/a Length/maxlength 3. Zoned Decimal Discussed whether zonedDecimalSignStyle should be moved out of numberFormat. Noticed that no option for specifying overpunching or not. (added to action 020) Need more investigation of ICU behaviour. 4. AOB Next call 17 December Meeting closed, 13:00 GMT Actions raised at this meeting No Action Current Actions: No Action 003 AP: Update spec from WTX document 13/8: All resolved issues have been added to spec. Remaining unresolved items need discussion 10/9: Alan will pull out the remaining unresolved items and circulate for discussion next week 17/9: Doc distributed and partially discussed 15/10: All updateds complete except minlength 22/10: No update 12/11: Updates have been sent to wtx to review. Will chase up for comments 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 018 AP: Agree padding to minimum length 15/10: Needs documenting 22/10: No update 26/11: discussed and AP to suggest dfdl:minlength 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 023 MB: Review Schema 1.1 024 String XML type 025 Escape schemes 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) Closed actions: 033 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 004 Selectors (Suman/Alan) - Apr 30, 2008 complete 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 013 (from F2F) New scoping rules complete 014 (from F2F) Occurs, OccurSeparator changes complete 015 (from F2F) choices and Output (Mike) 016 (from F2F) xpath forward references (Alan) complete 017 (IBM WTX review) Minor agreed updates (Alan) complete 018 (IBM WTX review) Review generateNewLine (Alan) complete 020 (IBM WTX review) Special value for zero seen eg 'zero'. (Steve) complete 021 (IBM WTX review) 'EndOfData' changes (Alan) complete 022 (IBM WTX review) Unresolvable choices - infoset changes complete 023 (IBM WTX review) separatorKind=?prefix? ?infix? or ?postfix? complete 024 (IBM WTX review) StopValue clarification (Alan) complete 025 Augmented infoset and unparsing (Alan) added but needs work 026 Hidden changes complete 027 Add decimal supp to main spec complete 028 Remove extended floats complete 029 AP: enum + expression wording complete 030 All: Document new EndOfData changes complete 031 AP: Document defineTextNumber etc complete 032 AP: Document textNumberRepresenation and binaryNumberRepresentation complete complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Padding to Minimum length proposal and Implicit lengths
by Alan Powell 10 Dec '08

10 Dec '08

Mike, Steve The proposal to allow padding to minimum length on output consists of new property ouputMinLength Separate pad properties for string, number and calendar types new textxxxxPad property to control padding . Moved from from textxxxxxxJustification 1 Padding Property Name Description outputMinLength String. Only used when textStringPad, textNumberPad or textCalendarPad is ?minlength?. Specifies the minimum output length of this element using either a fixed number or an expression to refer to an element earlier in the data,. Annotation: dfdl:element (all simple types) 1.1 Properties Specific to Strings with Text representation Property Name Description textStringPad Enum Valid values ?none?, ?fixed?, ?minLength? Controls how an item is justified on output when the actual length of a text string is different from the specified length. Controls how an item is trimmed on input if textTrimKind is padChar If ?none? the string is expected to match the length. Otherwise: - If lengthKind results in a fixed length: If shorter than the specified length it is padded with the pad character. If longer than the specified length it is truncated. - If lengthKind is ?prefixed?: If the string is longer than any specified maximum length it is truncated. textStringJustification Enum Valid values ?left?, ?right?, ?center? ?center? adds equal padChars before and after the string contents if the string is too short. It adds one extra padChar before if needed. ?center? truncates equal numbers of characters before and after if the string is too long. It removes one extra character at the front of the string if needed. [OMG/CAM property justification] Annotation: dfdl:element (simple type ?string?) textStringPadCharacter String. The padding character used as the default for justification of text elements. In fixed width character sets this must contain a full single character representation. So if hex entities are used, enough hex bytes must be supplied to provided the full width of a character. In variable width character sets, this character must be a minimum-width character unless it is specified as an entity (see section TBD Entities) Annotation: dfdl:element (simple type ?string?) 1.2 Properties Specific to Number with Text representation Property Name Description textNumberPad Enum Valid values ?none?, ?fixed?, ?minLength? Controls how an item is justified on output when the actual length of a text string is different from the specified length. Controls how an item is trimmed on input if textTrimKind is padChar If ?none? the string is expected to match the length. Otherwise: - If lengthKind results in a fixed length: If shorter than the specified length it is padded with the pad character. If longer than the specified length it is truncated. - If lengthKind is ?prefixed?: If the string is longer than any specified maximum length it is truncated. textNumberJustification Enum Valid values ?left?, ?right?, ?center? ?center? adds equal padChars before and after the string contents if the string is too short. It adds one extra padChar before if needed. ?center? truncates equal numbers of characters before and after if the string is too long. It removes one extra character at the front of the string if needed. [OMG/CAM property justification] Annotation: dfdl:element (simple type ?string?) textNumberPadCharacter String. The padding character used as the default for justification of text elements. In fixed width character sets this must contain a full single character representation. So if hex entities are used, enough hex bytes must be supplied to provided the full width of a character. In variable width character sets, this character must be a minimum-width character unless it is specified as an entity (see section TBD Entities) Annotation: dfdl:element (simple type ?string?) 1.3 Properties Specific to Calendar with Text representation Property Name Description textCalendarPad Enum Valid values ?none?, ?fixed?, ?minLength? Controls how an item is justified on output when the actual length of a text string is different from the specified length. Controls how an item is trimmed on input if textTrimKind is padChar If ?none? the string is expected to match the length. Otherwise: - If lengthKind results in a fixed length: If shorter than the specified length it is padded with the pad character. If longer than the specified length it is truncated. - If lengthKind is ?prefixed?: If the string is longer than any specified maximum length it is truncated. textCalendarJustification Enum Valid values ?left?, ?right?, ?center? ?center? adds equal padChars before and after the string contents if the string is too short. It adds one extra padChar before if needed. ?center? truncates equal numbers of characters before and after if the string is too long. It removes one extra character at the front of the string if needed. [OMG/CAM property justification] Annotation: dfdl:element (simple type ?string?) textCalendarPadCharacter String. The padding character used as the default for justification of text elements. In fixed width character sets this must contain a full single character representation. So if hex entities are used, enough hex bytes must be supplied to provided the full width of a character. In variable width character sets, this character must be a minimum-width character unless it is specified as an entity (see section TBD Entities) Annotation: dfdl:element (simple type ?string?) I also completed the table of meaning of implicit lengths for different types 1.1 Lengths of simple types for lengthKind=implicit Implemented in WMB GUI helper function Simple type text (+ xml) binary String Length/maxlength n/a Float Text: numberPattern (length of positive subpattern) Zoned: TotalDigits 32 Double Text: numberPattern Zoned: TotalDigits 64 Decimal/Integer Text: numberPattern Zoned: TotalDigits Pkd/bcd: TotalDigits+1/2 Long, UnsignedLong Text: numberPattern Zoned: 19 binary: 64 Pkd/bcd: TotalDigits+1/2 Int, Unsignedint Text: numberPattern Zoned: 10 binary: 32 Pkd/bcd: TotalDigits+1/2 Short, Unsignedshort Text: numberPattern Zoned: 5 binary: 16 Pkd/bcd: TotalDigits+1/2 Byte, Unsignedbyte Text: numberPattern Zoned: 3 binary: 8 Pkd/bcd: TotalDigits+1/2 DateTime Calendarpattern Date Calendarpattern Time Calendarpattern Duration Calendarpattern Boolean Length of textBooleanTrue 32 HexBinary n/a Length/maxlength Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

DFDL: Minutes from OGF WG call, 03 December 2008
by Alan Powell 05 Dec '08

05 Dec '08

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 13:00 GMT, 03 December 2008 Attendees Steve Hanson (IBM) Alan Powell (IBM) Appolgies Mike Beckerle(Oco) 1. Padding to minlength Discussion on best way to pad to minlength for non-string text representations Decided to split textpadCharacter into textStringPadChar, textNumberPadChar and textCalenedarPadChar so that different defaults can be set up for each type, Decided that all the ICU pattern special characters should be allowed, in particular the padding character to allow padding after the sign or currency symbol. Normal dfdl padding will be applied after the pattern has been used to generate output item. Continue on next call 3. AOB Next call 10 December Meeting closed, 13:00 GMT Actions raised at this meeting No Action Current Actions: No Action 003 AP: Update spec from WTX document 13/8: All resolved issues have been added to spec. Remaining unresolved items need discussion 10/9: Alan will pull out the remaining unresolved items and circulate for discussion next week 17/9: Doc distributed and partially discussed 15/10: All updateds complete except minlength 22/10: No update 12/11: Updates have been sent to wtx to review. Will chase up for comments 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 018 AP: Agree padding to minimum length 15/10: Needs documenting 22/10: No update 26/11: discussed and AP to suggest dfdl:minlength 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 023 MB: Review Schema 1.1 024 String XML type 025 Escape schemes 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) Closed actions: 033 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 004 Selectors (Suman/Alan) - Apr 30, 2008 complete 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 013 (from F2F) New scoping rules complete 014 (from F2F) Occurs, OccurSeparator changes complete 015 (from F2F) choices and Output (Mike) 016 (from F2F) xpath forward references (Alan) complete 017 (IBM WTX review) Minor agreed updates (Alan) complete 018 (IBM WTX review) Review generateNewLine (Alan) complete 020 (IBM WTX review) Special value for zero seen eg 'zero'. (Steve) complete 021 (IBM WTX review) 'EndOfData' changes (Alan) complete 022 (IBM WTX review) Unresolvable choices - infoset changes complete 023 (IBM WTX review) separatorKind=?prefix? ?infix? or ?postfix? complete 024 (IBM WTX review) StopValue clarification (Alan) complete 025 Augmented infoset and unparsing (Alan) added but needs work 026 Hidden changes complete 027 Add decimal supp to main spec complete 028 Remove extended floats complete 029 AP: enum + expression wording complete 030 All: Document new EndOfData changes complete 031 AP: Document defineTextNumber etc complete 032 AP: Document textNumberRepresenation and binaryNumberRepresentation complete complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, 26 November 2008
by Alan Powell 26 Nov '08

26 Nov '08

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 13:00 GMT, 26 November 2008 Attendees Steve Hanson (IBM) Mike Beckerle(Oco) Alan Powell (IBM) 1. Moving to XSD 1.1 as base for DFDL Continued discussion on benefit of moving to XSD 1.1 Discussed XSD assert and XPATH Restrictions - Only on complex types - Path steps are restricted to only match information items within the element being ·assessed·. Decided XSD 1.1 assert cannot replace dfdl:assert as it occurs in more places and has more function Decided not to rename dfdl:assert even though it is similar to xsd:assert as it properly describes function XSD 1.1 assert is for validation only and has no effect on dfdl parsing and unparsing. MB to investigate whether dfdl variables can be used to allow dfdl xpath paths to be restricted in the same way as XDS 1.1 as part of the existing variable action item. 2. Action Items/ Work Items. Actions 003/018 padding to minlength - discussed WTX response and AP to investigate adding dfdl:minlength Work items 013 Scoping rules. Reviewed current description in spec and agreed section on overriding should be deleted. 3. AOB Next call 3 December Meeting closed, 13:15 GMT Actions raised at this meeting No Action Current Actions: No Action 003 AP: Update spec from WTX document 13/8: All resolved issues have been added to spec. Remaining unresolved items need discussion 10/9: Alan will pull out the remaining unresolved items and circulate for discussion next week 17/9: Doc distributed and partially discussed 15/10: All updateds complete except minlength 22/10: No update 12/11: Updates have been sent to wtx to review. Will chase up for comments 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 018 AP: Agree padding to minimum length 15/10: Needs documenting 22/10: No update 26/11: discussed and AP to suggest dfdl:minlength 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 023 MB: Review Schema 1.1 024 String XML type 025 Escape schemes 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) Closed actions: 033 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 004 Selectors (Suman/Alan) - Apr 30, 2008 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 013 (from F2F) New scoping rules complete 014 (from F2F) Occurs, OccurSeparator changes complete 015 (from F2F) choices and Output (Mike) 016 (from F2F) xpath forward references (Alan) complete 017 (IBM WTX review) Minor agreed updates (Alan) complete 018 (IBM WTX review) Review generateNewLine (Alan) complete 020 (IBM WTX review) Special value for zero seen eg 'zero'. (Steve) complete 021 (IBM WTX review) 'EndOfData' changes (Alan) 022 (IBM WTX review) Unresolvable choices - infoset changes complete 023 (IBM WTX review) separatorKind=?prefix? ?infix? or ?postfix? complete 024 (IBM WTX review) StopValue clarification (Alan) complete 025 Augmented infoset and unparsing (Alan) 026 Hidden changes complete 027 Add decimal supp to main spec complete 028 Remove extended floats complete 029 AP: enum + expression wording complete 030 All: Document new EndOfData changes 031 AP: Document defineTextNumber etc complete 032 AP: Document textNumberRepresenation and binaryNumberRepresentation complete complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for DFDL WG call - 26-11-2008
by Alan Powell 26 Nov '08

26 Nov '08

Suggested agenda for todays call 1. XSD 1.1 2. Go through action items 3. Review current state of Spec 33 I have spent some time this week updating the spec with work items. Status below. I noticed the lengthkind/ length section needs major rework. Not sure scoping section is correct: Overload vs Override, examples look the same. 033 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 004 Selectors (Suman/Alan) - Apr 30, 2008 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 013 (from F2F) New scoping rules complete 014 (from F2F) Occurs, OccurSeparator changes complete 015 (from F2F) choices and Output (Mike) 016 (from F2F) xpath forward references (Alan) complete 017 (IBM WTX review) Minor agreed updates (Alan) complete 018 (IBM WTX review) Review generateNewLine (Alan) complete 020 (IBM WTX review) Special value for zero seen eg 'zero'. (Steve) complete 021 (IBM WTX review) 'EndOfData' changes (Alan) 022 (IBM WTX review) Unresolvable choices - infoset changes complete 023 (IBM WTX review) separatorKind=?prefix? ?infix? or ?postfix? complete 024 (IBM WTX review) StopValue clarification (Alan) complete 025 Augmented infoset and unparsing (Alan) 026 Hidden changes complete 027 Add decimal supp to main spec complete 028 Remove extended floats complete 029 AP: enum + expression wording complete 030 All: Document new EndOfData changes 031 AP: Document defineTextNumber etc complete 032 AP: Document textNumberRepresenation and binaryNumberRepresentation complete Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, 15 October 2008
by Alan Powell 19 Nov '08

19 Nov '08

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 13:00 GMT, 19 November 2008 Attendees Steve Hanson (IBM) Mike Beckerle(Oco) Alan Powell (IBM) 1. Moving to XSD 1.1 as base for DFDL Went through MB email of 12/11/08 [DFDL-WG] XML Schema 1.1 implications for DFDL Agreed the following warranted further investigation - open content - Wildcard Element Name Namespace Flexibility - needs clarification of use in dfdl - Xsd:all groups - as a replacement for unordered groups - XPath Usage - Type Tables and Alternative Conditional Types - xsd:override - support primitive datatypes and facets beyond those defined in [XML Schema: Datatypes] 5. Action Items/ Work Items. Didn't go through actions. Agree a new version of the spec would be issuesd at the end of November and all included work items would be highlighted in work item table 6. AOB Next call 26 November Meeting closed, 13:10 GMT Actions raised at this meeting No Action Current Actions: No Action 003 AP: Update spec from WTX document 13/8: All resolved issues have been added to spec. Remaining unresolved items need discussion 10/9: Alan will pull out the remaining unresolved items and circulate for discussion next week 17/9: Doc distributed and partially discussed 15/10: All updateds complete except minlength 22/10: No update 12/11: Updates have been sent to wtx to review. Will chase up for comments 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 018 AP: Agree padding to minimum length 15/10: Needs documenting 22/10: No update 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 023 MB: Review Schema 1.1 024 String XML type 025 Escape schemes 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) Closed actions: 033 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 004 Selectors (Suman/Alan) - Apr 30, 2008 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 *Mostly complete* 008 (from draft 32) Property precedence for writing (Steve) - *complete but under review* 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 *proposal needs writing up* 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) * in progress * 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 013 (from F2F) New scoping rules 014 (from F2F) Occurs, OccurSeparator changes 015 (from F2F) choices and Output (Mike) 016 (from F2F) xpath forward references (Alan) *complete* 017 (IBM WTX review) Minor agreed updates (Alan) *complete* 018 (IBM WTX review) Review generateNewLine (Alan) *complete* 020 (IBM WTX review) Special value for zero seen eg 'zero'. (Steve) 021 (IBM WTX review) 'EndOfData' changes (Alan) 022 (IBM WTX review) Unresolvable choices - infoset changes *complete* 023 (IBM WTX review) separatorKind=?prefix? ?infix? or ?postfix? (Alan) 024 (IBM WTX review) StopValue clarification (Alan) 025 Augmented infoset and unparsing (Alan) 026 Hidden changes 027 Add decimal supp to main spec 028 Remove extended floats 029 AP: enum + expression wording 030 AP: Add hidden changes to spec 031 All: Document EndOfData changes 032 AP: Document defineTextNumber etc 032 AP: Document textNumberRepresenation and binaryNumberRepresentation Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, 12 November 2008
by Alan Powell 14 Nov '08

14 Nov '08

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 13:00 GMT, 12 November 2008 Attendees Steve Hanson (IBM) Mike Beckerle(Oco) Alan Powell (IBM) 1. Schema Component Designator specification changes. DFDL currently uses as part of the infoset to point to schema item associated with infoset item. Sandy Gao had alerted us to some changes in the SCD spec. MB reviewed these changes and concluded that the changes removed some limitations an so DFDL could now just reference the SCD spec without caveats. 2. Selectors Discussed Suman's comment on last call's decision on selectors. Reconfirmed the decision 3. Moving to XSD 1.1 as base for DFDL MB has done an initial review of the XSD 1.1 changes and identified some new features that might be used instead of DFDL extensions. Such as Alternative types instead of selectors and xpath instead of DFDL expression language. MB to review further. 4. Multi-level semantics. Discussed why schema had use this style and concluded it was not necessary for DFDL as we could build on the schema work. In passing noted the need to standard for examples in the dfdl spec for showing scoped properties Subsequent MB email In the DFDL spec., the problem exists of how to express the behavior of a construct in the situation where there are properties in scope surrounding the construct. I believe this kind of syntax can be used in the spec. to make clear the implications of scope: Properties in scope: lengthKind="implicit" representation="text" <sequence dfdl:separator=?,? dfdl:terminator=?;? dfdl:lengthKind="delimited"> // these are local <element name=?f1? type=?string? /> <element name=?f2? type=?string? /> </sequence> I would suggest the above syntax for the DFDL spec as a way to discuss property semantics. The only way properties can get in scope is by way of an enclosing complexType definition; but using that syntax explicitly begs the question of whether one means true lexical enclosure, can there be other enclosing complexTypes also, etc. The above style eliminates these issues. 5. Action Items/ Work Items. Went through actions. Updated below 6. AOB Next call 18 November Meeting closed, 13:10 GMT Actions raised at this meeting No Action 024 String XML type 025 Escape schemes 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) Current Actions: No Action 003 AP: Update spec from WTX document 13/8: All resolved issues have been added to spec. Remaining unresolved items need discussion 10/9: Alan will pull out the remaining unresolved items and circulate for discussion next week 17/9: Doc distributed and partially discussed 15/10: All updateds complete except minlength 22/10: No update 12/11: Updates have been sent to wtx to review. Will chase up for comments 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 018 AP: Agree padding to minimum length 15/10: Needs documenting 22/10: No update 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 023 MB: Review Schema 1.1 Closed actions: 004 MB: Mike will also author a new section in response to comments from Sandy Gao and Suman Kalia, explicitly connecting syntax with DFDL semantics 16/7: no progress 10/9: Mike not convinced this is necessary. Steve to talk to Sandy to see if it is needed for any IBM implementation that extends the existing XML parsing framework 15/10: SH to talk to Sandy 22/10: SH needs reminding of what this issue is about 12/11: Decided not needed. Closed 007 AP: enum + expression wording 13/8: no progress 10/9: No update 17/9: No update 22/10: No update 12/11: converted to work item 014 SH: Use cases for lengthKind pattern on binary fields. 10/9: Not done 17/9: No update 22/10: SH not sure he can provide a concrete use case 12/11: SH had identified need for pattern of complex structures with binary comtent which already included. Decided to allow patterns on simple binary fields for symetry. Closed 016 AP: Add hidden changes to spec 22/10: No update 12/11:converted to work item 017 All: Review EndOfData changes 22/10: SH response mailed to Mike 5/11: Discussed on call and design agreed 12/11: converted to work item 019 AP: Document defineTextNumber etc 22/10: No update 12/11:converted to work item 021 AP: Document textNumberRepresenation and binaryNumberRepresentation 22/10: No update 12/11:converted to work item 022 SH: Talk to Sandy Gao to assess use of schema 1.1 22/10: Mail sent to Sandy 12/11: track as action 023 033 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 004 Selectors (Suman/Alan) - Apr 30, 2008 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 *Mostly complete* 008 (from draft 32) Property precedence for writing (Steve) - *complete but under review* 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 *proposal needs writing up* 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) * in progress * 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 013 (from F2F) New scoping rules 014 (from F2F) Occurs, OccurSeparator changes 015 (from F2F) choices and Output (Mike) 016 (from F2F) xpath forward references (Alan) *complete* 017 (IBM WTX review) Minor agreed updates (Alan) *complete* 018 (IBM WTX review) Review generateNewLine (Alan) *complete* 020 (IBM WTX review) Special value for zero seen eg 'zero'. (Steve) 021 (IBM WTX review) 'EndOfData' changes (Alan) 022 (IBM WTX review) Unresolvable choices - infoset changes *complete* 023 (IBM WTX review) separatorKind=?prefix? ?infix? or ?postfix? (Alan) 024 (IBM WTX review) StopValue clarification (Alan) 025 Augmented infoset and unparsing (Alan) 026 Hidden changes 027 Add decimal supp to main spec 028 Remove extended floats 029 AP: enum + expression wording 030 AP: Add hidden changes to spec 031 All: Document EndOfData changes 032 AP: Document defineTextNumber etc 032 AP: Document textNumberRepresenation and binaryNumberRepresentation Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Expressing semantics of DFDL: is complexType syntax enough to capture scoping behavior impact?
by Mike Beckerle 12 Nov '08

12 Nov '08

In the DFDL spec., the problem exists of how to express the behavior of a construct in the situation where there are properties in scope surrounding the construct. Because of the large degree of simplification that the latest scoping rule change implies, (this being that properties are local unless placed on complexType forms), I believe this kind of syntax can be used in the spec. to make clear the implications of scope: <complexType dfdl:lengthKind="implicit" dfdl:representation="text" > // these are in the scope .... <sequence dfdl:separator="," dfdl:terminator=";" dfdl:lengthKind="delimited"> // these are local <element name="f1" type="string" /> <element name="f2" type="string" /> </sequence> .... </complexType> Notice my use of an enclosing complexType and ellipsis in order to achieve the notion that certain property bindings surround the example in the scope. There are some implicit things going on here. The "...." ellipsis doesn't mean "anything", it means "anything except something that changes the scoped properties" because that's the whole point of the illustration here after all! It was previously proposed by Sandy Gao, that we adopt the XSD Schema Components, and the notations they use in the specs that use them, as a means to specify DFDL semantics. The schema components have a model, and specs which discuss them must have a textual syntax for discussing instances of the model. They give the semantics of XSD in terms of translation of the XSD syntax into the model, then the semantics in terms of the model instances. This syntax for the schema components model is specifically not XSD. A key reason XSD needed this two-level approach is that they have the job of explaining XSD Schema syntax, namespaces, import/include/redefine (and now override in 1.1). By adoping the schema components model they can dispense with all issues about how multiple schema documents are assembled into a single "schema" consisting of a collection of schema components. Then they can get to the task of explaining what the collection of schema components means. Now, to me it is quite problematic to express DFDL semantics in terms of an entire intermediate language we create just because XSD did it this way. We don't need to explain the way namespaces and multi-file schema composition work. XSD has already done this. We want DFDL's spec not to reproduce that work, but to just assume that is understood from the XSD work, and focus on what is unique about DFDL. I would prefer to appeal to reason here, express the semantics in terms of familiar DFDL syntactic elements. I believe there is one tricky issue: element reference, and group reference. This is the issue of the properties being combined, so that annotions on the reference override annotations on the top level of the declaration. I believe we can describe this sufficiently without adopting a whole two-layer model. It's not worth it just for this. I lied. There is a second tricky issue: recursive types - however, we've ruled these out of DFDL v1.0, so we're ok here. Comments? Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451 | <mailto:mbeckerle.dfdl@gmail.com> mbeckerle.dfdl(a)gmail.com

1 1

DFDL: Minutes from OGF WG call, 05 Octobet 2008
by Alan Powell 12 Nov '08

12 Nov '08

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 13:00 GMT, 5 November 2008 Attendees Steve Hanson (IBM) Mike Beckerle(Oco) Alan Powell (IBM) 1. End Of Parent Discussed last proposal and subsequent emails. Decided: Only last element in a sequence can have lenghtkind=EndOfParent Sequence can also have lengthKind=EndOfParent - means take length from parent sequence. LengthKind=endOfParent on outermost sequence means end of data stream. So for the last field in a definition to extend to the end of the data stream the element has lengthKind="endOfParent" and the enclosing sequence, and all parent sequences including outermost also have lengthKind="endOfParent" Examples 1. Last element to end of data stream <xsd:sequence> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="endOfParent" /> </xsd:appinfo> </xsd:annotation> <xs:element name="mandt" type="xsd:string"/> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="explicit" length="3"/> </xsd:appinfo> </xsd:annotation> </xsd:element> .... <xs:element name="last" type="xsd:string"/> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="endOfParent"/> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> 2. last element to fixed length container <xsd:sequence> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="explicit" length="100" /> </xsd:appinfo> </xsd:annotation> <xs:element name="mandt" type="xsd:string"/> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="explicit" length="3"/> </xsd:appinfo> </xsd:annotation> </xsd:element> .... <xs:element name="last" type="xsd:string"/> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="endOfParent"/> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> 3. last element to container terminator <xsd:sequence> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="delimited" terminator="$" /> </xsd:appinfo> </xsd:annotation> <xs:element name="mandt" type="xsd:string"/> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="explicit" length="3"/> </xsd:appinfo> </xsd:annotation> </xsd:element> .... <xs:element name="last" type="xsd:string"/> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format lengthKind="endOfParent"/> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> Isn't this the same as "last " with lengthKind = "delimited"? 2. Selectors Discussed Suman's suggestion from last weeks call. - If a selector is supplied, then only dfdl:format blocks matching that selector are used - If a selector is not supplied, then only dfdl:format blocks without a selector are used Decided this is too restrictive as it requires a format block for each selector on every item even if they define the same properties so stayed with previous decision - If selector is supplied and matches one of the dfdl:format blocks then that block is used. - Otherwise the dfdl:format block with no selector is used. Reiterated that selectors are qnames. 3. Moving to XSD 1.1 as base for DFDL Benfits: Can use xsd:all instead of introducing unordered sequences. weakened wildcard rules. Agreed in principle. MB to investigate to look at the other changes in Schema 1.1 to check effect on other dfdl functions and restrictions 4. Not dicussed SCD spec changes Multi-level semntics 2. AOB Next call 12 November Meeting closed, 13:10 GMT Actions raised at this meeting No Action 023 MB: Review Schema 1.1 Current Actions: No Action 003 AP: Update spec from WTX document 13/8: All resolved issues have been added to spec. Remaining unresolved items need discussion 10/9: Alan will pull out the remaining unresolved items and circulate for discussion next week 17/9: Doc distributed and partially discussed 15/10: All updateds complete except minlength 22/10: No update 004 MB: Mike will also author a new section in response to comments from Sandy Gao and Suman Kalia, explicitly connecting syntax with DFDL semantics 16/7: no progress 10/9: Mike not convinced this is necessary. Steve to talk to Sandy to see if it is needed for any IBM implementation that extends the existing XML parsing framework 15/10: SH to talk to Sandy 22/10: SH needs reminding of what this issue is about 007 AP: enum + expression wording 13/8: no progress 10/9: No update 17/9: No update 22/10: No update 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 014 SH: Use cases for lengthKind pattern on binary fields. 10/9: Not done 17/9: No update 22/10: SH not sure he can provide a concrete use case 016 AP: Add hidden changes to spec 22/10: No update 017 All: Review EndOfData changes 22/10: SH response mailed to Mike 018 AP: Agree padding to minimum length 15/10: Needs documenting 22/10: No update 019 AP: Document defineTextNumber etc 22/10: No update 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 021 AP: Document textNumberRepresenation and binaryNumberRepresentation 22/10: No update 022 SH: Talk to Sandy Gao to assess use of schema 1.1 22/10: Mail sent to Sandy Closed actions: 033 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 004 Selectors (Suman/Alan) - Apr 30, 2008 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 *Mostly complete* 008 (from draft 32) Property precedence for writing (Steve) - *complete but under review* 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 *proposal needs writing up* 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) * in progress * 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 013 (from F2F) New scoping rules 014 (from F2F) Occurs, OccurSeparator changes 015 (from F2F) choices and Output (Mike) 016 (from F2F) xpath forward references (Alan) *complete* 017 (IBM WTX review) Minor agreed updates (Alan) *complete* 018 (IBM WTX review) Review generateNewLine (Alan) *complete* 020 (IBM WTX review) Special value for zero seen eg 'zero'. (Steve) 021 (IBM WTX review) 'EndOfData' changes (Alan) 022 (IBM WTX review) Unresolvable choices - infoset changes *complete* 023 (IBM WTX review) separatorKind=?prefix? ?infix? or ?postfix? (Alan) 024 (IBM WTX review) StopValue clarification (Alan) 025 Augmented infoset and unparsing (Alan) 026 Hidden changes 027 Add decimal supp to main spec 028 Remove extended floats Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

5 6

DFDL & support for XML Schema 1.1
by Steve Hanson 29 Oct '08

29 Oct '08

Sandy Gao at IBM has looked at the reasons for moving to schema 1.1 and concluded that DFDL should move. Grounds: - Weakened wildcard support - there is less ambiguity in 1.1 because an optional element x followed by a wildcard will match x, instead of giving a UPA error - this will enable DFDL to model more formats - Relaxation of <xs:all> means that it is suitable for DFDL use - this will allow us to drop the dfdl:unordered property - In Schema 1.0, annotations are lost on particles, but they are needed by DFDL. Schema 1.1 captures all annotations. Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2