January 2009 - dfdl-wg - lists.ogf.org

DFDL: Minutes from OGF WG call, 21 January 2009
by Alan Powell 30 Jan '09

30 Jan '09

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 14:00 GMT, 28 January 2009 Attendees Alan Powell (IBM) Mike Beckerle(Oco) Steve Hanson (IBM) 1. XSD 1.1 SH and AP will arrange a call with Sandy Gao 2. Escape Schemes Discussed proposal distributed by AP Escape schemes apply to all logical types with text representation As with other defnexxxFormats it is the escapeSchemeRefermce that is in scope not the properties within the escape scheme. SH proposed allowing multiple escape refs for each alternate escape delimiter pairs rather than defining them in on escape scheme. When escapeSchemeRef is blank then no escape scheme applies Discussion about whether escape schemes apply to known length as well as variable length text fields. Earlier decision to scan known length text fields for parent syntax items implies need to allow escape. Suggested syntax 1. escapeSchemeRef= blank means no escaping and don't scan 2. escapeSchemeRef=something but escape delimiters and escapecharacter are blank means no escape but scan for syntax 3. escapeSchemeRef=something but escape delimiters and escapeCharacter are something means escape and scan for syntax Subsequent discussions indicate that some products scan if there are syntax items in scope even if known length. Needs further discussion. Need to discuss if comment escapes should be supported. 4. AOB Next call 04 February 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action 031 Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 023 MB: Review Schema 1.1 29/1: AP and SH to talk to Sandy Gao 024 String XML type 025 Escape schemes 21/1: discussed requirements 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 030 AP: confirm with WTX that can drop duration 21/6: WTX confirm that they do not have a duration type so do not need it in dfdl. Will drop from spec. Closed Closed actions: 030 AP: confirm with WTX that can drop duration 21/6: WTX confirm that they do not have a duration type so do not need it in dfdl. Will drop from spec. Closed 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Introduction and Participation
by Dave Glick 30 Jan '09

30 Jan '09

Hi, I would like to briefly introduce myself. I am currently leading a SBIR project for the US Navy (SBIR is a program designed to grow small businesses and emphasizes innovative research and application). A large part of what I'm working on involves the conversion of arbitrary binary data to XML for processing and back again. After evaluating several alternatives, I have come to believe that DFDL is pretty close to perfect for what we need. As there are no commercially available DFDL parsers available, part of my project will be to create such a parser (at least for the subset of the standard relevant to my effort - hopefully building to a complete implementation at some point). To that end, I will be approaching the standard from the perspective of an implementation developer and hope to be able to participate in the working group in whatever ways are useful. The level of my involvement depends on funding (relatively small right now), but I should know within a short timeframe (2 months or so) whether I will be funded for the next several years at a much higher level. In any case, I am already generating questions and comments for the most recently published draft (033). Would it be valuable for me to send them to someone? How else can I be of assistance and participate now and in the future? Thanks, Dave --- David Glick | dglick(a)dracorp.com <mailto:dglick@dracorp.com> | 703.299.0700 x212 Data Research and Analysis Corp. | www.dracorp.com <http://www.dracorp.com>

2 1

DFDL: Minutes from OGF WG call, 21 January 2009
by Alan Powell 28 Jan '09

28 Jan '09

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 14:00 GMT, 21 January 2009 Attendees Alan Powell (IBM) Mike Beckerle(Oco) Apologies Steve Hanson (IBM) 1. XSD 1.1 Deferred to next call 2. Calendar formats Discussed updated (v4) supplement emailed by AP Agreed millisec/secSinceEpoc cannot be implied by length of logical data so need seperate enumerations. Observed that these options were really combination of 3 properties binary, length and sec/millisec. Suggested renaming to binarySeconds and binaryMilliseconds Packed calendars: decided that need to be able to specify at least the packedDecimalSignCodes property rather than assuming a default so reference will be added to calendar description Locale needs to be specified for numberformats and calendarFormats (didn't identify any other areas) as it modifies the behaviour of ICU. Decided to add locale to numberFormat and CalendarFormat 3. Escape Schemes Agreed need for multiple escape delimiter pairs but not nested. Need an escape for escape character even though in most cases this will be the same character, eg /n //, There are some formats that have a different escape, eg /n &/. Only need single escape characters and one level of escape characters. Discussed how to deal with comments of the form /* comment */ where the escape delimiters are also the initiator and terminator of the field. Semantic needed is 'only look for field terminator not any parent terminator or any other syntax elements'. May fall out naturally from the speculative parsing rules. Need further discussion. 4. AOB Next call 28 January 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action 031 Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 023 MB: Review Schema 1.1 024 String XML type 025 Escape schemes 21/1: discussed requirements 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 030 AP: confirm with WTX that can drop duration 21/6: WTX confirm that they do not have a duration type so do not need it in dfdl. Will drop from spec. Closed Closed actions: 030 AP: confirm with WTX that can drop duration 21/6: WTX confirm that they do not have a duration type so do not need it in dfdl. Will drop from spec. Closed 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Escape Scheme propsal
by Alan Powell 26 Jan '09

26 Jan '09

All Attached is a proposal for specifying escape schemes Points of discussion As specified escaping applies only to String logical types represented as text. Is it needed for Numbers or Calendars types with text representation? I think not. Message Broker only escapes variable length text fields but I have not made that restriction. Is the interaction between escaping and padding clear enough? Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Fw: Thirdn pass at CalendarFormat
by Alan Powell 20 Jan '09

20 Jan '09

Updated proposal attached. Still need to decide how to specify locale for calendarformat and numberformat Main changes: dropped zoned (and therefore textCalendarRep) dropped binary for c-time structure Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Alan Powell/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, mbeckerle.dfdl(a)gmail.com Date: 18/12/2008 16:54 Subject: Fw: Second pass at CalendarFormat Guys I have added more info to the calendar supplement. see changes table. An open question is how to deal with locales. ICU by default uses the local locale which is probably not correct for dfdl. It is also not good that we a property with an enumeration of Monday. Tuesday etc. Also need to consider nulls, defaults and zeroRepresentation (added) I couldn't decide if it was better to have separate calendarformats for text and binary as we do for numberformats. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 ----- Forwarded by Alan Powell/UK/IBM on 18/12/2008 16:47 ----- From: Alan Powell/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, mbeckerle.dfdl(a)gmail.com Date: 12/12/2008 16:42 Subject: First pass at CalendarFormat I have done an initial updated of the Data/time supplement following the numberFormat pattern. Comments [attachment "ggf-dfdl-supplement-date-time-properties-v1.0-002.doc" deleted by Alan Powell/UK/IBM] Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU #### ggf-dfdl-supplement-date-time-properties-v1.0-003.doc moved to MyAttachments Repository V3.8 () on 06 January 2009 by Alan Powell. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, 16 January 2009
by Alan Powell 20 Jan '09

20 Jan '09

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 15:00 GMT, 16 January 2009 Attendees Steve Hanson (IBM) Alan Powell (IBM) Mike Beckerle(Oco) 1. Non-supported XSD constructs in DFDL schema It was agreed that dfdl v1 will not allow non-supported XSD constructs, such as attributes, to exist in dfdl schema. That is they will be treated as errors rather than ignored. 2. XSD 1.1 For next meeting we will revisit the list of areas to be investigated produced by MB 3. Padding to minlength Proposal documented in last call ( 10 December 2008) was approved 4. Implicit Lengths Decided to disallow zoned for Float and Double. Approved Use numberPattern for all zoned lengths instead of fixed length implied by type - Approved Discussed using virtual decimal point and/or farctional digits for implied length of decimal. Decided against. Need to complete for Date/Time, Date, Time Drop DFDL support for duration as a logical type. Action: check with WTX Simple type text (+ xml) binary String Length/maxlength n/a Float Text: numberPattern (length of positive subpattern) 32 Double Text: numberPattern 64 Decimal/Integer Text: numberPattern Zoned:numberPattern Pkd/bcd: (TotalDigits+1)/2 Long, UnsignedLong Text: numberPattern Zoned:numberPattern binary: 64 Pkd/bcd: (TotalDigits+1)/2 Int, Unsignedint Text: numberPattern Zoned:numberPattern binary: 32 Pkd/bcd: (TotalDigits+1)/2 Short, Unsignedshort Text: numberPattern Zoned:numberPattern binary: 16 Pkd/bcd: (TotalDigits+1)/2 Byte, Unsignedbyte Text: numberPattern Zoned:numberPattern binary: 8 Pkd/bcd: (TotalDigits+1)/2 DateTime Calendarpattern Date Calendarpattern Time Calendarpattern Duration Calendarpattern Boolean Length of textBooleanTrue 32 HexBinary n/a Length/maxlength 5. Calendar formats Briefly discussed supplement emailed by AP. Has been updated and will be redistributed Note change from SecondsSinceEpoc and milliSecondsSinceEpoc to binary not implemented as it was based on using length of logical type to decide which to use but DateTime, Date and Time types do not have variable length. 3. Zoned Decimal Discussed behaviour of IBM iSeries not overpunching positive numbers. Approved changes. 4. AOB Next call 21 January 14:00 Meeting closed, 16:00 GMT Actions raised at this meeting No Action 030 AP: confirm with WTX that can drop duration Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 023 MB: Review Schema 1.1 024 String XML type 025 Escape schemes 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 030 AP: confirm with WTX that can drop duration Closed actions: 003 AP: Update spec from WTX document 13/8: All resolved issues have been added to spec. Remaining unresolved items need discussion 10/9: Alan will pull out the remaining unresolved items and circulate for discussion next week 17/9: Doc distributed and partially discussed 15/10: All updateds complete except minlength 22/10: No update 12/11: Updates have been sent to wtx to review. Will chase up for comments 16/1: Updates have been made to spec v033. Closed 018 AP: Agree padding to minimum length 15/10: Needs documenting 22/10: No update 26/11: discussed and AP to suggest dfdl:minlength 16/1: proposal approved. Closed 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Rescheduled: OGF DFDL Working Group weekly call (16 Jan 15:00 GMT in DE2K09)
by Steve Hanson 13 Jan '09

13 Jan '09

Participant Passcode: 8190865 Country Toll Numbers Freephone/Toll Free Number AUSTRIA 43-1-928-9652 BELGIUM 32-2-400-6791 0800-4-9612 DENMARK 45-7014-0237 FRANCE LYON: 33-4-26-69-12-74 080-511-3725 FRANCE MARSEILLE: 33-4-86-06-00-74 080-511-3725 FRANCE PARIS: 33-1-70-70-81-58 080-511-3725 GERMANY 49-69-2222-20363 0800-100-6504 IRELAND 353-1-247-7768 ITALY 39-02-3600-3648 800-985-679 LUXEMBOURG 352-27-000-1312 NETHERLANDS 31-20-713-2741 0800-020-3237 SPAIN 34-91-414-1542 800-099-667 SWEDEN 46-8-566-10-780 SWITZERLAND 41-44-580-3334 0800-000-516 UNITED KINGDOM BIRMINGHAM: 44-121-210-9014 0800-279-9193 UNITED KINGDOM GLASGOW: 44-141-202-3214 0800-279-9193 UNITED KINGDOM LEEDS: 44-113-301-2114 0800-279-9193 UNITED KINGDOM LONDON: 44-20-7019-0808 0800-279-9193 UNITED KINGDOM MANCHESTER: 44-161-601-1414 0800-279-9193 USA 866-505-4412 Rescheduling just this week's call to Friday. Same call-in details.

1 0

Agenda for DFDL WG call - 07-01-2009
by Alan Powell 06 Jan '09

06 Jan '09

Suggested agenda (from cancelled December call) 1. Moving to XSD 1.1 as base for DFDL 2. Padding to minlength Comments on last call discussion 3. Implicit Lengths Comments on last call discussion 4. Calendar format. review AP email 5. Review actions 6. AOB Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL Example for 2D arrays
by Jyotishman Pathak 06 Jan '09

06 Jan '09

Hello all, I am a newbie in DFDL, and trying to grasp some of the key aspects going through the latest core spec: http://forge.gridforum.org/sf/go/doc15397?nav=1 Pardon me if this question is nonsensical, but I wonder if there are any examples of how a DFDL schema/document will look like if Matrix.data is representing a 2D array? Or if there are any sample DFDL documents for an image file (jpg, png etc.). The simple example that you have in pages 19--20 of the core spec is indeed helpful to understand the guts of the idea, but I would like to see somewhat more complex examples. I look forward to your help. Thanks, - Jyoti

2 1