- dfdl-wg - lists.ogf.org

Rescheduled: OGF DFDL Working Group weekly call (24 Mar 13:00 GMT in Participant Passcode: 8190865 )
by Steve Hanson 22 Mar '10

22 Mar '10

IBM Hursley attendees - DE3S09 Reducing the time to 1 hour now that spec has been sent to OGF

1 0

WG call this week
by Alan Powell 22 Mar '10

22 Mar '10

I cannot make the WG call this week and as we don't have anything urgent to discuss I propose that we cancel it. Let me know if you would prefer it to go ahead. Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Minutes for OGF DFDL Working Group Call, March 17-2010
by Alan Powell 18 Mar '10

18 Mar '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, March 17-2010 Attendees Suman Kalia (IBM) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Stephanie Fetzer (IBM) Mike Beckerle (Oco) Apologies Tim Kimber(IBM) Steve H reported on the DFDL presentation at OGF 28 Overall OGF registered attendees thought to be around 250. Number at DFDL session: 7. Apart from Erwin (Data AD), DFDL was new to everyone. Notes from session (from minutes taken by Erwin Laure based on questions asked): Validation on input and output? Means conforming to the schema defined (e.g. integer between 0 and 100). dfdl assert could be used for asserting correctness of the data. Further complex validation would be via an external step (eg, Schematron invocation). Using DFDL to model data structures generated by OO code, particularly use of inheritance? Could be a use case for allowing complex type inheritance in DFDL in the future. Data often comes with units, infoset would want to reflect this? Could be achieved using expressions, dfdl:inputValueCalc and dfdl:hidden to apply a scaling factor based on units. What's the efficiency of DFDL? Particularly, can tests be turned off for fast read/write? The spec defines the behavior of the parser but not how to implement it. For instance, validation is not mandatory. Open source reference implementation would be good to have. (Lots of nodding). Scientific floating point data compresses badly. Knowing data structure can allow a more intelligent compression. DFDL is not intending to do transformations but that could be done on top. Encryption/compression could also be a use case for multi-layers, or for additional functions in the expression language. Need to reach out to DAIS-WG and DR-WG for (public) comments and to see whether DFDL will be actively used by those groups. Open source implementation would obviously help here. Comment on the spec: Should be a proposed recommendation (GFD-P-R) not informational (GFD-I). Draft 040 has been submitted to OGF and will be reviewed by the technical committee on March 30th. It is expected to go out for public comment in early April. There needs to be some publicity to ensure that enough comments are made. Mike suggested a press release from his company with IBM comment. Action raised May be able to get IBM representatives on other standards bodies such as ACCORD. OMG, to interest their members, Should contact all the contributors to the WG to ask them to review. Mike will update the information he sent in response to an RFI from OMG 1 Process for dealing with internal issues during Public comment phase. The public comments process will be used to make updates 2. Nils and Defaults during unparsing The table 17 in section 13.16.2 was corrected in draft 40 but here are still some ambiguities, for example when nil is the default. Logical Value nilValueInitiatorPolicy Has default value specified missingValueInitiatorPolicy initiator region contains content region contains Nil (implies nillable) prohibited don't care don't care empty representation of nil based on nilKind, nilValue, etc. required initiator string "" (empty string) Note that this implies that the element type is xs:string don't care prohibited empty empty string required initiator string a non-nil non-empty-string value don't care don't care initiator string The representation of the logical value Not supplied don?t care Yes (non-empty sting) don't care Initiator string The representation of the default value. Yes (empty string) prohibited empty empty required initiator string empty Alan will go update the table. 3 dfdl:choiceKind The main issues are: a) The calculation of the length of the longest branch is not obvious. b) The length units to use - the dfdl:lengthUnits property does not exist on a choice c) The name could be better Proposal is therefore to retain the property but to: i) State the conditions that must apply to use this property, and enforce them in the validator => schema definition error otherwise ii) Decouple the choice from its parent by calculating the length of each branch based solely on the properties of the branches components, irrespective of any parent dfdl:lengthKind Alan document the problem. Name change agreed. 4 DFDL time functions fn:timezone-from-dateTime Returns the timezone from an xs:dateTime value. fn:timezone-from-date Returns the timezone from an xs:date value. fn:timezone-from-time Returns the timezone from an xs:time value. Function Meaning fn:adjust-dateTime-to-timezone Adjusts an xs:dateTime value to a specific timezone, or to no timezone at all. fn:adjust-date-to-timezone Adjusts an xs:date value to a specific timezone, or to no timezone at all. fn:adjust-time-to-timezone Adjusts an xs:time value to a specific timezone, or to no timezone at all. All return an xs:duration. Do we need these functions? As there is no known use case for these functions they will be dropped. Meeting closed, 14:10 Next call Wednesday 17 March January 2010 13:00 UK (9:00 ET) Call will be for one hour only NOTE: East coast is 4 hours behind the UK for the next two weeks Next action: 087 Actions raised at this meeting No Action 085 ALL: publicize Public comments phase to ensure a good review.. 086 AP: Nils and Defaults during unparsing - update table Current Actions: No Action 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 24/03: No progress 03/03: Discussions have been taking place on the subset of tests that will be provided. 10/03: work is progressing 17/03: work is progressin 084 Check behaviour of dfdl:inputValueCalc and outputValueCalc. 085 ALL: publicize Public comments phase to ensure a good review.. 086 AP: Nils and Defaults during unparsing - update table Closed actions No Action Work items: No Item target version status 005 Improvements on property descriptions not started 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 070 Write DFDL primer 071 Write test cases. 083 Implement RFC2116 097 Remove functions that returns duration Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Minutes from DFDL session at OGF 28
by Steve Hanson 17 Mar '10

17 Mar '10

Overall OGF registered attendees thought to be around 250. Number at DFDL session: 7. Apart from Erwin (Data AD), DFDL was new to everyone. Notes from session (from minutes taken by Erwin Laure based on questions asked): Validation on input and output? Means conforming to the schema defined (e.g. integer between 0 and 100). dfdl assert could be used for asserting correctness of the data. Further complex validation would be via an external step (eg, Schematron invocation). Using DFDL to model data structures generated by OO code, particularly use of inheritance? Could be a use case for allowing complex type inheritance in DFDL in the future. Data often comes with units, infoset would want to reflect this? Could be achieved using expressions, dfdl:inputValueCalc and dfdl:hidden to apply a scaling factor based on units. What's the efficiency of DFDL? Particularly, can tests be turned off for fast read/write? The spec defines the behavior of the parser but not how to implement it. For instance, validation is not mandatory. Open source reference implementation would be good to have. (Lots of nodding). Scientific floating point data compresses badly. Knowing data structure can allow a more intelligent compression. DFDL is not intending to do transformations but that could be done on top. Encryption/compression could also be a use case for multi-layers, or for additional functions in the expression language. Need to reach out to DAIS-WG and DR-WG for (public) comments and to see whether DFDL will be actively used by those groups. Open source implementation would obviously help here. Comment on the spec: Should be a proposed recommendation (GFD-P-R) not informational (GFD-I). Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG call 3 March 2010- 13:00 UK (9:00 ET)
by Alan Powell 16 Mar '10

16 Mar '10

NOTE: The east coast is 4 hours behind the UK for the next two weeks so start time is 9:00 am ET 1 Process for dealing with internal issues during Public comment phase. 2. Nils and Defaults during unparsing The table 17 in section 13.16.2 was corrected in draft 40 but here are still some ambiguities, for example when nil is the default. Logical Value nilValueInitiatorPolicy Has default value specified missingValueInitiatorPolicy initiator region contains content region contains Nil (implies nillable) prohibited don't care don't care empty representation of nil based on nilKind, nilValue, etc. required initiator string "" (empty string) Note that this implies that the element type is xs:string don't care prohibited empty empty string required initiator string a non-nil non-empty-string value don't care don't care initiator string The representation of the logical value Not supplied don?t care Yes (non-empty sting) don't care Initiator string The representation of the default value. Yes (empty string) prohibited empty empty required initiator string empty 3 dfdl:choiceKind The main issues are: a) The calculation of the length of the longest branch is not obvious. b) The length units to use - the dfdl:lengthUnits property does not exist on a choice c) The name could be better Proposal is therefore to retain the property but to: i) State the conditions that must apply to use this property, and enforce them in the validator => schema definition error otherwise ii) Decouple the choice from its parent by calculating the length of each branch based solely on the properties of the branches components, irrespective of any parent dfdl:lengthKind 4 DFDL time functions fn:timezone-from-dateTime Returns the timezone from an xs:dateTime value. fn:timezone-from-date Returns the timezone from an xs:date value. fn:timezone-from-time Returns the timezone from an xs:time value. Function Meaning fn:adjust-dateTime-to-timezone Adjusts an xs:dateTime value to a specific timezone, or to no timezone at all. fn:adjust-date-to-timezone Adjusts an xs:date value to a specific timezone, or to no timezone at all. fn:adjust-time-to-timezone Adjusts an xs:time value to a specific timezone, or to no timezone at all. All return an xs:duration. Do we need these functions? 5 Current Actions: No Action 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 24/03: No progress 03/03: Discussions have been taking place on the subset of tests that will be provided. 10/03: work is progressing 084 Check behaviour of dfdl:inputValueCalc and outputValueCalc. Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Minutes for OGF DFDL Working Group Call, March 10-2010
by Alan Powell 16 Mar '10

16 Mar '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, March 10-2010 Attendees Suman Kalia (IBM) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Stephanie Fetzer (IBM) Tim Kimber(IBM) Apologies Mike Beckerle (Oco) 1. Spec issues documentFinalTerminatorCanBeMissing: Let's try to avoid creating another property for the postfix separator scenario. I think this property provides a way of modelling the data naturally. We can recommend use of infix-with-a-terminator rather than 'postfix' if the final terminator can be missing. WG deferred any changes the property at this stage dfdl:lengthKind endOfParent 'endOfParent' has almost the same meaning as 'delimited' so should have the same semantics. · the item?s terminator (if specified) · an enclosing construct?s separator or terminator · the end of an enclosing construct designated by its known length · the end of the data stream The effect would be the the element could be ended by the nearest known length parent not just the immediate parent. Also the immediate parent could have lengthKind 'implicit' Agreed with the definition and that dfdl:lengthKind 'endOfParent should apply to simpleTypes only. dfdl:lengthUnits 'characaters' LengthUnits 'characters' is currently for simple representation 'text' only. Should it be allowed for complex? Agreed dfdl:lengthUnits should be allowed on complex elements. The characters are in the encoding of the complex element. The endocings of the children must be the same as the parent.. Selectors Should they be moved to an appendix. Agreed should moved DFDL Expression Syntax Value comparison operators are 'eq' 'lt' 'gt' etc not '=' '<' '>' because these were defined poorly in XPATH v1 and replaced in XPATH V2. Unfortunately most examples us '=' etc The examples need changing and '=' '<' will not be accepted Syntax of expressions in string literals. Add A DFDL string literal which begins with { and does not end with } does not need to have its leading { escaped. The syntax rules for expressions are · Must start with a '{' in the first position and end with '}' in the last position. · '{' in any other position if treated as a literal · '}' in any position other than the last position is treated as a literal. · '{{' as the first characters are treated as the literal '{' and not a DFDL expression. IgnoreCase Should ignoreCase apply to numberInfinityRep and numberNaNRep Agreed Escapes. Should escapeCharacter, escapeEscapeCharacater etc allow DFDL expressions. Agreed OutputNewLine should allow an expression Agreed Clarify dfdl: InputValueCalc and outputValueCalc It is a schema definition error if both inputvalueCalc and outputValueCalc are specified. InputValueCalc is examined on unparsing so needs to be added to unparsing property precedence. Action to review inputValueCalc and putputValueCalc 2. Current Actions: Updated below 3 Status of specification (for OGF28) Draft 40 will be published by tomorrow and submitted to OGF on friday. Meeting closed, 14:10 Next call Wednesday 10 March January 2010 13:00 UK (9:00 ET) NOTE: East coast is 4 hours behind the UK for the next two weeks Next action: 085 Actions raised at this meeting No Action 084 Check behaviour of dfdl:inputValueCalc and outputValueCalc. Current Actions: No Action 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 24/03: No progress 03/03: Discussions have been taking place on the subset of tests that will be provided. 10/03: work is progressing 084 Check behaviour of dfdl:inputValueCalc and outputValueCalc. Closed actions No Action 080 AP:Clarify semantics of fn:poisition and fn:count 17/02: no progress 24/03: No progress 03/03: no progress. There are other functions which return a duration which need investgating 10/03: changed to dfdl:position and dfdl:count. closed Work items: No Item target version status 005 Improvements on property descriptions not started 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 070 Write DFDL primer 071 Write test cases. 083 Implement RFC2116 Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF 28: DFDL 1.0 Public Comment session
by Steve Hanson 15 Mar '10

15 Mar '10

Finalised details of this Wednesday's session at OGF 28 in Munich are here http://www.ogf.org/gf/event_schedule/index.php?id=1933, including slides. Regards Steve Hanson Programming Model Architect, WebSphere Message Broker, Co-Chair, OGF DFDL WG Hursley, UK, Internet: smh(a)uk.ibm.com, Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Data Format Description Language specification submitted to OGF for review.
by Alan Powell 12 Mar '10

12 Mar '10

All The Data Format Description Language V1 specification has been submitted to the OGF committee for review. There were a couple of minor editorial changed and a not-so-minor update to the grammar production table from the draft sent out yesterday. Updated versions can be found at Draft 040: https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… Submitted version: https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL V1 Draft 040 and OGF submission copies available
by Alan Powell 11 Mar '10

11 Mar '10

DFDL v1 draft 040 is available at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… The version to be submitted to OGF is at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… This is the same as draft 040 but with the revision history removed and all the changes accepted If there are no serious comments, this draft will be submitted to OGF tomorrow afternoon. As far as I am aware all the issues and most of the editorial comments have been incorporated. Revision History Latest entry at the top please Version Author/ Contributor History Date(yyyy-mm-dd) 040 Alan Powell Mike Beckerle Steve Hanson Tim Kimber dfdl:textNumberRepresentation changed to textNumberRep numberXXX from textNumberFormat changed to textNumberXXX Fixed up grammar (double tables), dropped grammar for simple types (wasn't used anywhere) replacing it with a single grammar terminal 'simpleRepresentation' which is used for all the simple types. Modified properties to refer to grammar terminals (some - not done comprehensively). Moved sections from sequences that have to do with specified length, and pattern length that rightfully aren't just about sequences. These were moved up to the general section about Framing and length. Change all boolean properties to enum 'yes''no' Added textNumberGroupSeparator Corrected Nils and Deafults on Unparsing table Clarified the valid types for properties Moved selectors to appendix Added dfdl:inputValueCalc to unparsing property precedence. Changed all examples to use 'eq' 'gt' rather than '=' '>' Allow dfdl:lengthUnits 'characters' on complex elements Allow escapeCharacter and escapeEscapeCharacater to be an expression. outputNewLine can be an expression fixed examples added copyright statements for ICU and XPATH change fn:count and fn:position to dfdl:count and dfdl:position Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG call 10 March 2010- 13:00 UK (8:00 ET)
by Alan Powell 10 Mar '10

10 Mar '10

1. Spec issues documentFinalTerminatorCanBeMissing: Let's try to avoid creating another property for the postfix separator scenario. I think this property provides a way of modelling the data naturally. We can recommend use of infix-with-a-terminator rather than 'postfix' if the final terminator can be missing. We didn't come to a conclusion on the alst WG call. dfdl:lengthKind endOfParent 'endOfParent' has almost the same meaning as 'delimited' so should have the same semantics. · the item?s terminator (if specified) · an enclosing construct?s separator or terminator · the end of an enclosing construct designated by its known length · the end of the data stream The effect would be the the element could be ended by the nearest known length parent not just the immediate parent. Also the immediate parent could have lengthKind 'implicit' dfdl:lengthKind endOfParent LengthUnits 'characters' is currently for simple representation 'text' only. Should it be allowed for complex? Selectors Should they be moved to an appendix. DFDL Expression Syntax Value comparison operators are 'eq' 'lt' 'gt' etc not '=' '<' '>' because these were defined poorly in XPATH v1 and replaced in XPATH V2. Unfortunately most examples us '=' etc Syntax of expressions in string literals. Add A DFDL string literal which begins with { and does not end with } does not need to have its leading { escaped. IgnoreCase Should ignoreCase apply to numberInfinityRep and numberNaNRep Escapes. Should escapeCharacter, escapeEscapeCharacater etc allow DFDL expressions. 2. Current Actions: No Action 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 13/01: no update 20/01: no update 27/01: no progress 29/01: no progress 03/02: IBM is still investigating 10/02: IBM is still investigating 17/02: IBM is willing in principle to publish the test case format and some of the test cases. May need some time to build a 'compliance suite' 24/03: No progress 03/03: Discussions have been taking place on the subset of tests that will be provided. 080 AP:Clarify semantics of fn:poisition and fn:count 17/02: no progress 24/03: No progress 03/03: no progress. There are other functions which return a duration which need investgating 3 Status of specification (for OGF28) Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0