- dfdl-wg - lists.ogf.org

DFDL: Minutes from OGF WG call, 16 Nov 2007
by Ian W Parkinson 19 Nov '07

19 Nov '07

Open Grid Forum: Data Format Description Language Working Group Working Group Conference Call 14:00 GMT, 16 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Suman Kalia (IBM) Ian Parkinson (IBM) Alan Powell (IBM) 1. Introduction Mike called the meeting to discuss the revised Data Grammar portion of the DFDL specification, which he had previously circulated. This was discussed on the regular, weekly, conference call on 14th November, in Mike's absence. Mike will not be able to attend the next scheduled conference call. 2. Review of DFDL Data Grammar The meeting discussed the following points: As noted in the previous meeting, both the ComplexContent and ComplexElement productions include a "Prefix" element. Mike stated that this redundancy does exist in DFDL, but it sometimes necessary, e.g. for embedded groups. Steve suggested that, if this did need to be changed, we would do so with restrictions rather than a change to the productions. Suman suggested that the present grammar specification does not satisfactorily cover the case where the start of an array needs to be aligned, with individual array elements automatically being aligned. Mike suggested that this can be dealt with, in a DFDL schema, by encapsulating the array in an element or sequence.This would require the author to build additional encapsulation around the array, which Suman felt would be unnatural, and suggested it would be more natural to introduce a property to explicitly deal with this situation. Mike observed that the XML specification treats arrays as simply adjacent elements, so we should minimize the number of properties which affect an entire array rather than each individual element. Mike and Suman will collaborate on a concrete proposal to address this. Mike expressed discomfort with the use of "binary" and "opaque" to describe raw data, as these words ,ay be ambiguous. "opaque" in particular may refer to unparsed data, or to anyType elements. Suggested alternatives included "octets", "bits", "raw" and "blob". The SequenceContent production currently contains a "finalUnused" element. Mike wondered whether this would best be moved to the Sequence production. Suman felt that there would then be issues with prefixes and suffixes. The meeting saw no good reason to change this. Precedence order needs to be defined. Steve and Geoff will attempt to define a static order. Regarding the query raised at the previous meeting concerning initial prefix Separators and final postfix Separators, Mike confirmed that these had been removed following comments in a previous draft of the document. They had originally been intended to support certain properties which could be now be defined using other features of the grammar. Meeting closed 15:00 GMT Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Grammar as adapted to infoset
by Steve Hanson 19 Nov '07

19 Nov '07

Or allow xs:choice to carry dfdl:length & dfdl:lengthKind. Then lengthKind="implicit" could mean the length is the longest of the modelled alternatives. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle/Worcester/IBM@IBMUS 16/11/2007 19:29 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset There are of course alternatives. E.g., a hidden or non-hidden field which is a byte array or hexBinary to absorb the padding bytes at the ends of the shorter variants. Or one could put trailing skip bytes on the last element or group of each of the shorter variants. One can also encapsulate the whole choice in a sequence whose length is the fixed maximum length. This is my personal favorite, since it uses the FinalUnused that is already in the grammar. E.g., suppose the length of the longest variant is 258 bytes. <sequence dfdl:lengthKind="explicit" dfdl:length="258" dfdl:lengthUnits="bytes" dfdl:applies="hereOnly"> <choice> ..... </choice> </sequence> The only thing I don't like about this is I don't have a way to express the length other than to hard-code the constant 258, but since tooling would typically put this in based on Cobol descriptors or other information I don't mind so much. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Steve Hanson/UK/IBM@IBMGB 11/16/2007 11:24 AM To Mike Beckerle/Worcester/IBM cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset Hi Mike One extra thought post-call, caused by thinking about the FinalUnused part of a sequence. Some fixed length choices have a rule that each branch of the choice must be of equal length. This results in unused bytes for branches shorter than the maximum. In a bitstream instance, a user will see these bytes. Perhaps the grammar should acknowledge their existence, even if there are no DFDL properties to express the bytes (like FinalUnused)? Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 10/11/2007 00:16 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Grammar as adapted to infoset I took a crack at this. I ripped it out of the main document for faster review/turnaround. I think this matches the latest infoset email diagram. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Grammar as adapted to infoset
by Mike Beckerle 16 Nov '07

16 Nov '07

There are of course alternatives. E.g., a hidden or non-hidden field which is a byte array or hexBinary to absorb the padding bytes at the ends of the shorter variants. Or one could put trailing skip bytes on the last element or group of each of the shorter variants. One can also encapsulate the whole choice in a sequence whose length is the fixed maximum length. This is my personal favorite, since it uses the FinalUnused that is already in the grammar. E.g., suppose the length of the longest variant is 258 bytes. <sequence dfdl:lengthKind="explicit" dfdl:length="258" dfdl:lengthUnits="bytes" dfdl:applies="hereOnly"> <choice> ..... </choice> </sequence> The only thing I don't like about this is I don't have a way to express the length other than to hard-code the constant 258, but since tooling would typically put this in based on Cobol descriptors or other information I don't mind so much. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Steve Hanson/UK/IBM@IBMGB 11/16/2007 11:24 AM To Mike Beckerle/Worcester/IBM cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset Hi Mike One extra thought post-call, caused by thinking about the FinalUnused part of a sequence. Some fixed length choices have a rule that each branch of the choice must be of equal length. This results in unused bytes for branches shorter than the maximum. In a bitstream instance, a user will see these bytes. Perhaps the grammar should acknowledge their existence, even if there are no DFDL properties to express the bytes (like FinalUnused)? Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 10/11/2007 00:16 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Grammar as adapted to infoset I took a crack at this. I ripped it out of the main document for faster review/turnaround. I think this matches the latest infoset email diagram. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

Grammar as adapted to infoset
by Mike Beckerle 16 Nov '07

16 Nov '07

I took a crack at this. I ripped it out of the main document for faster review/turnaround. I think this matches the latest infoset email diagram. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

Invitation: DFDL OGF WG - discuss Grammar (Nov 16 09:00 AM EST in Teleconference)
by Mike Beckerle 15 Nov '07

15 Nov '07

Calendar Entry ---------------------------------------------------------------------------------------------------------------------------- Subject Chair DFDL OGF WG - discuss Grammar Mike Beckerle/Worcester/IBM Invitees Date Required (to) 11/16/2007 Time Optional (cc) 09:00 AM - 10:00 AM (1 hour) dfdl-wg(a)ogf.org FYI (bcc) Proposed start -1 at Proposed end -1 at Original start 11/16/2007 at 09:00 AM Original end 11/16/2007 at 10:00 AM Where Location Teleconference I'd like to discuss the DFDL Grammar draft doc before next week, as I am going to miss that meeting also unfortunately. I'm hoping to fold the Infoset material and with luck this grammar stuff back into the core document "real soon now" like Fri 11/16. the grammar note/doc attached again here in case you've lost it. This is in response to the InfoSet work and must be read in context of that work. (See attached file: dfdl-grammar.doc) ----------------------------------------------------------------------------------------------------------------------------------------- Call in details here: Participant Passcode: 605528 ITN: 2-421-0033 Tieline: 421-0033 Participants from countries not listed in the table below should dial the ITN, Tieline or one of the Toll numbers. |------------------+----------------------+--------------------| | Country | Toll-free (IBM Pays) | Toll (Caller Pays) | |------------------+----------------------+--------------------| | Australia | 1-800-24-2283 | | |------------------+----------------------+--------------------| | Austria | 0800-292-738 | +43 179576264 | |------------------+----------------------+--------------------| | Belgium | 0800-7-3026 | +32 22006114 | |------------------+----------------------+--------------------| | Canada | see the USA number | see the USA number | |------------------+----------------------+--------------------| | China North | 10800-711-0770 | | | China South | 10800-110-0713 | | |------------------+----------------------+--------------------| | Czech Republic | 800-143-241 | +420-272-133-001 | | | | or | | | | +420-533-413-001 | |------------------+----------------------+--------------------| | Denmark | 80-888377 | +45 38323070 | |------------------+----------------------+--------------------| | Finland | 0800-914-630 | +358 972519061 | |------------------+----------------------+--------------------| | France | 0800-902366 | +33 157323040 or | | | | +33 157323041 | |------------------+----------------------+--------------------| | Germany | 0800-181-6323 | +49 6951709081 | |------------------+----------------------+--------------------| | Greece | 00-800-11-004-9793 | | |------------------+----------------------+--------------------| | Hungary | 06-800-16-381 | | |------------------+----------------------+--------------------| | India | 000-800-100-1156 | | |------------------+----------------------+--------------------| | Ireland | 1800-558728 | +353 16569209 | |------------------+----------------------+--------------------| | Israel | 1-809-317-098 | +972-3-9165956 | |------------------+----------------------+--------------------| | Italy | 800-788634 | +39 0269430413 | |------------------+----------------------+--------------------| | Mexico | see the USA number | see the USA number | |------------------+----------------------+--------------------| | Netherlands | 0800-022-8558 | +31 20513 4100 | |------------------+----------------------+--------------------| | Norway | 800-18373 | +47 24159528 | |------------------+----------------------+--------------------| | Poland | 0-0800111-1712 or | +48 22 366 5400 or | | | IBM PlusGSM 85400 | +48 22 609 5400 | |------------------+----------------------+--------------------| | Portugal | 800-8-11111 | | |------------------+----------------------+--------------------| | Russia | 8-10-800-2-172-10-44 | | |------------------+----------------------+--------------------| | Slovakia | | +421 2 4954 6539 | | | | or | | | | +421 2 4964-9935 | |------------------+----------------------+--------------------| | South Africa | 0800-981-078 | | |------------------+----------------------+--------------------| | Spain | 900-95-1089 | +34 912754171 | |------------------+----------------------+--------------------| | Sweden | 020-799414 | +46 850163259 | |------------------+----------------------+--------------------| | Switzerland | 0800-564-331 | +41 58 333 7337 | |------------------+----------------------+--------------------| | United Kingdom | 0808-234-1969 | +44 2070260533 | |------------------+----------------------+--------------------| | USA | 877-421-0033 | 770-615-1250 | |------------------+----------------------+--------------------|

1 0

DFDL: Minutes from OGF WG call, 14 Nov 2007
by Ian W Parkinson 15 Nov '07

15 Nov '07

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 14 Nov 2007 Attendees Geoff Judd (IBM) Steve Hanson (IBM) Suman Kalia (IBM) Simon Parker (PolarLake) Ian Parkinson (IBM) Apologies Mike Beckerle (IBM) 1. Information Set Proposal Following last week's review of the Information Set proposal, Simon updated the proposal and distributed a new draft, version 7. Alan queried the reference to empty lists under "No Value", asking if this was necessary as DFDL does not use a concept of lists - this is carried over from the XML Information Set specification, and is only theoretical. The proposal will be distributed wider within IBM to solicit final comments. 2. Review of Revised DFDL Data Grammar Mike has updated the Data Grammar portion of the DFDL specification, to take into account the InfoSet proposal, and distributed this via the DFDL-WG mailing list. The meeting reviewed the revised grammar, touching upon the following points: Both the ComplexContent and ComplexElement productions include a "Prefix" element. Alan asked whether this duplication was necessary. Steve and Simon noted that this had not changed during the revision and that there are cases where this is useful. The meeting agreed not to change this. Alan asked why we need both Element and ComplexContent in the SequenceItem production. Element effectively introduces a name. Simon noted that most of the changes were in the productions, and wondered what influence the InfoSet proposal actually had on the specification. Steve said that there had been real changes, for example with SimpleElement and ComplexElement. Steve observed that Prefix and Postfix Separators no longer appear in the productions table. The meeting agreed that these were probably no longer necessary, and that this was a useful simplification, but to check this with Mike. 3. Object Constraint Language The UML diagram in the revised InfoSet proposal uses Object Constraint Language1 (OCL) to specify an invariant on the Element class. Simon asked the meeting to consider whether this might be a useful approach in other sections of the DFDL specification, in particular when describing parser behaviour. Suman was concerned that it might prove useful only for specifying parser internals which should not be covered in the language specification. 4. Other Business Suman asked if we need to define a parser API. Steve suggested that this should not form part of the language specification, but that there may need to be other specification documents. Steve and Alan will be compiling a list of remaining work items to move the specification towards completion. Work has started within IBM, to attempt to describe a number of industry standard data formats, including SWIFT and ISO 8583, using DFDL. Simon has a long standing work item to attempt to describe a number of custom application formats. These efforts should help validate the utility of DFDL and suggest specification revisions. Thanksgiving is next week - will this affect attendence of next weeks meeting? Meeting closed 17:35 GMT Actions Mike to comment on the omission of initial prefix Separators and final postfix Separators from the productions table, in the revised DFDL Data Grammar. Are these intended to be covered by other elements in the productions, or do they need to be added? References Object Constraint Language; specification at http://www.omg.org/technology/documents/formal/ocl.htm Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Call today - MikeB cannot attend
by Mike Beckerle 14 Nov '07

14 Nov '07

I have an unmovable conflict so I can't make the DFDL OGF call today. ...mikeb

1 0

Dfdl information set 7
by Simon Parker 12 Nov '07

12 Nov '07

Good morning. Version 7 of the proposal incorporates the outcome of last week's conference and subsequent email discussion. If I've overlooked or misunderstood any amendments, please let me know. Regards, Simon Simon Parker Software Consultant PolarLake 80 Harcourt Street Dublin 2, Ireland Phone +353 1 449-1075 Fax +353 1 449-1011 Web http://www.polarlake.com/ The information transmitted in this email is intended for the addressee only and may contain confidential and/or privileged material. Any review, retransmission, dissemination, reliance upon or other use of this information by persons or entities other than the addressee is prohibited. If you think, for any reason, that this message may have been addressed to you in error, we would ask you to notify the sender immediately by return email and delete the material. PolarLake Limited | Registered in Dublin, Ireland | Number 357324 | Registered office as above.

1 0

DFDL: *CORRECTED* Minutes from OGF WG call, 7 Nov 2007
by Ian W Parkinson 09 Nov '07

09 Nov '07

One correction in the revised minutes below: some duplicate text removed from para 2.1. With thanks to Simon for spotting the error. Ian Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 7 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Suman Kalia (IBM) Simon Parker (PolarLake) Ian Parkinson (IBM) 1. Introduction The previous meeting left two areas for consideration: the Information Set (InfoSet) proposal, and the work regarding defaulting and nulls. This meeting would cover a review of a revised InfoSet proposal. The defaults/nulls issue has not been progressed. 2. Review of Revised Information Set Proposal Steve and Simon have collaborated on a revision, v6, of the InfoSet proposal. This has been circulated via the DFDL-WG mailing list. The meeting reviewed the v6 proposal, highlighting the following points: 1. There remain two types of information item: Element Information Items and Document Information Items. There are now two categories of Element Information Item: simple, and complex. 2. The description of Document Information Items incorrectly lists a "root" member. This should be replaced by a "children" member. Geoff pointed out that the associated UML diagram suggests that a Document Information Item may contain multiple children; this will be restricted in the text. 3. The "dfdl version" member is now used to describe the version of DFDL to which the InfoSet conforms; it does not describe the version of the parser generating the InfoSet. In particular, for an InfoSet to inlcude a dfdl version identifier, there is now no requirement for that InfoSet to have been generated by a DFDL parser. 4. Mike queried the meaning of "absolute Schema Component Descriptor"; it should identify a schema as a whole. This is a term defined by the SCD specification, and an appropriate citation should be added to the proposal. 5. The meeting discussed, at length, the "data value" member of Element Information Item, which the proposal lists as being of type String. Mike pointed out that this is insufficient to distinguish between some binary floating-point values, and suggested that it should be of a type defined by the "data type" member. Suman suggested this would also make for a more natural API. Simon remarked on the precedent set by XML Schema where all values are represented as strings. Mike closed the discussion advocating dropping the word "String" from the definition of the "data value" member. 6. Suman queried the purpose of the document information item. This is an in-memory artifact only, which acts as a container of the InfoSet's data, and has no representation in a serialization of the InfoSet. This should be clarified in the proposal. 7. The proposal now contains a UML diagram illustrating the object model. The relationship between Parent and Element requires some clarifying: it should be modelled as a composite containment relationship. 8. Simon highlighted that the "schema" member of Element Information Item may be either a relative or an absolute SCD; while the "schema" member of Document Information Item must be an absolute SCD. 9. The current version makes no mention of arrays, preferring to treat arrays as repeated children with the same name. The meeting consdered how to denote arrays in the InfoSet, but opted instead to add a note regarding arrays to the specification. Mike will circulate his notes regarding the agreed changes to the spec. Simon will circulate a new version, v7, incorporating further comments. 3. Other Business There was no other business. Meeting closed Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, 7 Nov 2007
by Ian W Parkinson 08 Nov '07

08 Nov '07

Hi all, Steve and Mike have asked me to record minutes at the regular DFDL-WG conference calls. Here are the minutes from yesterday's meeting - I'd obviously welcome any suggestions you may have on the format or style of these minutes. Thanks, Ian Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 7 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Suman Kalia (IBM) Simon Parker (PolarLake) Ian Parkinson (IBM) 1. Introduction The previous meeting left two areas for consideration: the Information Set (InfoSet) proposal, and the work regarding defaulting and nulls. This meeting would cover a review of a revised InfoSet proposal. The defaults/nulls issue has not been progressed. 2. Review of Revised Information Set Proposal Steve and Simon have collaborated on a revision, v6, of the InfoSet proposal. This has been circulated via the DFDL-WG mailing list. The meeting reviewed the v6 proposal, highlighting the following points: There remain two types of information item: Element Information Items and Document Information Items. Element Information Items. There are now two categories of Element Information Item: simple, and complex. The description of Document Information Items incorrectly lists a "root" member. This should be replaced by a "children" member. Geoff pointed out that the associated UML diagram suggests that a Document Information Item may contain multiple children; this will be restricted in the text. The "dfdl version" member is now used to describe the version of DFDL to which the InfoSet conforms; it does not describe the version of the parser generating the InfoSet. In particular, for an InfoSet to inlcude a dfdl version identifier, there is now no requirement for that InfoSet to have been generated by a DFDL parser. Mike queried the meaning of "absolute Schema Component Descriptor"; it should identify a schema as a whole. This is a term defined by the SCD specification, and an appropriate citation should be added to the proposal. The meeting discussed, at length, the "data value" member of Element Information Item, which the proposal lists as being of type String. Mike pointed out that this is insufficient to distinguish between some binary floating-point values, and suggested that it should be of a type defined by the "data type" member. Suman suggested this would also make for a more natural API. Simon remarked on the precedent set by XML Schema where all values are represented as strings. Mike closed the discussion advocating dropping the word "String" from the definition of the "data value" member. Suman queried the purpose of the document information item. This is an in-memory artifact only, which acts as a container of the InfoSet's data, and has no representation in a serialization of the InfoSet. This should be clarified in the proposal. The proposal now contains a UML diagram illustrating the object model. The relationship between Parent and Element requires some clarifying: it should be modelled as a composite containment relationship. Simon highlighted that the "schema" member of Element Information Item may be either a relative or an absolute SCD; while the "schema" member of Document Information Item must be an absolute SCD. The current version makes no mention of arrays, preferring to treat arrays as repeated children with the same name. The meeting consdered how to denote arrays in the InfoSet, but opted instead to add a note regarding arrays to the specification. Mike will circulate his notes regarding the agreed changes to the spec. Simon will circulate a new version, v7, incorporating further comments. 3. Other Business There was no other business. Meeting closed Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0