- dfdl-wg - lists.ogf.org

review slides from f2f
by Mike Beckerle 05 Oct '05

05 Oct '05

Coarse notes for sure, but our summary/action items slides are at the end of this. Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA 01581 voice and FAX 508-599-7148 home/mobile office 508-915-4767 skype ID mbeckerle

1 0

Fw: [dfdl-wg] F2F status on discussions here about Scoping #2
by Steve Hanson 05 Oct '05

05 Oct '05

Geoff Judd/UK/IBM 05/10/2005 17:44 To Steve Hanson/UK/IBM@IBMGB cc Mike Beckerle/Worcester/IBM@IBMUS, Suman Kalia/Toronto/IBM@IBMCA Subject Re: Fw: [dfdl-wg] F2F status on discussions here about Scoping #2 (Document link: Steve Hanson) Steve, This is how I would see the definitions with selectors. Regards, Geoff Judd Websphere MQ Integrator Development IBM UK Ltd Hursley Telephone: +44-1962-818461 E-mail: JUDDG(a)uk.ibm.com Steve Hanson/UK/IBM To 05/10/2005 12:30 Mike Beckerle/Worcester/IBM cc Geoff Judd/UK/IBM, Suman Kalia/Toronto/IBM@IBMCA Subject Fw: [dfdl-wg] F2F status on discussions here about Scoping #2 I think I see what selectors solve. It's the embedded message scenario where I have a different format, and hence configuration, for the embedded message. I can use selectors to pull together multiple different configurations in a single parse. Correct? Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/10/2005 12:27 ----- Steve Hanson/UK/IBM To 05/10/2005 12:18 Mike Beckerle <beckerle(a)us.ibm.com> cc Geoff Judd/UK/IBM, Suman Kalia/Toronto/IBM@IBMCA Subject Re: [dfdl-wg] F2F status on discussions here about Scoping (Document link: Steve Hanson) Hi, I've taken a look through the slides, looks like you had a productive day. The scheme looks ok in principle, but I can't glean from the foils exactly how the selector mechanism works, and (consequently) why we can't achieve the same result using configuration blocks alone. I'm not trying to be awkward here, just trying to justify to myself why we need the extra complexity selectors introduce. It would be great if we could rubber stamp scoping by the end of the call today. What would be really helpful for the call would be to mail out a single DFDL schema that used configurations and selectors to model the following COBOL message and its equivalent tagged/delimited form. As a comparison, here's a DFDL schema that models this just using configurations. This works because the extends attribute is mandatory on any dfdl element if more than one configuration is defined. 1 MESSAGE 5 STR-FIELD PIC X(10) <----------- logical string length 10 5 INT-FIELD PIC 9(8) COMP <----------- logical integer, physical integer length 4 5 DEC-FIELD PIC 9999V99 DISPLAY <----------- logical decimal, physical extended decimal length 6 with VDP 2 Tagged/delimited example instance using postfix delimiters <MSG><STR>Example;<INT>4000;<DEC>9999.99;<END> <dfdl:configuration name="cobol-params" repType="binary" ...... /> <dfdl:configuration name="tagged-params" repType="text" ...... /> <xsd:element name="message"> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:defaults> <dfdl:configuration selector="cobol" extends="cobol-params" charset="937" byteOrder="bigEndian" /> <dfdl:configuration selector="tagged" extends="tagged-params" charset="850" /> <dfdl:defaults> </xsd:appinfo> </xsd:annotation> <xsd:complexType> <xsd:sequence> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:sequence selector="tagged" initiator="<MSG>" terminator="<END>" separator=";" separatorType="postfix" /> </xsd:appinfo"> </xsd:annotation> <xsd:element name="str_field" xsd:type="string"> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:string selector="cobol" baseWidth="10" /> <dfdl:string selector="tagged" initiator="<STR>" /> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element name="int_field" xsd:type="int"> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:int selector="cobol" baseWidth="4" /> <dfdl:int selector="tagged" initiator="<INT>" /> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element name="string" xsd:type="decimal"> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:decimal selector="cobol" baseWidth="6" virtualDecimalPoint="2" /> <dfdl:decimal selector="tagged" initiator="<DEC>" virtualDecimalPoint="2" /> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> </xsd:complexType> </xsd:element> Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm. com> To Sent by: dfdl-wg(a)gridforum.org owner-dfdl-wg@ggf cc .org Subject [dfdl-wg] F2F status on discussions 05/10/2005 03:08 here about Scoping Attached are slides we'll review as a group tomorrow. Could be typos in these. I did them rather fast. [attachment "ggf15-dfdl-wg-f2f-scoping-summary.ppt" deleted by Geoff Judd/UK/IBM]

1 0

Fw: [dfdl-wg] F2F status on discussions here about Scoping #2
by Steve Hanson 05 Oct '05

05 Oct '05

----- Forwarded by Steve Hanson/UK/IBM on 05/10/2005 17:46 ----- Steve Hanson/UK/IBM To 05/10/2005 17:46 Mike Beckerle/Worcester/IBM cc Geoff Judd/UK/IBM, Suman Kalia/Toronto/IBM@IBMCA Subject Fw: [dfdl-wg] F2F status on discussions here about Scoping #2 I think I see what selectors solve. It's the embedded message scenario where I have a different format, and hence configuration, for the embedded message. I can use selectors to pull together multiple different configurations in a single parse. Correct? Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/10/2005 12:27 ----- Steve Hanson/UK/IBM To 05/10/2005 12:18 Mike Beckerle <beckerle(a)us.ibm.com> cc Geoff Judd/UK/IBM, Suman Kalia/Toronto/IBM@IBMCA Subject Re: [dfdl-wg] F2F status on discussions here about Scoping (Document link: Steve Hanson) Hi, I've taken a look through the slides, looks like you had a productive day. The scheme looks ok in principle, but I can't glean from the foils exactly how the selector mechanism works, and (consequently) why we can't achieve the same result using configuration blocks alone. I'm not trying to be awkward here, just trying to justify to myself why we need the extra complexity selectors introduce. It would be great if we could rubber stamp scoping by the end of the call today. What would be really helpful for the call would be to mail out a single DFDL schema that used configurations and selectors to model the following COBOL message and its equivalent tagged/delimited form. As a comparison, here's a DFDL schema that models this just using configurations. This works because the extends attribute is mandatory on any dfdl element if more than one configuration is defined. 1 MESSAGE 5 STR-FIELD PIC X(10) <----------- logical string length 10 5 INT-FIELD PIC 9(8) COMP <----------- logical integer, physical integer length 4 5 DEC-FIELD PIC 9999V99 DISPLAY <----------- logical decimal, physical extended decimal length 6 with VDP 2 Tagged/delimited example instance using postfix delimiters <MSG><STR>Example;<INT>4000;<DEC>9999.99;<END> <dfdl:configuration name="cobol" repType="binary" ...... /> <dfdl:configuration name="tagged" repType="text" ...... /> <xsd:element name="message"> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:defaults> <dfdl:configuration extends="cobol" charset="937" byteOrder="bigEndian" /> <dfdl:configuration extends="tagged" charset="850" /> <dfdl:defaults> </xsd:appinfo> </xsd:annotation> <xsd:complexType> <xsd:sequence> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:sequence extends="tagged" initiator="<MSG>" terminator="<END>" separator=";" separatorType="postfix" /> </xsd:appinfo"> </xsd:annotation> <xsd:element name="str_field" xsd:type="string"> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:string extends="cobol" baseWidth="10" /> <dfdl:string extends="tagged" initiator="<STR>" /> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element name="int_field" xsd:type="int"> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:int extends="cobol" baseWidth="4" /> <dfdl:int extends="tagged" initiator="<INT>" /> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element name="string" xsd:type="decimal"> <xsd:annotation> <xsd:appinfo source="http://dataformat.org"> <dfdl:decimal extends="cobol" baseWidth="6" virtualDecimalPoint="2" /> <dfdl:decimal extends="tagged" initiator="<DEC>" virtualDecimalPoint="2" /> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> </xsd:complexType> </xsd:element> Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm. com> To Sent by: dfdl-wg(a)gridforum.org owner-dfdl-wg@ggf cc .org Subject [dfdl-wg] F2F status on discussions 05/10/2005 03:08 here about Scoping Attached are slides we'll review as a group tomorrow. Could be typos in these. I did them rather fast. [attachment "ggf15-dfdl-wg-f2f-scoping-summary.ppt" deleted by Steve Hanson/UK/IBM]

1 0

Slides on layering and extensibility
by Talbott, Tara D 05 Oct '05

05 Oct '05

Here are the slides and writeup on layering and extensibility for reviewing as a group today. Tara

1 0

Call-in number for sessions at F2F and at GGF15
by Mike Beckerle 03 Oct '05

03 Oct '05

Those who would like to call in here's how: During the GGF sessions on monday use: Dom: 1-888-464-9078 Int: 1-712-455-7096 Participants passcode: 225210 Session #1 is a working group meeting - 2005-10-03 11:00:00 Beacon Hill Room Session #2 is overview and status - 2005-10-03 14:00:00 Berkeley/Clarendon Room These are US EDT time zone which is UTC-4. Note that the morning session is more important for workgroup participants. I think the afternoon one is just status and recruiting for new membership. During the F2F sessions, starting at 9am monday, I'd like to try something different. Rather than keep a conference call line open all day I would like to use my skype number. So you can skype me if you have skype installed (free, effective, recommended). my skype ID is mbeckerle, or you can phone call 508-915-4767 from a regular phone and that "rings" on skype on my computer. If for audio quality reasons or network flakyness or whatever that doesn't work out then we'll fall back on my conference call number which is above. Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA office 508-599-7148, IBM tieline 518-7148 cell 508-287-3093 home & mobile office (skype) 508-915-4767

1 1

Usage scenarios
by Martin Westhead 29 Sep '05

29 Sep '05

Hi Bob, Here is my list of ways in which DFDL could be used - I have probably missed some but here's enough to kick off with. If you need more details on any of the real-world examples let me know. Cheers, Martin Description ----------- In QCD physics independent research groups from all over the world have data which is always a 4d array of floating point values. However, different groups have different standards for precision, dimension order, byte order. It would be useful for them to have a simple, canonical XML language for describing the format of a file. In the first instance this need only be human readable. Archiving --------- Data needs to be stored, but the programs and systems for reading it become obsolete. DFDL provides a valuable possibility of describing all the details of a particular format so that even if there were no programs able to read a format, the description (and the standard) would provide sufficient information to access archived data. (There are lots of examples of this type they might include atmospheric measurement). A sophistication of this is that archived data may need to be transformed as it is moved to up to data physical media (changes of precision) etc. It would be nice if DFDL could (a) annotate these changes (b) (perhaps) be used to ensure that the changes did not result in data loss. Format abstraction ------------------ At the simplest level the QCD physicists (described above) would like to be able to have a single API that would allow them to read any described piece of data, and carry out all the transformations required to ensure that they get the correct array in memory. I have examples of potential users who are just interested in describing byte-order in a standard way. At the next level we would like to supply a high level DFDL description that captures a standard view of the data, and have generic DFDL logic that can transform an existing DFDL-described format into this generic view. This is one of the primary motivations for "layers" in the standard. It is a very powerful feature but it introduces scoping issues: What transformations can DFDL not describe? (also what transformations can DFDL not describe efficiently). Generic data access ------------------- A DFDL library should provide the ability to interrogate a data description and read all aspects of the data into memory. An example of a generic tool is a browser that will allow arbitrary DFDL-described data to be displayed in some sensible human-readable form. This case requires the standard to specify an API for reading and interrogating the data. The favoured suggestion for this is to extend DOM/SAX to allow the reading of data fields directly into in-memory types (float, int, char etc.) Data queries ------------ The DFDL description implies an associated XML document. This document can be queried using XPath/XQuery to extract pieces of data. [Note: If the data comes back as an XML-XPath result then this process is straight forward. With BinX we tried to return the data in a similar format to the one it is represented in with an accompanying description. We found a number of issues arose in this case that may or may not also arise for DFDL]. Data annotations ---------------- The same XPath/XQuery expressions that can be used to query a document can provide external (format independent) annotations. For example NASA stores photographic images of hurricanes. A scientist can identify a blob of pixels that correspond to the hurricane in an image. They could like to store this annotation is such a way that the will be preserved through future transformations (e.g. new image format, or different pixel depth, or compression level). Note the point here is that a byte offset into the image data cannot do this. XML without the tags -------------------- There are groups who would like to use DFDL as a sort of cheap data compression technique. An example here is particle physics collision data. This is stored as a set of sparse (hence variable sized) trees of results. The data is richly structured trees and they would like to access it and talk about it as if it were in XML but they don't want to (cannot afford to) represent it using XML markup or use conventional XML tools to parse it. The idea is that such a group would design a new binary format that could be described in XML and then they would work with the implied XML data. Note: naturally these folks do not want to access their floating point values as strings so they would want the sort of DOM extensions that we alluded to earlier. For this same reason things like Binary XML do not solve their problem. Another example comes from the astronomy community has recently moved from a long-standing binary data format (FITS) to an XML version (VOTable). FITS was very rich in metadata but also included binary images and large tables of observational data representations. VOTable is great for capturing the metadata in a standard way but leads to excessive bloat for images and large tables. The community has ended up with a complicated compromise in which they allow raw binary data in at the bottom of the XML file. A DFDL-described format could provide a cleaner solution.

3 3

F2F discussion documents for GGF15
by Geoff Judd 29 Sep '05

29 Sep '05

I have created the following 3 documents in the current document folder to kick off discussions next week. Support of Default and Null values in DFDL Support of Choices in DFDL Support of Wildcards in DFDL Regards, Geoff Judd Websphere MQ Integrator Development IBM UK Ltd Hursley Telephone: +44-1962-818461 E-mail: JUDDG(a)uk.ibm.com

1 0

Wednesday's telcon
by Martin Westhead 29 Sep '05

29 Sep '05

9am PDT Wednesday 27th September 2005 Preliminary Agenda ------------------ - F2F at GGF (any outstanding issues) - Primer use cases (?) - Issues list (starting from item 28) Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call.

2 1

Telcon notes 2005-09-28
by Martin Westhead 28 Sep '05

28 Sep '05

Present ======= Martin Westhead Tara Talbot Robert McGrath Geoff Judd Steve Hanson Jim Meyers Agenda ====== F2F at GGF (any outstanding issues) ----------------------------------- No issues raised at this time. Primer use cases (?) -------------------- Short discussion of use-cases. Existing examples seem complicated, Rob explained that this was just things he knew about that he'd like DFDL to do. Action: Martin and Steve agreed to produce some simple examples. Extensibility discussion ------------------------ Jim raised the topic of extensibility. He is concerned that aspects of the current proposal will not allow the sort of extensibility that he would like to see. Jim is suggesting that perhaps a separate group (possibly overlapping) should be exploring this issue. He would like this group to consider approaches that in the short term may require changes to what is currently on the table. Action [Martin]: Extensibility tracker add Steve and Suman Action [Mike]: Add Extensibility to F2F Agenda Choices and wildcards (Geoff) ----------------------------- This was not discussed due to lack of time and the fact that Mike was not available. Action [Martin]: On the agenda for next week. Issues list (starting from item 28) ----------------------------------- No progress - start from 28 next week.

1 0

New Primer: I need some help
by Robert E. McGrath 26 Sep '05

26 Sep '05

Greetings, As many of you know, I have been tasked to compile a revised "primer" for DFDL. Based on the draft spec. and other information, I have made some good progress, but there are a number of topics that I don't have enough information on. This note summarizes areas where I need substantial contributions from you all. This may not be things to tackle via the mail list, I don't know. Perhaps we can discuss these issues at the F2F. 1. Use cases Actually, there should be a "Use Cases" tracker and a separate UC document, IMO. By "Use Case" I mean statements of how DFDL is intended to be used. This is different from the current example suites which illustrate the language but do not discuss why you need DFDL or what it helps you do. More on this one later. I'm developing a note with some interesting use cases I'd like to discuss. 2. Overview of the language I would like the primer to say "the DFDL is the following XSD elements, with the following restrictions. DFDL defines the following tags and attributes". Just a summary, circa 2 pages. I'm sure this informtion is implicit in the current spec and demo software, but I need help to write a clear summary. 3. Processing model I would like the primer to explain how DFDL is used, i.e., Parse DFDL, which tells how to parse data, ... Again, this is probably implicit in current work, but I need a short and clear summary. 4. Explanation of configurations, layers etc. Related to above. Explain the non-trivial processing of DFDL. Some of this awaits the outcome of trackers.

2 1