September 2005 - dfdl-wg

Call-in number for sessions at F2F and at GGF15
by Mike Beckerle 03 Oct '05

03 Oct '05

Those who would like to call in here's how: During the GGF sessions on monday use: Dom: 1-888-464-9078 Int: 1-712-455-7096 Participants passcode: 225210 Session #1 is a working group meeting - 2005-10-03 11:00:00 Beacon Hill Room Session #2 is overview and status - 2005-10-03 14:00:00 Berkeley/Clarendon Room These are US EDT time zone which is UTC-4. Note that the morning session is more important for workgroup participants. I think the afternoon one is just status and recruiting for new membership. During the F2F sessions, starting at 9am monday, I'd like to try something different. Rather than keep a conference call line open all day I would like to use my skype number. So you can skype me if you have skype installed (free, effective, recommended). my skype ID is mbeckerle, or you can phone call 508-915-4767 from a regular phone and that "rings" on skype on my computer. If for audio quality reasons or network flakyness or whatever that doesn't work out then we'll fall back on my conference call number which is above. Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA office 508-599-7148, IBM tieline 518-7148 cell 508-287-3093 home & mobile office (skype) 508-915-4767

1 1

Usage scenarios
by Martin Westhead 29 Sep '05

29 Sep '05

Hi Bob, Here is my list of ways in which DFDL could be used - I have probably missed some but here's enough to kick off with. If you need more details on any of the real-world examples let me know. Cheers, Martin Description ----------- In QCD physics independent research groups from all over the world have data which is always a 4d array of floating point values. However, different groups have different standards for precision, dimension order, byte order. It would be useful for them to have a simple, canonical XML language for describing the format of a file. In the first instance this need only be human readable. Archiving --------- Data needs to be stored, but the programs and systems for reading it become obsolete. DFDL provides a valuable possibility of describing all the details of a particular format so that even if there were no programs able to read a format, the description (and the standard) would provide sufficient information to access archived data. (There are lots of examples of this type they might include atmospheric measurement). A sophistication of this is that archived data may need to be transformed as it is moved to up to data physical media (changes of precision) etc. It would be nice if DFDL could (a) annotate these changes (b) (perhaps) be used to ensure that the changes did not result in data loss. Format abstraction ------------------ At the simplest level the QCD physicists (described above) would like to be able to have a single API that would allow them to read any described piece of data, and carry out all the transformations required to ensure that they get the correct array in memory. I have examples of potential users who are just interested in describing byte-order in a standard way. At the next level we would like to supply a high level DFDL description that captures a standard view of the data, and have generic DFDL logic that can transform an existing DFDL-described format into this generic view. This is one of the primary motivations for "layers" in the standard. It is a very powerful feature but it introduces scoping issues: What transformations can DFDL not describe? (also what transformations can DFDL not describe efficiently). Generic data access ------------------- A DFDL library should provide the ability to interrogate a data description and read all aspects of the data into memory. An example of a generic tool is a browser that will allow arbitrary DFDL-described data to be displayed in some sensible human-readable form. This case requires the standard to specify an API for reading and interrogating the data. The favoured suggestion for this is to extend DOM/SAX to allow the reading of data fields directly into in-memory types (float, int, char etc.) Data queries ------------ The DFDL description implies an associated XML document. This document can be queried using XPath/XQuery to extract pieces of data. [Note: If the data comes back as an XML-XPath result then this process is straight forward. With BinX we tried to return the data in a similar format to the one it is represented in with an accompanying description. We found a number of issues arose in this case that may or may not also arise for DFDL]. Data annotations ---------------- The same XPath/XQuery expressions that can be used to query a document can provide external (format independent) annotations. For example NASA stores photographic images of hurricanes. A scientist can identify a blob of pixels that correspond to the hurricane in an image. They could like to store this annotation is such a way that the will be preserved through future transformations (e.g. new image format, or different pixel depth, or compression level). Note the point here is that a byte offset into the image data cannot do this. XML without the tags -------------------- There are groups who would like to use DFDL as a sort of cheap data compression technique. An example here is particle physics collision data. This is stored as a set of sparse (hence variable sized) trees of results. The data is richly structured trees and they would like to access it and talk about it as if it were in XML but they don't want to (cannot afford to) represent it using XML markup or use conventional XML tools to parse it. The idea is that such a group would design a new binary format that could be described in XML and then they would work with the implied XML data. Note: naturally these folks do not want to access their floating point values as strings so they would want the sort of DOM extensions that we alluded to earlier. For this same reason things like Binary XML do not solve their problem. Another example comes from the astronomy community has recently moved from a long-standing binary data format (FITS) to an XML version (VOTable). FITS was very rich in metadata but also included binary images and large tables of observational data representations. VOTable is great for capturing the metadata in a standard way but leads to excessive bloat for images and large tables. The community has ended up with a complicated compromise in which they allow raw binary data in at the bottom of the XML file. A DFDL-described format could provide a cleaner solution.

3 3

F2F discussion documents for GGF15
by Geoff Judd 29 Sep '05

29 Sep '05

I have created the following 3 documents in the current document folder to kick off discussions next week. Support of Default and Null values in DFDL Support of Choices in DFDL Support of Wildcards in DFDL Regards, Geoff Judd Websphere MQ Integrator Development IBM UK Ltd Hursley Telephone: +44-1962-818461 E-mail: JUDDG(a)uk.ibm.com

1 0

Wednesday's telcon
by Martin Westhead 29 Sep '05

29 Sep '05

9am PDT Wednesday 27th September 2005 Preliminary Agenda ------------------ - F2F at GGF (any outstanding issues) - Primer use cases (?) - Issues list (starting from item 28) Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call.

2 1

Telcon notes 2005-09-28
by Martin Westhead 28 Sep '05

28 Sep '05

Present ======= Martin Westhead Tara Talbot Robert McGrath Geoff Judd Steve Hanson Jim Meyers Agenda ====== F2F at GGF (any outstanding issues) ----------------------------------- No issues raised at this time. Primer use cases (?) -------------------- Short discussion of use-cases. Existing examples seem complicated, Rob explained that this was just things he knew about that he'd like DFDL to do. Action: Martin and Steve agreed to produce some simple examples. Extensibility discussion ------------------------ Jim raised the topic of extensibility. He is concerned that aspects of the current proposal will not allow the sort of extensibility that he would like to see. Jim is suggesting that perhaps a separate group (possibly overlapping) should be exploring this issue. He would like this group to consider approaches that in the short term may require changes to what is currently on the table. Action [Martin]: Extensibility tracker add Steve and Suman Action [Mike]: Add Extensibility to F2F Agenda Choices and wildcards (Geoff) ----------------------------- This was not discussed due to lack of time and the fact that Mike was not available. Action [Martin]: On the agenda for next week. Issues list (starting from item 28) ----------------------------------- No progress - start from 28 next week.

1 0

New Primer: I need some help
by Robert E. McGrath 26 Sep '05

26 Sep '05

Greetings, As many of you know, I have been tasked to compile a revised "primer" for DFDL. Based on the draft spec. and other information, I have made some good progress, but there are a number of topics that I don't have enough information on. This note summarizes areas where I need substantial contributions from you all. This may not be things to tackle via the mail list, I don't know. Perhaps we can discuss these issues at the F2F. 1. Use cases Actually, there should be a "Use Cases" tracker and a separate UC document, IMO. By "Use Case" I mean statements of how DFDL is intended to be used. This is different from the current example suites which illustrate the language but do not discuss why you need DFDL or what it helps you do. More on this one later. I'm developing a note with some interesting use cases I'd like to discuss. 2. Overview of the language I would like the primer to say "the DFDL is the following XSD elements, with the following restrictions. DFDL defines the following tags and attributes". Just a summary, circa 2 pages. I'm sure this informtion is implicit in the current spec and demo software, but I need help to write a clear summary. 3. Processing model I would like the primer to explain how DFDL is used, i.e., Parse DFDL, which tells how to parse data, ... Again, this is probably implicit in current work, but I need a short and clear summary. 4. Explanation of configurations, layers etc. Related to above. Explain the non-trivial processing of DFDL. Some of this awaits the outcome of trackers.

2 1

Discussion document: some non-trivial uses of DFDL
by Robert E. McGrath 23 Sep '05

23 Sep '05

As I have been studying the DFDL and working on the primer, I have been thinking about how it might be used in Grid applications. A virtual walk around NCSA has produced quite a few non-trivial use cases. What I would like to do is put some of these out to the group, for discussion. Since I am relatively new to the DFDL game, I expect that some of my ideas are naive or flat wrong. I have typed up detailed notes on 4 cases. I have a couple of other cases I'm trying to document. I attach these notes here. Sorry, it's a bit long! -- --- Robert E. McGrath, Ph.D. National Center for Supercomputing Applications University of Illinois, Urbana-Champaign Urban, Illinois 61820 (217)-333-6549 mcgrath(a)ncsa.uiuc.edu

1 0

[Fwd: BOUNCE dfdl-wg@ggf.org: Non-member submission from ["Dave Berry" <daveb@nesc.ac.uk>]]
by Martin Westhead 23 Sep '05

23 Sep '05

Forwarded from Dave Berry, advertising a BOF on Data streams at GGF.

1 0

F2F logistics - IBM Copley Place offices
by Mike Beckerle 23 Sep '05

23 Sep '05

During the upcoming GGF15 we'll be having a concurrently scheduled F2F meeting. This meeting will be hosted by IBM at its Boston facility. I have booked Conference Room 101 for Monday, Tuesday, and Wednesday all day. This seats 10 and has a speakerphone, network connection, whiteboard. I have booked Conference Room 102 for Tuesday and Wednesday all day. This seats 4 and is for smaller side discussions breakouts. It also has a speakerphone, network connection, can't recall about whiteboard. I think so. I will bring a LCD projector for slide showing. Here's information on the physical location: This map shows both Copley place where IBM's office is, and the Park Plaza Hotel (where GGF15 is being held). http://maps.google.com/maps?q=Copley+Place.+Boston+MA&ll=42.348855,-71.0739… Here's a google earth image where I've put graffiti marking the Park Plaza Hotel where GGF15 is being held, and the Copley Place complex where IBM's offices are. It's 3.5 blocks walking. It is barely worth a cab ride unless it's raining. Don't bother with the subway either. The walks to/from the subway are the same distance as walking directly. Copley place has two buildings. We're in "2" Copley place, though I don't think your average person on the street knows which building is 1 and which is 2. Number one has a Westin hotel and some shopping. If you enter there you'll need to take the walkway across to the other building which has the larger bulk of the shopping, Neiman Marcus dept. store, and the Marriott hotel, oh...and the IBM offices. Here's another Google Earth picture showing the Copley Place Buildings, which the hotels are, and roughly where in the complex you'll find IBM's offices. Once you are in the correct building, go to the very center of Copley Place Mall and look for a 3-story Waterfall. Behind it are elevators. Take these to the 'Sky lobby' (which is up 2 levels) and at the reception desk ask for IBM. You may need a security pass from the building security which will take a few minutes as they scan your picture ID and make a picture temporary badge from it for you. Everyone - BE SURE TO BRING A PICTURE ID IBMers - BE SURE TO BRING YOUR BADGE AND A PICTURE ID Then go right and the IBM office is on the left all on the same level. Here's a very low quality floorplan of the Copley Place Mall. Not sure it's helpful, but it's all I could find online. http://www.simon.com/mall/images/floorplans/784Copley_PlaceMall.gif

1 0

Telcon notes 2005-9-21
by Martin Westhead 21 Sep '05

21 Sep '05

Present ======= Martin Westhead Tara Talbot Bob McGrath Geoff Judd Suman Kalia Items ===== Agenda for sessions/F2F at GGF ------------------------------ Is registration to GGF necessary? Probably not if you are only interested in DFDL then group members can probably skip the GGF altogether. 3 day meeting Monday, Tuesday, Wednesday: GGF meetings o Monday 1100 DFDL intro session (Park Plaza) o Monday 1400 DFDL discussion session (Park Plaza) Topics (Mike will arrange them) o Summary of approach (Mike) o Summary of issues on major topics (Topic holders) o Arrays (Bob) o Scoping (Mike) o Layers (Tara) o Null and Default (Geoff) o Properties discussion (Suman) o Process (documents and timescales) (Martin) o Conclusions and priorities o Issues List (remaining time) Note: This is a reasonable list of topics based primarily on the list of trackers at the start of the meeting (of which there were 4). During the meeting we added 6 new trackers. In light of this we may want to revisit this list of topics next week and consider the priority of the new items against the old. - Martin Tracking document modifications ------------------------------- "Edit required" status added to issues list to denote items that have been discussed but for which document edits are still outstanding. Issues list (starting from item 16) ----------------------------------- Resolved 16-27 (list has been updated). Start next week on item 28. New trackers: - Properties discussion (Mike, Suman) - Open content and wild cards (Geoff, Steve) - Choice and guards (Geoff) - Use statements (Mike) - Extensibility (Martin, Tara, Jim, Mike, Bob) - Expression language (Mike, Suman, Martin) N.B. we did not discuss priorities on the call. All these tracker items have been set to medium.

1 0