- dfdl-wg - lists.ogf.org

Telcon notes 2006-06-21
by Westhead, Martin (Martin) 21 Jun '06

21 Jun '06

Present Martin Westhead Bob McGrath Mike Beckerle Geoff Judd Steve Hanson Discussion We reviewed status: * Geoff and Martin have been discussing the issues of separators at different levels * Mike has been working on bringing the concepts of properties and variables together * We still have an unassigned action to assemble a draft document from the current pieces - Mike is going to make a start on this. * There are outstanding edits to the arrays document that Bob is ready to make he is waiting for the outline document. Note: Martin will not be on the next 2 calls due to training and vacation.

1 0

Telcon now
by Westhead, Martin (Martin) 21 Jun '06

21 Jun '06

Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672 Spec document outline: Introduction section * glossary * What is DFLD? * Model Language basics * syntax * scoping and context * guards and selectors on annotations * basic arrays * variables * layering (hidden layers and value calc/referencing) Detailed semantics * properties and conversions Arrays Built-in specifications Extensions

1 0

DFDL Telcon notes
by Westhead, Martin (Martin) 14 Jun '06

14 Jun '06

Present Mike Beckerle Martin Westhead Geoff Judd Bob McGrath Discussion This was a very useful discussion. We started by formulating the outline of the spec document and then reviewed some specific aspects of the new format document. Spec document outline: Introduction section * glossary * What is DFLD? * Model Language basics * syntax * scoping and context * guards and selectors on annotations * basic arrays * variables * layering (hidden layers and value calc/referencing) Detailed semantics * properties and conversions Arrays Built-in specifications Extensions Extensibility - constructs like <dfdl:sequence> need to have their list of allowed properties extensible Format/defineFormat - is the dfdl:format tag correctly named. Martin assumed a catch-all dfdl annotation tag. Mike assumed that dfdl:format was *just* format and that other things (variable assignments, conversion definitions etc.) would go in different tags. More examples and investigation are required to see how this works in practice. defineFormat is probably too specific right now, this is an important reuse mechanism and it should be available to apply to all top level dfdl tags. One solution would be to have a more generic "define" tag that can contain "dfdl:format" tags along with others. Properties and variables - we expect that properties and variables have the same scoping, assignment and reference semantics. Properties are only different in the sense that they represent parameters to conversions. Linked properties - we discussed the need for linked properties, it seems like extra stuff that might not be required and it was useful to see if there were different ways to achieve the same goal. From the discussion it did not seem an unreasonable thing to include. Applies default - we reviewed the issues around the "applies" default (toScope or hereOnly) the current default is "hereOnly" different examples suggest different defaults are more appropriate. We decided to leave the decision where it was but a bigger selection of examples will probably help inform this decision. Actions * Assemble draft of spec document from the pieces that we have. (Currently unassigned - volunteers?)

1 0

Telcon Now
by Westhead, Martin (Martin) 14 Jun '06

14 Jun '06

(Sorry for the late notice) Main item for the agenda is again Mike's scoping document. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. https://webconferencing.avaya.com/default.htm?ConfRef=158671&Pin=139469

1 0

Telcon notes 2006-06-07
by Westhead, Martin (Martin) 07 Jun '06

07 Jun '06

Present Martin Westhead Mike Beckerle Steve Hanson Suman Kali Bob McGrath Discussion We had a short discussion where Mike gave a brief overview of some of the features of the new scoping document. We agreed on two agenda items for next week: * Detailed discussion of scoping * Assembling a complete spec document: what pieces are required? What work needs to be done? Deadline to present at GGF in DC in October.

1 0

Telcon tomorrow
by Westhead, Martin (Martin) 07 Jun '06

07 Jun '06

Main item for the agenda is Mike's scoping document. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

1 0

Defuddle Questions and Pull-Parsing Thoughts
by Tom Sugden 06 Jun '06

06 Jun '06

Hi all, Apologies for missing the telcon this week due to other work pressures. I haven't made much progress with any implementation, but have been taking a look at the Defuddle code. I have some questions that someone may be able to answer, and a few thoughts for discussion. The current Defuddle implementation is based upon JAXME, the Java/XML binding implementation. Presumably JAXME is used to generate an object model representation of the data format described by the DFDL schema. And then, I think the underlying data stream would be marshaled into an instance of that object model. Is this understanding correct? If my understanding is correct, I'm concerned that this approach may not be suitable for large data streams, since the entire object model instance would probably have to be assembled and stored in memory, like a DOM tree. Has anybody considered using a streamed pull-parsing approach instead, based upon or similar to StAX (Streaming API for XML)? I was thinking along the lines of parsing the DFDL schema into DOM or some other internal representation. Then pull-parsing the data stream, producing a sequence of StAX-like events corresponding to the data in the stream and its structure. During the pull-parsing, the context would need to be maintained and the conversion algorithm used for transforming parts of the data stream into values of the correct type. These values would then be wrapped in corresponding event objects. If this approach was viable, then these StAX-like APIs could be used to implement higher-level applications or APIs. For instance, it would be straight-forward to produce an XML serialization of any data described by a DFDL schema. One could also imagine binding any data described by a DFDL schema to auto-generated Java beans, or to a DOM object, when desirable. The process may even be reversible, so that data could be written back to a data stream as well as being read from one. I haven't thought this through very deeply yet and my understandings of the issues are still quite naive, so I will be very interested to hear any comments. Sorry if this avenue has already been explored, or I've misunderstood the mechanics of Defuddle or JAXME. Cheers, Tom

2 1

Re: [dfdl-wg] Defuddle Questions and Pull-Parsing Thoughts
by Jim Myers 02 Jun '06

02 Jun '06

Tom, The short answer (if my understanding is correct) is that we started basically as you describe but Tara has been implementing on-demand reading of data : Only structure is read up front to create empty parsing classes that get compiled. When you connect that structure to a data source(s), nothing happens to start. When you ask for an element, you either read or skip (by knowing/calculating their length) everything required to find that element's data (which may not simply be all elements above it in the schema) and return the value. I don't think we have a mechanism to free the memory of that element if you are now done with it, though I think we could. We've at least thought about StAX but mostly at the beginning of creating defuddle when it was very new, so part of our decision on JaxMe was relative maturity. So - I think we can do better with the current architecture than a straight fill-the-structure model, but for truly large data, while we might be able to scale, starting directly from a streaming approach might be better or at least more natural (presumably fits the model of the surrounding program better). Jim At 06:54 AM 6/2/2006, Tom Sugden wrote: >Hi all, > >Apologies for missing the telcon this week due to other work >pressures. I haven't made much progress with any implementation, but >have been taking a look at the Defuddle code. I have some questions >that someone may be able to answer, and a few thoughts for discussion. > >The current Defuddle implementation is based upon JAXME, the Java/XML >binding implementation. Presumably JAXME is used to generate an object >model representation of the data format described by the DFDL schema. >And then, I think the underlying data stream would be marshaled into >an instance of that object model. Is this understanding correct? > >If my understanding is correct, I'm concerned that this approach may >not be suitable for large data streams, since the entire object model >instance would probably have to be assembled and stored in memory, >like a DOM tree. Has anybody considered using a streamed pull-parsing >approach instead, based upon or similar to StAX (Streaming API for >XML)? > >I was thinking along the lines of parsing the DFDL schema into DOM or >some other internal representation. Then pull-parsing the data stream, >producing a sequence of StAX-like events corresponding to the data in >the stream and its structure. During the pull-parsing, the context >would need to be maintained and the conversion algorithm used for >transforming parts of the data stream into values of the correct type. >These values would then be wrapped in corresponding event objects. > >If this approach was viable, then these StAX-like APIs could be used >to implement higher-level applications or APIs. For instance, it would >be straight-forward to produce an XML serialization of any data >described by a DFDL schema. One could also imagine binding any data >described by a DFDL schema to auto-generated Java beans, or to a DOM >object, when desirable. The process may even be reversible, so that >data could be written back to a data stream as well as being read from >one. > >I haven't thought this through very deeply yet and my understandings >of the issues are still quite naive, so I will be very interested to >hear any comments. Sorry if this avenue has already been explored, or >I've misunderstood the mechanics of Defuddle or JAXME. > >Cheers, >Tom James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

DFDL Telcon Notes 2006-05-31
by Westhead, Martin (Martin) 31 May '06

31 May '06

Present Bob McGrath Geoff Judd Steve Hanson Mike Beckerle Discussions Scoping proposal [Mike] - simplified greatly hope to have a new version of the document by next week. Arrays document [Bob] - we discussed the arrays document modifications. Bob has added XSIL as a logical type which is very similar to the implicit internal DFDL representation. Martin wanted more details on the internal representation and was pushing for the document to present this as a possible final logical model too. UML diagrams of DFDL Parser [Geoff] - Geoff presented a pair of UML diagrams that presented a description of the operation of a DFDL parser. This is a first stage of this work and Geoff proposes to take it further and fill in some of the details.

1 0

Date: Tue, 30 May 2006 21:41:42 -0700
by Westhead, Martin (Martin) 31 May '06

31 May '06

Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

2 1