- dfdl-wg - lists.ogf.org

Telcon tomorrow
by Westhead, Martin (Martin) 07 Jun '06

07 Jun '06

Main item for the agenda is Mike's scoping document. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

1 0

Defuddle Questions and Pull-Parsing Thoughts
by Tom Sugden 06 Jun '06

06 Jun '06

Hi all, Apologies for missing the telcon this week due to other work pressures. I haven't made much progress with any implementation, but have been taking a look at the Defuddle code. I have some questions that someone may be able to answer, and a few thoughts for discussion. The current Defuddle implementation is based upon JAXME, the Java/XML binding implementation. Presumably JAXME is used to generate an object model representation of the data format described by the DFDL schema. And then, I think the underlying data stream would be marshaled into an instance of that object model. Is this understanding correct? If my understanding is correct, I'm concerned that this approach may not be suitable for large data streams, since the entire object model instance would probably have to be assembled and stored in memory, like a DOM tree. Has anybody considered using a streamed pull-parsing approach instead, based upon or similar to StAX (Streaming API for XML)? I was thinking along the lines of parsing the DFDL schema into DOM or some other internal representation. Then pull-parsing the data stream, producing a sequence of StAX-like events corresponding to the data in the stream and its structure. During the pull-parsing, the context would need to be maintained and the conversion algorithm used for transforming parts of the data stream into values of the correct type. These values would then be wrapped in corresponding event objects. If this approach was viable, then these StAX-like APIs could be used to implement higher-level applications or APIs. For instance, it would be straight-forward to produce an XML serialization of any data described by a DFDL schema. One could also imagine binding any data described by a DFDL schema to auto-generated Java beans, or to a DOM object, when desirable. The process may even be reversible, so that data could be written back to a data stream as well as being read from one. I haven't thought this through very deeply yet and my understandings of the issues are still quite naive, so I will be very interested to hear any comments. Sorry if this avenue has already been explored, or I've misunderstood the mechanics of Defuddle or JAXME. Cheers, Tom

2 1

Re: [dfdl-wg] Defuddle Questions and Pull-Parsing Thoughts
by Jim Myers 02 Jun '06

02 Jun '06

Tom, The short answer (if my understanding is correct) is that we started basically as you describe but Tara has been implementing on-demand reading of data : Only structure is read up front to create empty parsing classes that get compiled. When you connect that structure to a data source(s), nothing happens to start. When you ask for an element, you either read or skip (by knowing/calculating their length) everything required to find that element's data (which may not simply be all elements above it in the schema) and return the value. I don't think we have a mechanism to free the memory of that element if you are now done with it, though I think we could. We've at least thought about StAX but mostly at the beginning of creating defuddle when it was very new, so part of our decision on JaxMe was relative maturity. So - I think we can do better with the current architecture than a straight fill-the-structure model, but for truly large data, while we might be able to scale, starting directly from a streaming approach might be better or at least more natural (presumably fits the model of the surrounding program better). Jim At 06:54 AM 6/2/2006, Tom Sugden wrote: >Hi all, > >Apologies for missing the telcon this week due to other work >pressures. I haven't made much progress with any implementation, but >have been taking a look at the Defuddle code. I have some questions >that someone may be able to answer, and a few thoughts for discussion. > >The current Defuddle implementation is based upon JAXME, the Java/XML >binding implementation. Presumably JAXME is used to generate an object >model representation of the data format described by the DFDL schema. >And then, I think the underlying data stream would be marshaled into >an instance of that object model. Is this understanding correct? > >If my understanding is correct, I'm concerned that this approach may >not be suitable for large data streams, since the entire object model >instance would probably have to be assembled and stored in memory, >like a DOM tree. Has anybody considered using a streamed pull-parsing >approach instead, based upon or similar to StAX (Streaming API for >XML)? > >I was thinking along the lines of parsing the DFDL schema into DOM or >some other internal representation. Then pull-parsing the data stream, >producing a sequence of StAX-like events corresponding to the data in >the stream and its structure. During the pull-parsing, the context >would need to be maintained and the conversion algorithm used for >transforming parts of the data stream into values of the correct type. >These values would then be wrapped in corresponding event objects. > >If this approach was viable, then these StAX-like APIs could be used >to implement higher-level applications or APIs. For instance, it would >be straight-forward to produce an XML serialization of any data >described by a DFDL schema. One could also imagine binding any data >described by a DFDL schema to auto-generated Java beans, or to a DOM >object, when desirable. The process may even be reversible, so that >data could be written back to a data stream as well as being read from >one. > >I haven't thought this through very deeply yet and my understandings >of the issues are still quite naive, so I will be very interested to >hear any comments. Sorry if this avenue has already been explored, or >I've misunderstood the mechanics of Defuddle or JAXME. > >Cheers, >Tom James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

DFDL Telcon Notes 2006-05-31
by Westhead, Martin (Martin) 31 May '06

31 May '06

Present Bob McGrath Geoff Judd Steve Hanson Mike Beckerle Discussions Scoping proposal [Mike] - simplified greatly hope to have a new version of the document by next week. Arrays document [Bob] - we discussed the arrays document modifications. Bob has added XSIL as a logical type which is very similar to the implicit internal DFDL representation. Martin wanted more details on the internal representation and was pushing for the document to present this as a possible final logical model too. UML diagrams of DFDL Parser [Geoff] - Geoff presented a pair of UML diagrams that presented a description of the operation of a DFDL parser. This is a first stage of this work and Geoff proposes to take it further and fill in some of the details.

1 0

Date: Tue, 30 May 2006 21:41:42 -0700
by Westhead, Martin (Martin) 31 May '06

31 May '06

Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

2 1

Telcon notes 2006-5-24
by Geoff Judd 24 May '06

24 May '06

Present Martin Westhead Geoff Judd 1. Conversions Had brief discussion on how the conversions work is progressing. Geoff has assigned properties to simple conversions but is trying to describe the overall algorithm using an OO class diagram that will also describe how properties relate to groups. This is only partially complete and is not ready for presentation to the group as yet but maybe be sufficiently complete by next week. Regards, Geoff Judd Websphere MQ Integrator Development IBM United Kingdom Ltd. Hursley Telephone: +44-1962-818461 E-mail: JUDDG(a)uk.ibm.com

1 0

Telcon tomorrow
by Westhead, Martin (Martin) 24 May '06

24 May '06

Hi folks, I hope to attend tomorrow but I'm heavily committed in the morning and likely to be late, if I make it - sorry. Martin Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

3 2

Telcon Notes 2006-05-17
by Westhead, Martin (Martin) 17 May '06

17 May '06

Present Martin Westhead Mike Beckerle Tom Sugden Bob McGrath Geoff Judd Steve Hanson Discussion We had no fixed agenda this week so we basically reviewed status. The following items are ongoing: * Resolving context and scope - Mike * Resolving conversions and properties - Geoff * Handling separators using conversions - Martin/all * Arrays, adding single dimension list as a logical representation - Bob

1 0

DFDL Telcon tomorrow
by Westhead, Martin (Martin) 17 May '06

17 May '06

Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

1 0

Meeting notes 2006-05-10
by Westhead, Martin (Martin) 12 May '06

12 May '06

Present Martin Westhead Bob McGrath Steve Hanson Geoff Judd Agenda Two items were discussed this week: * Document "DFDL Arrays" * Integrating conversions and properties etc. Most of the meeting was spent on the first item. Included below are Bob's notes on the discussion. The main conclusion of the discussion is that the approach seems like a very sensible direction. However it highlighted that we need to start putting in effort on pinning down a couple of the outstanding issues, context, conversions, XPath/Dpath. Steve undertook to start working on the integration of properties and convergence. Martin is going to focus on the outstanding issues with capturing our separators semantics. It was noted the Mike has been looking at bringing context and scoping together. Discussion on the document entitled "DFDL Arrays" Presents a proposed approach Comments: 1. Efficiency is an issue, particularly if there is no way for the user to know how the data is really laid out, which might lead to dramatically inefficient logical models a) this issue is not limited to DFDL b) it would be nice to have a markup to tell how the data is laid out WRT efficient access. It was argued that this is hard, and shouldn't be in the core DFDL. But if such a language exists, then it would work well with the proposed approach. 2. We hope that clever techniques a la compiler optimizations will be able to create efficient access methods. E.g., that drill through the logical layers to eliminate multiple data movements. a) want the model to make this optimization possible 3. A third logical model was suggested, which is thought to be similar to BFD. The model would be the 1D array plus a descriptor to say how to interpret it. Essentially, you may not need or want the additional XML logical model at all. a) look up BFD and correlate with that approach b) add this model to the notes 4. The Array examples highlight the need to solve two general issues: a) path expressions b) transforms, WB and BB 5. The 'occurs' is out of date, should read 'occursPath'.

1 0