- dfdl-wg - lists.ogf.org

Hello everybody!
by Yi Zhu[MSc] 08 Jun '05

08 Jun '05

Hi, My name is Yi Zhu. I am a MSc student of EPCC, majoring in High Performance Computing. I am here because my final project is about DFDL (I am sorry for that as the project has just begun, what exactly will be implemented has not been decided), and I believe this is a interesting, challenging and promising field. However, I am a real beginer in DFDL, so I need all your help. Thanks ever so much! Regards, Yi

2 4

Conference Call on 6/8
by Mike Beckerle 03 Jun '05

03 Jun '05

Unfortunately I can't make the semi-weekly conference call on 6/8. Go ahead without me though! I have the following to report in no particular order of occurance or relevance: 1) MikeB did review ideas on layering and compositionality with Jim Myers. Went well. Jim was supportive of the approach. We have a bit more to do conceptually, but I'm very optimistic right now that we'll be able to advance something to the whole group which will be crisply defined and meet needs without excessive complexity. 2) scoping proposal from F2F was written up and posted to grid forge. Only saw one supportive comment from Martin Westhead. I'm pretty sure this matches what we discussed at the F2F and would only ask that people at least verify if it matches what they expected. 3) other F2F materials were posted on grid forge. Slides, notes, and the testcases and schemas we looked at. Of note, the testcase schema was posted. This will help us all share test/use cases as we move toward having working implementations. 4) Suman sent his latest properties model. Here in westborough we're getting close to where we'll be able to take a look at this. Bear with us Suman, we appreciate your effort, it's just taking some time to get there. 5) I am going to GGF14 in chicago. I've requested two sessions on DFDL there. One to be status/overview as usual, one to be a working session w/conference call if we can swing it. I hope we'll have time before GGF14 to get some more proposals up on grid forge, though we can't ask for any formal comment on anything by GGF14 because we're past the date for that sort of thing. I'm still hoping to have something more on (1) and (4) above in hand (posted to this mailing list or gridforge) before GGF14. Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 0

DFDL Scoping proposal document
by Mike Beckerle 27 May '05

27 May '05

At the F2F we discussed scoping of DFDL annotation properties. This document summarizes the proposal that emerged from the discussion. https://forge.gridforum.org /projects/dfdl-wg/document/DFDL_Scoping_Proposal_v001/en/1. Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

2 1

Fw: TAG opinion on XML Binary Format
by Mike Beckerle 25 May '05

25 May '05

Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA ----- Forwarded by Mike Beckerle/Worcester/IBM on 05/24/2005 01:44 PM ----- ed.rice(a)hp.com 05/24/2005 01:26 PM To www-tag(a)w3.org, public-xml-binary(a)w3.org cc Subject TAG opinion on XML Binary Format TAG opinion on XML Binary Format The TAG has reviewed in detail the documents [1,2,3,4] prepared by the XBC workgroup [5]. While we very much appreciate the significant progress that these notes represent, the TAG believes that more detailed analysis is needed before a W3C Binary XML Recommendation is sufficiently justified. We are taking no position at this time as to whether Binary XML will prove to be warranted, as there seem to be good arguments on both sides of that question. Rather, we are suggesting that further careful analysis is needed before the W3C commits to a direction. The TAG believes there are disadvantages as well as potential advantages that will result from even a well crafted Binary XML Recommendation. The advantages are clear: a successful binary format is likely to provide speed gains or size reductions, at least for certain use cases. The drawbacks are likely to include reduced interoperability with XML 1.0 and XML 1.1 software, and an inability to leverage the benefits of text-based formats. These are important concerns. Quoting from the Web Architecture document[6]: "The trade-offs between binary and textual data formats are complex and application- dependent. Binary formats can be substantially more compact, particularly for complex pointer-rich data structures. Also, they can be consumed more rapidly by agents in those cases where they can be loaded into memory and used with little or no conversion. Note, however, that such cases are relatively uncommon as such direct use may open the door to security issues that can only practically be addressed by examining every aspect of the data structure in detail. "Textual formats are usually more portable and interoperable. Textual formats also have the considerable advantage that they can be directly read by human beings (and understood, given sufficient documentation). This can simplify the tasks of creating and maintaining software, and allow the direct intervention of humans in the processing chain without recourse to tools more complex than the ubiquitous text editor. Finally, it simplifies the necessary human task of learning about new data formats; this is called the "view source" effect." We therefore believe that the benefits of a binary XML must be predictable and compelling in order to justify development of a Recommendation. In particular, we suggest that a quantitative analysis is necessary. For at least a few key use cases, concrete targets should be set for the size and/or speed gains that would be needed to justify the disruption introduced by a new format. For example, a target might be that "in typical web services scenarios, median speed gains on the order of 3x in combined parsing and deserialization are deemed sufficient to justify a new format." We further suggest that representative binary technologies be benchmarked and analyzed to a sufficient degree that such speed or size improvements can be reasonably reliably predicted before we commit to a Recommendation. No doubt, any given set of goals or benchmarks will suffer from some degree of imprecision, but if the gains are sufficiently compelling to justify a new format, then they should be relatively easy to demonstrate. In short, actual measurements should be a prerequisite to preparing a Recommendation. In doing such measurements, we believe it is essential that comparisons be done to the best possible text-based XML 1.x implementations, which are not necessarily those that are most widely deployed. Stated differently: if XML 1.x is inherently capable of meeting the needs of users, then our efforts should go into tuning our XML implementations, not designing new formats. Benchmark environments should be as representative as possible of fully optimized implementations, not just of the XML parser, but of the surrounding application or middleware stack. We note that different application-level optimizations may be necessary to maximize the performance of the Binary or text cases respectively. Care should especially be taken to ensure that the performance of particular APIs such as DOM or SAX does not obscure the performance possible with either option (e.g. both SAX and DOM can easily result in high overhead string conversions when UTF-8 is used.) The TAG would also appreciate clarification as to how many formats are likely to be included in a Recommendation; it's not clear whether the proposal is for one binary xml format for all cases, or if multiple formats are to be endorsed. The use of multiple formats is likely to further reduce interoperability. We feel that introduction of a binary format would be an important development for those who might benefit from its size or speed, but also for those who might be impacted by its impact on interoperability and perspicuity. Therefore, in order to justify a potential new format, the TAG would like to see the above issues addressed. As stated above, we make no prediction as to whether such an analysis will ultimately confirm the need for Binary XML; if it does, we will be glad to support development of a Recommendation at the W3C. [1] http://www.w3.org/TR/xbc-use-cases/ <http://www.w3.org/TR/xbc-use-cases/> [2] http://www.w3.org/TR/xbc-properties/ <http://www.w3.org/TR/xbc-properties/> [3] http://www.w3.org/TR/xbc-measurement/ <http://www.w3.org/TR/xbc-measurement/> [4] http://www.w3.org/TR/xbc-characterization/ <http://www.w3.org/TR/xbc-characterization/> [5] http://www.w3.org/XML/Binary/ <http://www.w3.org/XML/Binary/> [6] http://www.w3.org/TR/webarch/#binary <http://www.w3.org/TR/webarch/#binary>

2 1

notes from 2005-05-11 F2F Day 1
by Mike Beckerle 12 May '05

12 May '05

Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 0

only a test
by Mike Beckerle 11 May '05

11 May '05

please ignore this message Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 0

Draft Agenda for F2F
by mike.beckerle＠ascentialsoftware.com 06 May '05

06 May '05

My ideas for draft agenda. This ppt should be openable in open-office and MS Office. Please put suggestions for reordering, topics you'd like to add, topics you think are higher/lower priority, and so forth and send back to me. I'll attempt to coalese into a unified "next spin" on the agenda and I'll repost it at that time. ...mikeb

2 1

Fw: Review of Mike's 'DFDL A Proposal' working draft
by Steve Hanson 28 Apr '05

28 Apr '05

I've thought of a further scenario where scoping at the complex type/model group level is problematic. Take a C structure. A complex type might consist of some fixed length fields and some null terminated strings. If there was no scoping then I would specify a terminator for my strings and a length for the other fields. We would also have a precedent rule that stated which was used first if both were present (or we give an error). Because all my terminators are clearly hex zero, I factor that out by setting the terminator on the complex type only. How does that fit with my precedent rules - everything now has a terminator 'set'? What if I want to factor out both a common length and a terminator? We either have to create a complicated set of precedent rules, or provide additional element properties that say 'property x value is given by parent' (yuk) or we limit the properties that can be scoped to avoid such clashes. I also think that we are using scoping at type/group level as a substitute for defining a simple type that carried those dfdl properties (which we have already discounted as bad because of type explosion and schema modification issues). So we are perhaps solving the problem in an unnatural way. The more I think about this, the more I think that 1.0 should not provide scoping at the type/group level, but make sure that we don't preclude its use in the future. At the end of the day, providing scoping does not allow dfdl to describe any more documents. I think we should concentrate on getting the list of properties complete, and establishing the parser behaviour when using those properties. (Don't underestimate how long the latter task will take - I'll provide some examples at the F2F). Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/04/2005 10:38 ----- Steve Hanson/UK/IBM To 27/04/2005 12:48 DFDL Working Group cc Subject Review of Mike's 'DFDL A Proposal' working draft Here's a quick review prior to the discussion on today's call. I went through it all bar the appendices, I'm happy to defer discussion of the non-scope points to the F2F. Section 1 - Arrays - does not include unbounded maxOccurs (-1) - is that intentional ? Section 3 <See separate mail on the schema subset> Section 4 - Example shows only one dataformat appinfo per schema object. I think we need multiple (named) DFDL descriptions per schema. Example - I want to read in a COBOL structure and output it another non-XML form, eg, in CSV format. We do see this in the messaging world. One issue with this suggestion is where includes/imports are involved - clearly the names must match up. I guess for 1.0 we could go with a single unnamed DFDL description per schema, that would not preclude later addition of named DFDL descriptions. Section 5.1 - No issue with the concept, but the name 'local element scope' is misleading. On first read I assumed this applied to schema 'local elements' and not to 'global elements'. 'Local object scope' might be better? Section 6.1 - Not convinced that the benefit of scopes outweighs the complexity disadvantage. DFDL would work quite happily for the vast majority of cases with a set of properties defined per schema object, each property having a DFDL-defined default that was overridable on a per schema basis. I think we should do this for 1.0 and see whether we get a requirement for scoping from initial users. This simple scheme I suggest does not preclude introducing scoping at a later date. - If you select an object randomly in a schema it should be easy to deduce the values in force for a given property. With scoping it is not easy, as it is containment dependent. - An example of why scoping can be confusing. I have a complex type. The type has a byte alignment property of 'word', being a scoped property meaning that all child elements take a byte alignment of 'word' unless they overridde it. But it could just as easily be interpreted as meaning that an element of that complex type should be 'word' aligned. (Though I guess scoping could handle this by having a <dfdl:scope> sub-element specifically for scoped properties?). - If you recall I sent an e-mail to one of IBM's XML Schema WG reps, asking whether they had considered using property scoping. The answer was they didn't think schema would ever adopt this, but I can't recall the precise reasoning. Section 6.2 - I guess this list will grow to include xsd:all groups, element references. attribute references and local/global attributes, as discussed - We should explicitly say that simple types are excluded with reasoning, as discussed in the past. Section 6.3 - Don't understand what 'dynamically included in the definition' means? - Location (1). As hinted above in my 6.1 comment, I see nothing wrong with using a global location to change DFDL property defaults for the scope of the whole schema. This is a very useful facility. - Location (6). I don't think providing global defaults here is a good idea. It sounds ok when the element truly is top level, but what about when it is referenced by an element reference? It makes working out the defaults in force at a given point in the schema very difficult. Section 6.5 - Differing opinion already noted above. Section 6.6 - In my scope-less suggestion, the global properties in an included/imported schema apply to the objects in that schema. So you can still deduce the values in force when a random object is selected. Only wrinkle is xsd:redefine, presumably the including schema properties apply. (I would also suggest that xsd:redefine not be supported for 1.0). Section 8. - Don't understand - can you clarify ? Section 9. - I don't think this is essential for 1.0. Section 10.4 - Yes this is needed. Other examples are EDIFACT, X12 and HL7. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

Review of Mike's 'DFDL A Proposal' working draft
by Steve Hanson 27 Apr '05

27 Apr '05

Here's a quick review prior to the discussion on today's call. I went through it all bar the appendices, I'm happy to defer discussion of the non-scope points to the F2F. Section 1 - Arrays - does not include unbounded maxOccurs (-1) - is that intentional ? Section 3 <See separate mail on the schema subset> Section 4 - Example shows only one dataformat appinfo per schema object. I think we need multiple (named) DFDL descriptions per schema. Example - I want to read in a COBOL structure and output it another non-XML form, eg, in CSV format. We do see this in the messaging world. One issue with this suggestion is where includes/imports are involved - clearly the names must match up. I guess for 1.0 we could go with a single unnamed DFDL description per schema, that would not preclude later addition of named DFDL descriptions. Section 5.1 - No issue with the concept, but the name 'local element scope' is misleading. On first read I assumed this applied to schema 'local elements' and not to 'global elements'. 'Local object scope' might be better? Section 6.1 - Not convinced that the benefit of scopes outweighs the complexity disadvantage. DFDL would work quite happily for the vast majority of cases with a set of properties defined per schema object, each property having a DFDL-defined default that was overridable on a per schema basis. I think we should do this for 1.0 and see whether we get a requirement for scoping from initial users. This simple scheme I suggest does not preclude introducing scoping at a later date. - If you select an object randomly in a schema it should be easy to deduce the values in force for a given property. With scoping it is not easy, as it is containment dependent. - An example of why scoping can be confusing. I have a complex type. The type has a byte alignment property of 'word', being a scoped property meaning that all child elements take a byte alignment of 'word' unless they overridde it. But it could just as easily be interpreted as meaning that an element of that complex type should be 'word' aligned. (Though I guess scoping could handle this by having a <dfdl:scope> sub-element specifically for scoped properties?). - If you recall I sent an e-mail to one of IBM's XML Schema WG reps, asking whether they had considered using property scoping. The answer was they didn't think schema would ever adopt this, but I can't recall the precise reasoning. Section 6.2 - I guess this list will grow to include xsd:all groups, element references. attribute references and local/global attributes, as discussed - We should explicitly say that simple types are excluded with reasoning, as discussed in the past. Section 6.3 - Don't understand what 'dynamically included in the definition' means? - Location (1). As hinted above in my 6.1 comment, I see nothing wrong with using a global location to change DFDL property defaults for the scope of the whole schema. This is a very useful facility. - Location (6). I don't think providing global defaults here is a good idea. It sounds ok when the element truly is top level, but what about when it is referenced by an element reference? It makes working out the defaults in force at a given point in the schema very difficult. Section 6.5 - Differing opinion already noted above. Section 6.6 - In my scope-less suggestion, the global properties in an included/imported schema apply to the objects in that schema. So you can still deduce the values in force when a random object is selected. Only wrinkle is xsd:redefine, presumably the including schema properties apply. (I would also suggest that xsd:redefine not be supported for 1.0). Section 8. - Don't understand - can you clarify ? Section 9. - I don't think this is essential for 1.0. Section 10.4 - Yes this is needed. Other examples are EDIFACT, X12 and HL7. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

reminder: call this week
by mike.beckerle＠ascentialsoftware.com 25 Apr '05

25 Apr '05

This is to remind people that this week on wednesday one of the topics I wanted to go through was the topic of scoped annotations as described in the document on gridforge https://forge.gridforum.org/projects/dfdl-wg/document/ggf-dfdl-proposed-2005 -004/en/2 <https://forge.gridforum.org/projects/dfdl-wg/document/ggf-dfdl-proposed-200 5-004/en/2> in particular section 6 of this (you can ignore the later sections for now) I was hoping to walk through this on the call so that people's concerns about scoping of annotations can be highlighted. Thanks _____ From: Mike Beckerle Sent: Wednesday, April 13, 2005 10:17 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Notes from call 2005-04-13 Here's my notes: DFDL WG Semi-Weekly Call 2005-04-13 Attendees Jim Meyers Tara Talbot Steve Hanson Mike Beckerle Karen Loughran Agenda 1) Scope of the v1.0 specification (not to be confused with "scoping of annotations") 2) Discussion of scoping of annotations based on Ascential posting. 3) Planning of agenda for upcomign F2F meeting on May 11.12, 13, at Ascential in Westborough, MA Suggested topics so far: - Review of TD Model and extension to cover DFDL properties - Examples from the Ascl prototype - go through several line by line to see what we do and don't like about them- F2F Who's coming: Mike Beckerle Jim Meyers Tara Talbot Suman Kalia Steve Hanson Who's not coming: Martin Westhead Karen Loughran Scope of v1.0 Spec. - Steve Hanson - don't bite off too much for v1.0 - QSAM formats with data split over other pieces. - Encrypted sections and how we handle those. - DFDL Validation i.e., check that the binary data is valid according to the DFDL Schema. (It's alot of work to do a validating parser. EDIFACT, X12 and such require more complex validation than just. Things like field a if present means field b should NOT be present.) Jim Meyers - multi-layer stuff is what supports those things. Scoping rules discussion - Defer discussion - Agenda for F2F - - model and parser behavior aspects: rules for default values, null handling, how to figure out how to figure ot the length precedence rules, etc. - open content - tagged, but some of the tags are unknown. HL7 people put extra fields into a group. Happens with every format though, not just tags. Polymorphic chunks. xsd:all - unordered data. - PNNL project and Jaxme implementation - here's what's on the ground. Focus should be on how that's changing the language. - Scoping of annotations - extensibility mechanisms (low prio) - multi-dimensional arrays (low prio) - reader/writier/filter/function - topic - don't bother with a deep dive on any implementation.

1 0