- dfdl-wg - lists.ogf.org

Fw: TAG opinion on XML Binary Format
by Mike Beckerle 25 May '05

25 May '05

Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA ----- Forwarded by Mike Beckerle/Worcester/IBM on 05/24/2005 01:44 PM ----- ed.rice(a)hp.com 05/24/2005 01:26 PM To www-tag(a)w3.org, public-xml-binary(a)w3.org cc Subject TAG opinion on XML Binary Format TAG opinion on XML Binary Format The TAG has reviewed in detail the documents [1,2,3,4] prepared by the XBC workgroup [5]. While we very much appreciate the significant progress that these notes represent, the TAG believes that more detailed analysis is needed before a W3C Binary XML Recommendation is sufficiently justified. We are taking no position at this time as to whether Binary XML will prove to be warranted, as there seem to be good arguments on both sides of that question. Rather, we are suggesting that further careful analysis is needed before the W3C commits to a direction. The TAG believes there are disadvantages as well as potential advantages that will result from even a well crafted Binary XML Recommendation. The advantages are clear: a successful binary format is likely to provide speed gains or size reductions, at least for certain use cases. The drawbacks are likely to include reduced interoperability with XML 1.0 and XML 1.1 software, and an inability to leverage the benefits of text-based formats. These are important concerns. Quoting from the Web Architecture document[6]: "The trade-offs between binary and textual data formats are complex and application- dependent. Binary formats can be substantially more compact, particularly for complex pointer-rich data structures. Also, they can be consumed more rapidly by agents in those cases where they can be loaded into memory and used with little or no conversion. Note, however, that such cases are relatively uncommon as such direct use may open the door to security issues that can only practically be addressed by examining every aspect of the data structure in detail. "Textual formats are usually more portable and interoperable. Textual formats also have the considerable advantage that they can be directly read by human beings (and understood, given sufficient documentation). This can simplify the tasks of creating and maintaining software, and allow the direct intervention of humans in the processing chain without recourse to tools more complex than the ubiquitous text editor. Finally, it simplifies the necessary human task of learning about new data formats; this is called the "view source" effect." We therefore believe that the benefits of a binary XML must be predictable and compelling in order to justify development of a Recommendation. In particular, we suggest that a quantitative analysis is necessary. For at least a few key use cases, concrete targets should be set for the size and/or speed gains that would be needed to justify the disruption introduced by a new format. For example, a target might be that "in typical web services scenarios, median speed gains on the order of 3x in combined parsing and deserialization are deemed sufficient to justify a new format." We further suggest that representative binary technologies be benchmarked and analyzed to a sufficient degree that such speed or size improvements can be reasonably reliably predicted before we commit to a Recommendation. No doubt, any given set of goals or benchmarks will suffer from some degree of imprecision, but if the gains are sufficiently compelling to justify a new format, then they should be relatively easy to demonstrate. In short, actual measurements should be a prerequisite to preparing a Recommendation. In doing such measurements, we believe it is essential that comparisons be done to the best possible text-based XML 1.x implementations, which are not necessarily those that are most widely deployed. Stated differently: if XML 1.x is inherently capable of meeting the needs of users, then our efforts should go into tuning our XML implementations, not designing new formats. Benchmark environments should be as representative as possible of fully optimized implementations, not just of the XML parser, but of the surrounding application or middleware stack. We note that different application-level optimizations may be necessary to maximize the performance of the Binary or text cases respectively. Care should especially be taken to ensure that the performance of particular APIs such as DOM or SAX does not obscure the performance possible with either option (e.g. both SAX and DOM can easily result in high overhead string conversions when UTF-8 is used.) The TAG would also appreciate clarification as to how many formats are likely to be included in a Recommendation; it's not clear whether the proposal is for one binary xml format for all cases, or if multiple formats are to be endorsed. The use of multiple formats is likely to further reduce interoperability. We feel that introduction of a binary format would be an important development for those who might benefit from its size or speed, but also for those who might be impacted by its impact on interoperability and perspicuity. Therefore, in order to justify a potential new format, the TAG would like to see the above issues addressed. As stated above, we make no prediction as to whether such an analysis will ultimately confirm the need for Binary XML; if it does, we will be glad to support development of a Recommendation at the W3C. [1] http://www.w3.org/TR/xbc-use-cases/ <http://www.w3.org/TR/xbc-use-cases/> [2] http://www.w3.org/TR/xbc-properties/ <http://www.w3.org/TR/xbc-properties/> [3] http://www.w3.org/TR/xbc-measurement/ <http://www.w3.org/TR/xbc-measurement/> [4] http://www.w3.org/TR/xbc-characterization/ <http://www.w3.org/TR/xbc-characterization/> [5] http://www.w3.org/XML/Binary/ <http://www.w3.org/XML/Binary/> [6] http://www.w3.org/TR/webarch/#binary <http://www.w3.org/TR/webarch/#binary>

2 1

notes from 2005-05-11 F2F Day 1
by Mike Beckerle 12 May '05

12 May '05

Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 0

only a test
by Mike Beckerle 11 May '05

11 May '05

please ignore this message Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 0

Draft Agenda for F2F
by mike.beckerle＠ascentialsoftware.com 06 May '05

06 May '05

My ideas for draft agenda. This ppt should be openable in open-office and MS Office. Please put suggestions for reordering, topics you'd like to add, topics you think are higher/lower priority, and so forth and send back to me. I'll attempt to coalese into a unified "next spin" on the agenda and I'll repost it at that time. ...mikeb

2 1

Fw: Review of Mike's 'DFDL A Proposal' working draft
by Steve Hanson 28 Apr '05

28 Apr '05

I've thought of a further scenario where scoping at the complex type/model group level is problematic. Take a C structure. A complex type might consist of some fixed length fields and some null terminated strings. If there was no scoping then I would specify a terminator for my strings and a length for the other fields. We would also have a precedent rule that stated which was used first if both were present (or we give an error). Because all my terminators are clearly hex zero, I factor that out by setting the terminator on the complex type only. How does that fit with my precedent rules - everything now has a terminator 'set'? What if I want to factor out both a common length and a terminator? We either have to create a complicated set of precedent rules, or provide additional element properties that say 'property x value is given by parent' (yuk) or we limit the properties that can be scoped to avoid such clashes. I also think that we are using scoping at type/group level as a substitute for defining a simple type that carried those dfdl properties (which we have already discounted as bad because of type explosion and schema modification issues). So we are perhaps solving the problem in an unnatural way. The more I think about this, the more I think that 1.0 should not provide scoping at the type/group level, but make sure that we don't preclude its use in the future. At the end of the day, providing scoping does not allow dfdl to describe any more documents. I think we should concentrate on getting the list of properties complete, and establishing the parser behaviour when using those properties. (Don't underestimate how long the latter task will take - I'll provide some examples at the F2F). Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/04/2005 10:38 ----- Steve Hanson/UK/IBM To 27/04/2005 12:48 DFDL Working Group cc Subject Review of Mike's 'DFDL A Proposal' working draft Here's a quick review prior to the discussion on today's call. I went through it all bar the appendices, I'm happy to defer discussion of the non-scope points to the F2F. Section 1 - Arrays - does not include unbounded maxOccurs (-1) - is that intentional ? Section 3 <See separate mail on the schema subset> Section 4 - Example shows only one dataformat appinfo per schema object. I think we need multiple (named) DFDL descriptions per schema. Example - I want to read in a COBOL structure and output it another non-XML form, eg, in CSV format. We do see this in the messaging world. One issue with this suggestion is where includes/imports are involved - clearly the names must match up. I guess for 1.0 we could go with a single unnamed DFDL description per schema, that would not preclude later addition of named DFDL descriptions. Section 5.1 - No issue with the concept, but the name 'local element scope' is misleading. On first read I assumed this applied to schema 'local elements' and not to 'global elements'. 'Local object scope' might be better? Section 6.1 - Not convinced that the benefit of scopes outweighs the complexity disadvantage. DFDL would work quite happily for the vast majority of cases with a set of properties defined per schema object, each property having a DFDL-defined default that was overridable on a per schema basis. I think we should do this for 1.0 and see whether we get a requirement for scoping from initial users. This simple scheme I suggest does not preclude introducing scoping at a later date. - If you select an object randomly in a schema it should be easy to deduce the values in force for a given property. With scoping it is not easy, as it is containment dependent. - An example of why scoping can be confusing. I have a complex type. The type has a byte alignment property of 'word', being a scoped property meaning that all child elements take a byte alignment of 'word' unless they overridde it. But it could just as easily be interpreted as meaning that an element of that complex type should be 'word' aligned. (Though I guess scoping could handle this by having a <dfdl:scope> sub-element specifically for scoped properties?). - If you recall I sent an e-mail to one of IBM's XML Schema WG reps, asking whether they had considered using property scoping. The answer was they didn't think schema would ever adopt this, but I can't recall the precise reasoning. Section 6.2 - I guess this list will grow to include xsd:all groups, element references. attribute references and local/global attributes, as discussed - We should explicitly say that simple types are excluded with reasoning, as discussed in the past. Section 6.3 - Don't understand what 'dynamically included in the definition' means? - Location (1). As hinted above in my 6.1 comment, I see nothing wrong with using a global location to change DFDL property defaults for the scope of the whole schema. This is a very useful facility. - Location (6). I don't think providing global defaults here is a good idea. It sounds ok when the element truly is top level, but what about when it is referenced by an element reference? It makes working out the defaults in force at a given point in the schema very difficult. Section 6.5 - Differing opinion already noted above. Section 6.6 - In my scope-less suggestion, the global properties in an included/imported schema apply to the objects in that schema. So you can still deduce the values in force when a random object is selected. Only wrinkle is xsd:redefine, presumably the including schema properties apply. (I would also suggest that xsd:redefine not be supported for 1.0). Section 8. - Don't understand - can you clarify ? Section 9. - I don't think this is essential for 1.0. Section 10.4 - Yes this is needed. Other examples are EDIFACT, X12 and HL7. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

Review of Mike's 'DFDL A Proposal' working draft
by Steve Hanson 27 Apr '05

27 Apr '05

Here's a quick review prior to the discussion on today's call. I went through it all bar the appendices, I'm happy to defer discussion of the non-scope points to the F2F. Section 1 - Arrays - does not include unbounded maxOccurs (-1) - is that intentional ? Section 3 <See separate mail on the schema subset> Section 4 - Example shows only one dataformat appinfo per schema object. I think we need multiple (named) DFDL descriptions per schema. Example - I want to read in a COBOL structure and output it another non-XML form, eg, in CSV format. We do see this in the messaging world. One issue with this suggestion is where includes/imports are involved - clearly the names must match up. I guess for 1.0 we could go with a single unnamed DFDL description per schema, that would not preclude later addition of named DFDL descriptions. Section 5.1 - No issue with the concept, but the name 'local element scope' is misleading. On first read I assumed this applied to schema 'local elements' and not to 'global elements'. 'Local object scope' might be better? Section 6.1 - Not convinced that the benefit of scopes outweighs the complexity disadvantage. DFDL would work quite happily for the vast majority of cases with a set of properties defined per schema object, each property having a DFDL-defined default that was overridable on a per schema basis. I think we should do this for 1.0 and see whether we get a requirement for scoping from initial users. This simple scheme I suggest does not preclude introducing scoping at a later date. - If you select an object randomly in a schema it should be easy to deduce the values in force for a given property. With scoping it is not easy, as it is containment dependent. - An example of why scoping can be confusing. I have a complex type. The type has a byte alignment property of 'word', being a scoped property meaning that all child elements take a byte alignment of 'word' unless they overridde it. But it could just as easily be interpreted as meaning that an element of that complex type should be 'word' aligned. (Though I guess scoping could handle this by having a <dfdl:scope> sub-element specifically for scoped properties?). - If you recall I sent an e-mail to one of IBM's XML Schema WG reps, asking whether they had considered using property scoping. The answer was they didn't think schema would ever adopt this, but I can't recall the precise reasoning. Section 6.2 - I guess this list will grow to include xsd:all groups, element references. attribute references and local/global attributes, as discussed - We should explicitly say that simple types are excluded with reasoning, as discussed in the past. Section 6.3 - Don't understand what 'dynamically included in the definition' means? - Location (1). As hinted above in my 6.1 comment, I see nothing wrong with using a global location to change DFDL property defaults for the scope of the whole schema. This is a very useful facility. - Location (6). I don't think providing global defaults here is a good idea. It sounds ok when the element truly is top level, but what about when it is referenced by an element reference? It makes working out the defaults in force at a given point in the schema very difficult. Section 6.5 - Differing opinion already noted above. Section 6.6 - In my scope-less suggestion, the global properties in an included/imported schema apply to the objects in that schema. So you can still deduce the values in force when a random object is selected. Only wrinkle is xsd:redefine, presumably the including schema properties apply. (I would also suggest that xsd:redefine not be supported for 1.0). Section 8. - Don't understand - can you clarify ? Section 9. - I don't think this is essential for 1.0. Section 10.4 - Yes this is needed. Other examples are EDIFACT, X12 and HL7. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

reminder: call this week
by mike.beckerle＠ascentialsoftware.com 25 Apr '05

25 Apr '05

This is to remind people that this week on wednesday one of the topics I wanted to go through was the topic of scoped annotations as described in the document on gridforge https://forge.gridforum.org/projects/dfdl-wg/document/ggf-dfdl-proposed-2005 -004/en/2 <https://forge.gridforum.org/projects/dfdl-wg/document/ggf-dfdl-proposed-200 5-004/en/2> in particular section 6 of this (you can ignore the later sections for now) I was hoping to walk through this on the call so that people's concerns about scoping of annotations can be highlighted. Thanks _____ From: Mike Beckerle Sent: Wednesday, April 13, 2005 10:17 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Notes from call 2005-04-13 Here's my notes: DFDL WG Semi-Weekly Call 2005-04-13 Attendees Jim Meyers Tara Talbot Steve Hanson Mike Beckerle Karen Loughran Agenda 1) Scope of the v1.0 specification (not to be confused with "scoping of annotations") 2) Discussion of scoping of annotations based on Ascential posting. 3) Planning of agenda for upcomign F2F meeting on May 11.12, 13, at Ascential in Westborough, MA Suggested topics so far: - Review of TD Model and extension to cover DFDL properties - Examples from the Ascl prototype - go through several line by line to see what we do and don't like about them- F2F Who's coming: Mike Beckerle Jim Meyers Tara Talbot Suman Kalia Steve Hanson Who's not coming: Martin Westhead Karen Loughran Scope of v1.0 Spec. - Steve Hanson - don't bite off too much for v1.0 - QSAM formats with data split over other pieces. - Encrypted sections and how we handle those. - DFDL Validation i.e., check that the binary data is valid according to the DFDL Schema. (It's alot of work to do a validating parser. EDIFACT, X12 and such require more complex validation than just. Things like field a if present means field b should NOT be present.) Jim Meyers - multi-layer stuff is what supports those things. Scoping rules discussion - Defer discussion - Agenda for F2F - - model and parser behavior aspects: rules for default values, null handling, how to figure out how to figure ot the length precedence rules, etc. - open content - tagged, but some of the tags are unknown. HL7 people put extra fields into a group. Happens with every format though, not just tags. Polymorphic chunks. xsd:all - unordered data. - PNNL project and Jaxme implementation - here's what's on the ground. Focus should be on how that's changing the language. - Scoping of annotations - extensibility mechanisms (low prio) - multi-dimensional arrays (low prio) - reader/writier/filter/function - topic - don't bother with a deep dive on any implementation.

1 0

Logistics for upcoming F2F: Location and Hotels
by mike.beckerle＠ascentialsoftware.com 20 Apr '05

20 Apr '05

<http://tinyurl.com/9t8oq> http://tinyurl.com/9t8oq is a URL to a mapquest page giving the location of our building. From there you can search for nearby hotels. I'm told the Wyndham is both nice and is quite nearby and has IBM rates for IBMers. You will need to rent a car at the airport. ...mike beckerle

1 0

Notes from call 2005-04-13
by mike.beckerle＠ascentialsoftware.com 14 Apr '05

14 Apr '05

Here's my notes: DFDL WG Semi-Weekly Call 2005-04-13 Attendees Jim Meyers Tara Talbot Steve Hanson Mike Beckerle Karen Loughran Agenda 1) Scope of the v1.0 specification (not to be confused with "scoping of annotations") 2) Discussion of scoping of annotations based on Ascential posting. 3) Planning of agenda for upcomign F2F meeting on May 11.12, 13, at Ascential in Westborough, MA Suggested topics so far: - Review of TD Model and extension to cover DFDL properties - Examples from the Ascl prototype - go through several line by line to see what we do and don't like about them- F2F Who's coming: Mike Beckerle Jim Meyers Tara Talbot Suman Kalia Steve Hanson Who's not coming: Martin Westhead Karen Loughran Scope of v1.0 Spec. - Steve Hanson - don't bite off too much for v1.0 - QSAM formats with data split over other pieces. - Encrypted sections and how we handle those. - DFDL Validation i.e., check that the binary data is valid according to the DFDL Schema. (It's alot of work to do a validating parser. EDIFACT, X12 and such require more complex validation than just. Things like field a if present means field b should NOT be present.) Jim Meyers - multi-layer stuff is what supports those things. Scoping rules discussion - Defer discussion - Agenda for F2F - - model and parser behavior aspects: rules for default values, null handling, how to figure out how to figure ot the length precedence rules, etc. - open content - tagged, but some of the tags are unknown. HL7 people put extra fields into a group. Happens with every format though, not just tags. Polymorphic chunks. xsd:all - unordered data. - PNNL project and Jaxme implementation - here's what's on the ground. Focus should be on how that's changing the language. - Scoping of annotations - extensibility mechanisms (low prio) - multi-dimensional arrays (low prio) - reader/writier/filter/function - topic - don't bother with a deep dive on any implementation.

1 0

RE: [dfdl-wg] DFDL subset of XML schema
by mike.beckerle＠ascentialsoftware.com 13 Apr '05

13 Apr '05

Hi Steve, Yes, we're not really proposing that subset for the "final" language, rather we're trying to be as minimalist as we can up front to facilitate the prototype. I think it's fine if the prototype is a subset of what we specify for v1.0. I just don't want it to be inconsistent with it. The prototype is "non-normative" in w3c lingo, so there will be two documents. One describing what the prototype does and implements, and the draft standard document can be different. Per your point 2 below. Let's split this into "single top level global element" and the others. The others cause no issue far as I can tell. In fact in our prototype it turns out that we don't even see any difference between an element with anonymous type and an element reference. The code just sees an element with name, type, etc. So those are easy to support. Attributes are easily supported also. We need to add a flag bit to our prototype that's all. The single top level issue is a tiny bit deeper. In XML you can get away with more than one global top level element because the documents are always tagged to make it unambiguous which one of the possible global top level elements describes the file. In DFDL we need specific information about which of the possible global element declarations applies to the actual file since there may be nothing in the data which makes it clear. We could do this with an annotation that indicates "this is the one that actually applies to the file". What we did in the prototype is just require there be only one global element declaration to make this unambiguous. The primary reason we left out element references is to be minimal. There's nothing you can do with them that you can't do with a type definition and an ordinary element declaration, so they seem simply unnecessary, and that solved our ambiguity problem too. Resusable groups - yes these are easily supported also modulo that they can have separate minOccurs/maxOccurs at point of use. Again I think our prototype never even sees them. The EMF XSD library essentially forward substitutes them for us so our code never deals with them. Right now we'd miss any additional min/max occurs information though, so that is a bug. Re: hexbinary - I don't understand your use of hexbinary. Can you clarify? Other simple types: yes we could put all of the date types in. We just chose to keep it minimal. I left out the obscure 'date fragment' types because I've never seen data containing things like that, but it's a very minor thing. If you think we need them, then we need them. However I would argue against putting in things for the sake of having more of XSD "covered". Substitution groups - I agree these could be a pseudo choice construct, but I prefer to make an XSD subset and be explicit about it being a subset rather than go for a way to assign meaning to everything in XSD. -----Original Message----- From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of Steve Hanson Sent: Wednesday, April 13, 2005 6:59 AM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] DFDL subset of XML schema Mike, looking at your proposal working draft, I don't agree with the DFDL subset you are proposing. I think it is too restrictive. Specifically: 1) xsd:all - we have discussed this as part of the unordered mail exchange last week so I think we now agree this is needed. 2) Single top-level global element, global attributes, element references, attribute references. This prevents re-use. 3) Reusable groups. Ditto. 4) Simple type hexBinary. This is the MRM model's default mapping for binary data. 5) Other simple types. Some of these could be discussed - eg, the date restrictions. 6) Substitution groups. We basically treat these a choice in non-XML data. But I would be ok with deferring support post 1.0. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

2 1