April 2005 - dfdl-wg - lists.ogf.org

Hello everybody!
by Yi Zhu[MSc] 08 Jun '05

08 Jun '05

Hi, My name is Yi Zhu. I am a MSc student of EPCC, majoring in High Performance Computing. I am here because my final project is about DFDL (I am sorry for that as the project has just begun, what exactly will be implemented has not been decided), and I believe this is a interesting, challenging and promising field. However, I am a real beginer in DFDL, so I need all your help. Thanks ever so much! Regards, Yi

2 4

Draft Agenda for F2F
by mike.beckerle＠ascentialsoftware.com 06 May '05

06 May '05

My ideas for draft agenda. This ppt should be openable in open-office and MS Office. Please put suggestions for reordering, topics you'd like to add, topics you think are higher/lower priority, and so forth and send back to me. I'll attempt to coalese into a unified "next spin" on the agenda and I'll repost it at that time. ...mikeb

2 1

Fw: Review of Mike's 'DFDL A Proposal' working draft
by Steve Hanson 28 Apr '05

28 Apr '05

I've thought of a further scenario where scoping at the complex type/model group level is problematic. Take a C structure. A complex type might consist of some fixed length fields and some null terminated strings. If there was no scoping then I would specify a terminator for my strings and a length for the other fields. We would also have a precedent rule that stated which was used first if both were present (or we give an error). Because all my terminators are clearly hex zero, I factor that out by setting the terminator on the complex type only. How does that fit with my precedent rules - everything now has a terminator 'set'? What if I want to factor out both a common length and a terminator? We either have to create a complicated set of precedent rules, or provide additional element properties that say 'property x value is given by parent' (yuk) or we limit the properties that can be scoped to avoid such clashes. I also think that we are using scoping at type/group level as a substitute for defining a simple type that carried those dfdl properties (which we have already discounted as bad because of type explosion and schema modification issues). So we are perhaps solving the problem in an unnatural way. The more I think about this, the more I think that 1.0 should not provide scoping at the type/group level, but make sure that we don't preclude its use in the future. At the end of the day, providing scoping does not allow dfdl to describe any more documents. I think we should concentrate on getting the list of properties complete, and establishing the parser behaviour when using those properties. (Don't underestimate how long the latter task will take - I'll provide some examples at the F2F). Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/04/2005 10:38 ----- Steve Hanson/UK/IBM To 27/04/2005 12:48 DFDL Working Group cc Subject Review of Mike's 'DFDL A Proposal' working draft Here's a quick review prior to the discussion on today's call. I went through it all bar the appendices, I'm happy to defer discussion of the non-scope points to the F2F. Section 1 - Arrays - does not include unbounded maxOccurs (-1) - is that intentional ? Section 3 <See separate mail on the schema subset> Section 4 - Example shows only one dataformat appinfo per schema object. I think we need multiple (named) DFDL descriptions per schema. Example - I want to read in a COBOL structure and output it another non-XML form, eg, in CSV format. We do see this in the messaging world. One issue with this suggestion is where includes/imports are involved - clearly the names must match up. I guess for 1.0 we could go with a single unnamed DFDL description per schema, that would not preclude later addition of named DFDL descriptions. Section 5.1 - No issue with the concept, but the name 'local element scope' is misleading. On first read I assumed this applied to schema 'local elements' and not to 'global elements'. 'Local object scope' might be better? Section 6.1 - Not convinced that the benefit of scopes outweighs the complexity disadvantage. DFDL would work quite happily for the vast majority of cases with a set of properties defined per schema object, each property having a DFDL-defined default that was overridable on a per schema basis. I think we should do this for 1.0 and see whether we get a requirement for scoping from initial users. This simple scheme I suggest does not preclude introducing scoping at a later date. - If you select an object randomly in a schema it should be easy to deduce the values in force for a given property. With scoping it is not easy, as it is containment dependent. - An example of why scoping can be confusing. I have a complex type. The type has a byte alignment property of 'word', being a scoped property meaning that all child elements take a byte alignment of 'word' unless they overridde it. But it could just as easily be interpreted as meaning that an element of that complex type should be 'word' aligned. (Though I guess scoping could handle this by having a <dfdl:scope> sub-element specifically for scoped properties?). - If you recall I sent an e-mail to one of IBM's XML Schema WG reps, asking whether they had considered using property scoping. The answer was they didn't think schema would ever adopt this, but I can't recall the precise reasoning. Section 6.2 - I guess this list will grow to include xsd:all groups, element references. attribute references and local/global attributes, as discussed - We should explicitly say that simple types are excluded with reasoning, as discussed in the past. Section 6.3 - Don't understand what 'dynamically included in the definition' means? - Location (1). As hinted above in my 6.1 comment, I see nothing wrong with using a global location to change DFDL property defaults for the scope of the whole schema. This is a very useful facility. - Location (6). I don't think providing global defaults here is a good idea. It sounds ok when the element truly is top level, but what about when it is referenced by an element reference? It makes working out the defaults in force at a given point in the schema very difficult. Section 6.5 - Differing opinion already noted above. Section 6.6 - In my scope-less suggestion, the global properties in an included/imported schema apply to the objects in that schema. So you can still deduce the values in force when a random object is selected. Only wrinkle is xsd:redefine, presumably the including schema properties apply. (I would also suggest that xsd:redefine not be supported for 1.0). Section 8. - Don't understand - can you clarify ? Section 9. - I don't think this is essential for 1.0. Section 10.4 - Yes this is needed. Other examples are EDIFACT, X12 and HL7. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

Review of Mike's 'DFDL A Proposal' working draft
by Steve Hanson 27 Apr '05

27 Apr '05

Here's a quick review prior to the discussion on today's call. I went through it all bar the appendices, I'm happy to defer discussion of the non-scope points to the F2F. Section 1 - Arrays - does not include unbounded maxOccurs (-1) - is that intentional ? Section 3 <See separate mail on the schema subset> Section 4 - Example shows only one dataformat appinfo per schema object. I think we need multiple (named) DFDL descriptions per schema. Example - I want to read in a COBOL structure and output it another non-XML form, eg, in CSV format. We do see this in the messaging world. One issue with this suggestion is where includes/imports are involved - clearly the names must match up. I guess for 1.0 we could go with a single unnamed DFDL description per schema, that would not preclude later addition of named DFDL descriptions. Section 5.1 - No issue with the concept, but the name 'local element scope' is misleading. On first read I assumed this applied to schema 'local elements' and not to 'global elements'. 'Local object scope' might be better? Section 6.1 - Not convinced that the benefit of scopes outweighs the complexity disadvantage. DFDL would work quite happily for the vast majority of cases with a set of properties defined per schema object, each property having a DFDL-defined default that was overridable on a per schema basis. I think we should do this for 1.0 and see whether we get a requirement for scoping from initial users. This simple scheme I suggest does not preclude introducing scoping at a later date. - If you select an object randomly in a schema it should be easy to deduce the values in force for a given property. With scoping it is not easy, as it is containment dependent. - An example of why scoping can be confusing. I have a complex type. The type has a byte alignment property of 'word', being a scoped property meaning that all child elements take a byte alignment of 'word' unless they overridde it. But it could just as easily be interpreted as meaning that an element of that complex type should be 'word' aligned. (Though I guess scoping could handle this by having a <dfdl:scope> sub-element specifically for scoped properties?). - If you recall I sent an e-mail to one of IBM's XML Schema WG reps, asking whether they had considered using property scoping. The answer was they didn't think schema would ever adopt this, but I can't recall the precise reasoning. Section 6.2 - I guess this list will grow to include xsd:all groups, element references. attribute references and local/global attributes, as discussed - We should explicitly say that simple types are excluded with reasoning, as discussed in the past. Section 6.3 - Don't understand what 'dynamically included in the definition' means? - Location (1). As hinted above in my 6.1 comment, I see nothing wrong with using a global location to change DFDL property defaults for the scope of the whole schema. This is a very useful facility. - Location (6). I don't think providing global defaults here is a good idea. It sounds ok when the element truly is top level, but what about when it is referenced by an element reference? It makes working out the defaults in force at a given point in the schema very difficult. Section 6.5 - Differing opinion already noted above. Section 6.6 - In my scope-less suggestion, the global properties in an included/imported schema apply to the objects in that schema. So you can still deduce the values in force when a random object is selected. Only wrinkle is xsd:redefine, presumably the including schema properties apply. (I would also suggest that xsd:redefine not be supported for 1.0). Section 8. - Don't understand - can you clarify ? Section 9. - I don't think this is essential for 1.0. Section 10.4 - Yes this is needed. Other examples are EDIFACT, X12 and HL7. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

reminder: call this week
by mike.beckerle＠ascentialsoftware.com 25 Apr '05

25 Apr '05

This is to remind people that this week on wednesday one of the topics I wanted to go through was the topic of scoped annotations as described in the document on gridforge https://forge.gridforum.org/projects/dfdl-wg/document/ggf-dfdl-proposed-2005 -004/en/2 <https://forge.gridforum.org/projects/dfdl-wg/document/ggf-dfdl-proposed-200 5-004/en/2> in particular section 6 of this (you can ignore the later sections for now) I was hoping to walk through this on the call so that people's concerns about scoping of annotations can be highlighted. Thanks _____ From: Mike Beckerle Sent: Wednesday, April 13, 2005 10:17 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Notes from call 2005-04-13 Here's my notes: DFDL WG Semi-Weekly Call 2005-04-13 Attendees Jim Meyers Tara Talbot Steve Hanson Mike Beckerle Karen Loughran Agenda 1) Scope of the v1.0 specification (not to be confused with "scoping of annotations") 2) Discussion of scoping of annotations based on Ascential posting. 3) Planning of agenda for upcomign F2F meeting on May 11.12, 13, at Ascential in Westborough, MA Suggested topics so far: - Review of TD Model and extension to cover DFDL properties - Examples from the Ascl prototype - go through several line by line to see what we do and don't like about them- F2F Who's coming: Mike Beckerle Jim Meyers Tara Talbot Suman Kalia Steve Hanson Who's not coming: Martin Westhead Karen Loughran Scope of v1.0 Spec. - Steve Hanson - don't bite off too much for v1.0 - QSAM formats with data split over other pieces. - Encrypted sections and how we handle those. - DFDL Validation i.e., check that the binary data is valid according to the DFDL Schema. (It's alot of work to do a validating parser. EDIFACT, X12 and such require more complex validation than just. Things like field a if present means field b should NOT be present.) Jim Meyers - multi-layer stuff is what supports those things. Scoping rules discussion - Defer discussion - Agenda for F2F - - model and parser behavior aspects: rules for default values, null handling, how to figure out how to figure ot the length precedence rules, etc. - open content - tagged, but some of the tags are unknown. HL7 people put extra fields into a group. Happens with every format though, not just tags. Polymorphic chunks. xsd:all - unordered data. - PNNL project and Jaxme implementation - here's what's on the ground. Focus should be on how that's changing the language. - Scoping of annotations - extensibility mechanisms (low prio) - multi-dimensional arrays (low prio) - reader/writier/filter/function - topic - don't bother with a deep dive on any implementation.

1 0

Logistics for upcoming F2F: Location and Hotels
by mike.beckerle＠ascentialsoftware.com 20 Apr '05

20 Apr '05

<http://tinyurl.com/9t8oq> http://tinyurl.com/9t8oq is a URL to a mapquest page giving the location of our building. From there you can search for nearby hotels. I'm told the Wyndham is both nice and is quite nearby and has IBM rates for IBMers. You will need to rent a car at the airport. ...mike beckerle

1 0

Notes from call 2005-04-13
by mike.beckerle＠ascentialsoftware.com 14 Apr '05

14 Apr '05

Here's my notes: DFDL WG Semi-Weekly Call 2005-04-13 Attendees Jim Meyers Tara Talbot Steve Hanson Mike Beckerle Karen Loughran Agenda 1) Scope of the v1.0 specification (not to be confused with "scoping of annotations") 2) Discussion of scoping of annotations based on Ascential posting. 3) Planning of agenda for upcomign F2F meeting on May 11.12, 13, at Ascential in Westborough, MA Suggested topics so far: - Review of TD Model and extension to cover DFDL properties - Examples from the Ascl prototype - go through several line by line to see what we do and don't like about them- F2F Who's coming: Mike Beckerle Jim Meyers Tara Talbot Suman Kalia Steve Hanson Who's not coming: Martin Westhead Karen Loughran Scope of v1.0 Spec. - Steve Hanson - don't bite off too much for v1.0 - QSAM formats with data split over other pieces. - Encrypted sections and how we handle those. - DFDL Validation i.e., check that the binary data is valid according to the DFDL Schema. (It's alot of work to do a validating parser. EDIFACT, X12 and such require more complex validation than just. Things like field a if present means field b should NOT be present.) Jim Meyers - multi-layer stuff is what supports those things. Scoping rules discussion - Defer discussion - Agenda for F2F - - model and parser behavior aspects: rules for default values, null handling, how to figure out how to figure ot the length precedence rules, etc. - open content - tagged, but some of the tags are unknown. HL7 people put extra fields into a group. Happens with every format though, not just tags. Polymorphic chunks. xsd:all - unordered data. - PNNL project and Jaxme implementation - here's what's on the ground. Focus should be on how that's changing the language. - Scoping of annotations - extensibility mechanisms (low prio) - multi-dimensional arrays (low prio) - reader/writier/filter/function - topic - don't bother with a deep dive on any implementation.

1 0

RE: [dfdl-wg] DFDL subset of XML schema
by mike.beckerle＠ascentialsoftware.com 13 Apr '05

13 Apr '05

Hi Steve, Yes, we're not really proposing that subset for the "final" language, rather we're trying to be as minimalist as we can up front to facilitate the prototype. I think it's fine if the prototype is a subset of what we specify for v1.0. I just don't want it to be inconsistent with it. The prototype is "non-normative" in w3c lingo, so there will be two documents. One describing what the prototype does and implements, and the draft standard document can be different. Per your point 2 below. Let's split this into "single top level global element" and the others. The others cause no issue far as I can tell. In fact in our prototype it turns out that we don't even see any difference between an element with anonymous type and an element reference. The code just sees an element with name, type, etc. So those are easy to support. Attributes are easily supported also. We need to add a flag bit to our prototype that's all. The single top level issue is a tiny bit deeper. In XML you can get away with more than one global top level element because the documents are always tagged to make it unambiguous which one of the possible global top level elements describes the file. In DFDL we need specific information about which of the possible global element declarations applies to the actual file since there may be nothing in the data which makes it clear. We could do this with an annotation that indicates "this is the one that actually applies to the file". What we did in the prototype is just require there be only one global element declaration to make this unambiguous. The primary reason we left out element references is to be minimal. There's nothing you can do with them that you can't do with a type definition and an ordinary element declaration, so they seem simply unnecessary, and that solved our ambiguity problem too. Resusable groups - yes these are easily supported also modulo that they can have separate minOccurs/maxOccurs at point of use. Again I think our prototype never even sees them. The EMF XSD library essentially forward substitutes them for us so our code never deals with them. Right now we'd miss any additional min/max occurs information though, so that is a bug. Re: hexbinary - I don't understand your use of hexbinary. Can you clarify? Other simple types: yes we could put all of the date types in. We just chose to keep it minimal. I left out the obscure 'date fragment' types because I've never seen data containing things like that, but it's a very minor thing. If you think we need them, then we need them. However I would argue against putting in things for the sake of having more of XSD "covered". Substitution groups - I agree these could be a pseudo choice construct, but I prefer to make an XSD subset and be explicit about it being a subset rather than go for a way to assign meaning to everything in XSD. -----Original Message----- From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of Steve Hanson Sent: Wednesday, April 13, 2005 6:59 AM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] DFDL subset of XML schema Mike, looking at your proposal working draft, I don't agree with the DFDL subset you are proposing. I think it is too restrictive. Specifically: 1) xsd:all - we have discussed this as part of the unordered mail exchange last week so I think we now agree this is needed. 2) Single top-level global element, global attributes, element references, attribute references. This prevents re-use. 3) Reusable groups. Ditto. 4) Simple type hexBinary. This is the MRM model's default mapping for binary data. 5) Other simple types. Some of these could be discussed - eg, the date restrictions. 6) Substitution groups. We basically treat these a choice in non-XML data. But I would be ok with deferring support post 1.0. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

2 1

Call Today
by mike.beckerle＠ascentialsoftware.com 13 Apr '05

13 Apr '05

Reminder: we have our semi-weekly call today. Here's some agenda items I've gathered from emails and such: 1) Scope of the v1.0 specification (not to be confused with "scoping of annotations") 2) Discussion of scoping of annotations based on Ascential posting. 3) Planning of agenda for upcomign F2F meeting on May 11.12, 13, at Ascential in Westborough, MA Suggested topics so far: - Review of TD Model and extension to cover DFDL properties - Examples from the Ascl prototype - go through several line by line to see what we do and don't like about them- Talk to you at 9am US.PDT, Noon US.EDT, 17h00 UK Mike Beckerle Ascential Software

2 1

DFDL subset of XML schema
by Steve Hanson 13 Apr '05

13 Apr '05

Mike, looking at your proposal working draft, I don't agree with the DFDL subset you are proposing. I think it is too restrictive. Specifically: 1) xsd:all - we have discussed this as part of the unordered mail exchange last week so I think we now agree this is needed. 2) Single top-level global element, global attributes, element references, attribute references. This prevents re-use. 3) Reusable groups. Ditto. 4) Simple type hexBinary. This is the MRM model's default mapping for binary data. 5) Other simple types. Some of these could be discussed - eg, the date restrictions. 6) Substitution groups. We basically treat these a choice in non-XML data. But I would be ok with deferring support post 1.0. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0