- dfdl-wg - lists.ogf.org

Telcon notes 2006-02-08
by Westhead, Martin (Martin) 08 Feb '06

08 Feb '06

Present: -------- Martin Westhead Tara Talbott Bob McGrath Geoff Judd Mike Bekerele Discussion ---------- 1. F2F Agenda The following topics/goals were raised for inclusion on the agenda: Conversions - basics: do we have a common understanding? Do we have consensus as this as a direction? Conversions - does the current proposal sufficiently address our extensibility needs (e.g. arrays, other trackers)? Conversions - can we capture the meaning of the existing properties effectively within this framework. Process - arriving at a clear set of milestones/actions with people attached to them required to reach a standards document. 2. Conversions discussion Geoff raised an issue that he was concerned about ordinary users being exposed to a lot of boiler-plate. Although this could be avoided altogether it was agreed that including a standard definitions file might be a sensible way to do it. Tara wanted to know if the whole stack of conversions could be specified. This seems like a sensible suggestion. It should be straight forward to specify all or even part of the conversion stack.

1 0

Updated documents
by Westhead, Martin (Martin) 06 Feb '06

06 Feb '06

Hi folks, I have taken another pass at the conversions material. Here are two documents that came from the document that was "Conversions". The first is still called "Conversions" and focuses entirely on the semantics of conversions and how they are chosen. The second document contains all the material that could be considered "extensions" (so that's what I called it :-) ) The intention here is to separate out the motivational issues. IMO we need the material in the first document in order to explain the operation of a DFDL parser. (I do not claim that this is the right material just that we need an explanation of how this works and this is a proposal to meet that need). The second document contains material that may be regarded differently in terms of implementation cost/benefit by different groups in the WG and we should discuss which bits make sense as required (for all implementations) and which might be optional. I have included new examples in both documents, inspired by Steve's suggestions. I am aware that all of this material needs the details attending to. There are a number of design decisions that I made arbitrarily in order to get a document out to try to communicate the broad ideas. Assuming the ideas are adopted we will need to work through all the details of the syntax etc. Mike brought up a number of such issues in his response to the first draft of this stuff last week. I have not attempted to really answer these yet, I am still working at communicating the ideas to the group as a whole. I did make some minor changes to the syntax largely to try to make the ideas clearer. Please let me know what you think. Thanks, Martin

2 1

RE: [dfdl-wg] More documents
by Westhead, Martin (Martin) 04 Feb '06

04 Feb '06

Wouldn't repType="xml" make more sense? >-----Original Message----- >From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of >Steve Hanson >Sent: Thursday, February 02, 2006 7:06 AM >To: dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] More documents > >I'd forgotten the isXML property - that covers this nicely. > >I'll stick with the remaining two: > >- A portion of the data is encrypted, with fields in the message prior to >the encrypted section providing the decryption keys etc. (X12 security >segment motivates this) >- Data where decimal fields (say) are in a wacky encoding not supported by >stock DFDL properties (TLOG retail standard motivates this) > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM Hursley, England >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > Mike Beckerle > <beckerle(a)us.ibm. > com> To > Steve Hanson/UK/IBM@IBMGB > 02/02/2006 13:41 cc > dfdl-wg(a)ggf.org, > owner-dfdl-wg(a)ggf.org > Subject > Re: [dfdl-wg] More documents > > > > > > > > > > > >I would second this approach. A payload string of XML data is just a string >of value content to us. > >Note however that in our proposed set of properties there is one "isXML" >which is intended to facilitate the usage pattern of XML payload strings. >This property is a boolean you can set to say that the string's content is >a well formed XML document or a well-formed fragment of XML. This is just a >shorthand for what would otherwise be a large set of quoting/escaping >conventions, the use of a dynamic character set selected based on the >encoding attribute in the <?xml version="1.0" encoding="US-ASCII"?> slug >line (if present), etc. > >(We would need to specify what the concept of "well formed fragment of XML" >means. I think intuitively people know what this means, something >intelligible to an XML parser, but we need to be explicit. It means a >fragment of XML that begins and ends between two elements. Hence, is not a >fragment that starts in the middle of any quoting construct, nor in the >middle of a tag or attribute, etc. ) > >Mike Beckerle >STSM, Architect, Scalable Computing >IBM Software Group >Information Integration Solutions >Westborough, MA 01581 >voice and FAX 508-599-7148 >home/mobile office 508-915-4767 > > > > > Steve Hanson <smh(a)uk.ibm.com> > Sent by: owner-dfdl-wg(a)ggf.org > > To > 02/01/2006 01:27 PM dfdl-wg(a)ggf.org > cc > > Subject > Re: [dfdl-wg] More > documents > > > > > > > > > > > > >I'll see what I can come up with. > >As far as the embedded XML goes, I put it there as we will be asked this >question. Thinking it through, maybe we should simply treat it as a BLOB >and leave it to the user to take and parse using an XML parser as an >independent operation. This is symmetric with an XML document containing a >non-XML BLOB as CDATA that needed to be parsed using DFDL. > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM Hursley, England >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > "Robert E. > McGrath" > <mcgrath(a)ncsa.uiu To > c.edu> dfdl-wg(a)ggf.org > Sent by: cc > owner-dfdl-wg@ggf > .org Subject > Re: [dfdl-wg] More documents > > 01/02/2006 15:03 > > > > > > > > >On Wednesday 01 February 2006 07:03, Steve Hanson wrote: > >> - A portion of the data is encrypted, with fields in the message prior to >> the encrypted section providing the decryption keys etc. (X12 security >> segment motivates this) >> - Data where some XML is embedded in the middle >> - Data where decimal fields (say) are in a wacky encoding not supported >by >> stock DFDL properties (TLOG retail standard motivates here) >> > >Thease are great examples. Can someone give me fully documented >data files from which to try to construct such examples? > >By the way, I'm not sure whether embedded XML is within the scope >of DFDL--it gets insanely hairy. > >But the others are exactly the kinds of things that the core standard >must either cover or have an extension mechanism that covers. > > >-- >--- >Robert E. McGrath, Ph.D. >National Center for Supercomputing Applications >University of Illinois, Urbana-Champaign >1205 West Clark >Urbana, Illinois 61801 >(217)-333-6549 > >mcgrath(a)ncsa.uiuc.edu > > > >

1 0

RE: [dfdl-wg] More documents
by Westhead, Martin (Martin) 02 Feb '06

02 Feb '06

Sorry to be a latecomer to this thread. I don't think embedded XML should provide any problems at the description level. I gave it some thought when I was writing the document. There is a very straightforward way to handle it within the set of mechanisms proposed. All you need is a conversion that does bytes -> XML (i.e. an XML parser) you end up with a stream of the root XML type. The conversion system will look at this and compare it to what it is expecting. In the absence of any alternative conversions, and assuming the types match it will apply the "null" conversion and drop the XML in place. I can include this example. Cheers, Martin >-----Original Message----- >From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of >Robert E. McGrath >Sent: Thursday, February 02, 2006 8:14 AM >To: Jim Myers; dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] More documents > >The main thing about XML is that it is very difficult to interpret >an arbitrary fragment within the context of another XML. E.g., >suppose there was a dfdl annotion inside the fragment? What >would that mean? DFDL to describe DFDL? Ick! > >Some kind of BLOB plus interpreter gives the necessary >mechanics. E.g., where conventions exist for such fragments, >standard code can check and dig out the stuff--this would be >published as a DFDL extension. (I think I'm just agreeing with Jim.) > >On Thursday 02 February 2006 09:29, Jim Myers wrote: >> One could certainly imagine a case where there's a text integer in >> the XML that is required for parsing subsequent data... I don't know >> that we have to support this case directly/to start, but I think we >> should consider a use case where Joe User is very excited by the >> power of DFDL to deal with his complex information and has now run >> across a case where info within an XML fragment in his doc is >> required for further parsing, he wants to use DFDL to get the >> fragment, has an XML parser he could connect (assuming a DFDL parser >> that supported extensions), and then needs a way to use one of the >> nodes identified as a source for further DFDL processing (the >> /mydata/length node contains a string representing the length of a >> subsequent array and he needs to get that string, convert to int >> using standard DFDL and use that as an input for the size of the >> array). The use case without extensibility is that Joe waits for DFDL >> 2.0 when we add an XML parser and he'd now like to do the same thing >> using his knowledge of DFDL 1.0 to process the string to int and use >> it as a repeat count for an array. >> >> Jim >> >> At 07:41 AM 2/2/2006, Mike Beckerle wrote: >> >I would second this approach. A payload string of XML data is just a >> >string of value content to us. >> > >> >Note however that in our proposed set of properties there is one >> >"isXML" which is intended to facilitate the usage pattern of XML >> >payload strings. This property is a boolean you can set to say that >> >the string's content is a well formed XML document or a well-formed >> >fragment of XML. This is just a shorthand for what would otherwise >> >be a large set of quoting/escaping conventions, the use of a dynamic >> >character set selected based on the encoding attribute in the <?xml >> >version="1.0" encoding="US-ASCII"?> slug line (if present), etc. >> > >> >(We would need to specify what the concept of "well formed fragment >> >of XML" means. I think intuitively people know what this means, >> >something intelligible to an XML parser, but we need to be explicit. >> >It means a fragment of XML that begins and ends between two >> >elements. Hence, is not a fragment that starts in the middle of any >> >quoting construct, nor in the middle of a tag or attribute, etc. ) >> > >> >Mike Beckerle >> >STSM, Architect, Scalable Computing >> >IBM Software Group >> >Information Integration Solutions >> >Westborough, MA 01581 >> >voice and FAX 508-599-7148 >> >home/mobile office 508-915-4767 >> > >> > >> > >> > >> >Steve Hanson <smh(a)uk.ibm.com> >> >Sent by: owner-dfdl-wg(a)ggf.org >> > >> >02/01/2006 01:27 PM >> >To >> >dfdl-wg(a)ggf.org >> >cc >> >Subject >> >Re: [dfdl-wg] More documents >> > >> > >> > >> > >> >I'll see what I can come up with. >> > >> >As far as the embedded XML goes, I put it there as we will be asked this >> >question. Thinking it through, maybe we should simply treat it as a BLOB >> >and leave it to the user to take and parse using an XML parser as an >> >independent operation. This is symmetric with an XML document containing >a >> >non-XML BLOB as CDATA that needed to be parsed using DFDL. >> > >> >Regards, Steve >> > >> >Steve Hanson >> >WebSphere Message Brokers, >> >IBM Hursley, England >> >Internet: smh(a)uk.ibm.com >> >Phone (+44)/(0) 1962-815848 >> > >> > >> > >> > "Robert E. >> > McGrath" >> > <mcgrath(a)ncsa.uiu >To >> > c.edu> dfdl-wg(a)ggf.org >> > Sent by: >cc >> > owner-dfdl-wg@ggf >> > .org >Subject >> > Re: [dfdl-wg] More documents >> > >> > 01/02/2006 15:03 >> > >> >On Wednesday 01 February 2006 07:03, Steve Hanson wrote: >> > > - A portion of the data is encrypted, with fields in the message >prior >> > > to the encrypted section providing the decryption keys etc. (X12 >> > > security segment motivates this) >> > > - Data where some XML is embedded in the middle >> > > - Data where decimal fields (say) are in a wacky encoding not >supported >> > >> >by >> > >> > > stock DFDL properties (TLOG retail standard motivates here) >> > >> >Thease are great examples. Can someone give me fully documented >> >data files from which to try to construct such examples? >> > >> >By the way, I'm not sure whether embedded XML is within the scope >> >of DFDL--it gets insanely hairy. >> > >> >But the others are exactly the kinds of things that the core standard >> >must either cover or have an extension mechanism that covers. >> > >> > >> >-- >> >--- >> >Robert E. McGrath, Ph.D. >> >National Center for Supercomputing Applications >> >University of Illinois, Urbana-Champaign >> >1205 West Clark >> >Urbana, Illinois 61801 >> >(217)-333-6549 >> > >> >mcgrath(a)ncsa.uiuc.edu >> >> James D. Myers >> Associate Director, Collaborative Systems, NCSA >> 1205 W. Clark St, MC-257 >> Urbana, IL 61801 >> 217-244-1934 >> jimmyers(a)ncsa.uiuc.edu > >-- >--- >Robert E. McGrath, Ph.D. >National Center for Supercomputing Applications >University of Illinois, Urbana-Champaign >1205 West Clark >Urbana, Illinois 61801 >(217)-333-6549 > >mcgrath(a)ncsa.uiuc.edu

1 0

Re: [dfdl-wg] More documents
by Jim Myers 02 Feb '06

02 Feb '06

One could certainly imagine a case where there's a text integer in the XML that is required for parsing subsequent data... I don't know that we have to support this case directly/to start, but I think we should consider a use case where Joe User is very excited by the power of DFDL to deal with his complex information and has now run across a case where info within an XML fragment in his doc is required for further parsing, he wants to use DFDL to get the fragment, has an XML parser he could connect (assuming a DFDL parser that supported extensions), and then needs a way to use one of the nodes identified as a source for further DFDL processing (the /mydata/length node contains a string representing the length of a subsequent array and he needs to get that string, convert to int using standard DFDL and use that as an input for the size of the array). The use case without extensibility is that Joe waits for DFDL 2.0 when we add an XML parser and he'd now like to do the same thing using his knowledge of DFDL 1.0 to process the string to int and use it as a repeat count for an array. Jim At 07:41 AM 2/2/2006, Mike Beckerle wrote: >I would second this approach. A payload string of XML data is just a >string of value content to us. > >Note however that in our proposed set of properties there is one >"isXML" which is intended to facilitate the usage pattern of XML >payload strings. This property is a boolean you can set to say that >the string's content is a well formed XML document or a well-formed >fragment of XML. This is just a shorthand for what would otherwise >be a large set of quoting/escaping conventions, the use of a dynamic >character set selected based on the encoding attribute in the <?xml >version="1.0" encoding="US-ASCII"?> slug line (if present), etc. > >(We would need to specify what the concept of "well formed fragment >of XML" means. I think intuitively people know what this means, >something intelligible to an XML parser, but we need to be explicit. >It means a fragment of XML that begins and ends between two >elements. Hence, is not a fragment that starts in the middle of any >quoting construct, nor in the middle of a tag or attribute, etc. ) > >Mike Beckerle >STSM, Architect, Scalable Computing >IBM Software Group >Information Integration Solutions >Westborough, MA 01581 >voice and FAX 508-599-7148 >home/mobile office 508-915-4767 > > > > >Steve Hanson <smh(a)uk.ibm.com> >Sent by: owner-dfdl-wg(a)ggf.org > >02/01/2006 01:27 PM >To >dfdl-wg(a)ggf.org >cc >Subject >Re: [dfdl-wg] More documents > > > > >I'll see what I can come up with. > >As far as the embedded XML goes, I put it there as we will be asked this >question. Thinking it through, maybe we should simply treat it as a BLOB >and leave it to the user to take and parse using an XML parser as an >independent operation. This is symmetric with an XML document containing a >non-XML BLOB as CDATA that needed to be parsed using DFDL. > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM Hursley, England >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > "Robert E. > McGrath" > <mcgrath(a)ncsa.uiu To > c.edu> dfdl-wg(a)ggf.org > Sent by: cc > owner-dfdl-wg@ggf > .org Subject > Re: [dfdl-wg] More documents > > 01/02/2006 15:03 > > > > > > > > >On Wednesday 01 February 2006 07:03, Steve Hanson wrote: > > > - A portion of the data is encrypted, with fields in the message prior to > > the encrypted section providing the decryption keys etc. (X12 security > > segment motivates this) > > - Data where some XML is embedded in the middle > > - Data where decimal fields (say) are in a wacky encoding not supported >by > > stock DFDL properties (TLOG retail standard motivates here) > > > >Thease are great examples. Can someone give me fully documented >data files from which to try to construct such examples? > >By the way, I'm not sure whether embedded XML is within the scope >of DFDL--it gets insanely hairy. > >But the others are exactly the kinds of things that the core standard >must either cover or have an extension mechanism that covers. > > >-- >--- >Robert E. McGrath, Ph.D. >National Center for Supercomputing Applications >University of Illinois, Urbana-Champaign >1205 West Clark >Urbana, Illinois 61801 >(217)-333-6549 > >mcgrath(a)ncsa.uiuc.edu > > > James D. Myers Associate Director, Collaborative Systems, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

2 1

Directions and hotels for Avaya Milpitas
by Westhead, Martin (Martin) 02 Feb '06

02 Feb '06

Hi Folks, For those coming to the F2F I have the attached information on hotels and directions. The Crowne Plaza seems nice and is very close - I have never stayed there. I did stay at the Courtyard Marriott once, a long time ago and its fine except it has no bar or restaurant. Martin

3 2

RE: [dfdl-wg] More documents
by Westhead, Martin (Martin) 02 Feb '06

02 Feb '06

Hi Steve, That is a good suggestion. An important point that I can try to make, in the abstract is that both cases you outline would have very similar annotations on the logical model. That part of it would not need to change. The explicit declarations are additional material that would (typically) be presented to the compiler in advance. I don't think its particularly important for us to get hung up on the declaration of new conversions at this stage. After discussing this with Mike, I think it makes sense to separate that section off into a new document, that we could discuss separately. At the heart of what I think is important in this conversions document is the semantics for describing the conversions and how they are chosen. In order for the standard to be comprehensible and comprehensive in its description of DFDL I think we need to nail down these details. It is, if you like a useful side effect, that doing so provides the hooks to allow extensibility, but I think it is important in of itself to have a clear description of how that all works. In principle we should be able to hand this standards doc to two developers, lock them in separate rooms and have them build two DFDL parsers that do the same thing for all the many complex corner cases and I think we need a framework like the one I have outlined to get that to work. That said I think there are a lot of issues with what I laid out, Mike identified many of them, which need attention and group discussion. Talk to you tomorrow, Martin >-----Original Message----- >From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of >Steve Hanson >Sent: Tuesday, January 31, 2006 1:41 AM >To: dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] More documents > >I've read the Conversions document and to be honest I had a great deal of >trouble trying to work out what was going on. I'd like to see an example of >a DFDL schema that models a text stream that contains (say) a complex type >with string, integer and decimal children, with DFDL properties used to >handle the implicit conversions from text to logical data type. Then I'd >like to see the same DFDL schema but with equivalent conversions explicitly >declared and used instead of the DFDL properties. > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM Hursley, England >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > Mike Beckerle > <beckerle(a)us.ibm. > com> To > Sent by: Mike Beckerle <beckerle(a)us.ibm.com> > owner-dfdl-wg@ggf cc > .org dfdl-wg(a)ggf.org, > owner-dfdl-wg(a)ggf.org, "Westhead, > Martin (Martin)" > 28/01/2006 04:17 <westhead(a)avaya.com> > Subject > Re: [dfdl-wg] More documents > > > > > > > > > > > >Oh yeah, forgot the attachment. >Mike Beckerle >STSM, Architect, Scalable Computing >IBM Software Group >Information Integration Solutions >Westborough, MA 01581 >voice and FAX 508-599-7148 >home/mobile office 508-915-4767 > > > > > Mike > Beckerle/Worcester/IBM@IBMUS > Sent by: owner-dfdl-wg(a)ggf.org > To > "Westhead, Martin (Martin)" > 01/27/2006 07:50 PM <westhead(a)avaya.com> > cc > dfdl-wg(a)ggf.org, > owner-dfdl-wg(a)ggf.org > Subject > Re: [dfdl-wg] More documents > > > > > > > > > > > > > >Sorry I missed the call this past week. Here are my comments and some >in-line editing on the > > >I actually think the primary reason to push this document forward is NOT >extensibility, but because it is needed to clarify the basic semantics of >DFDL. That is, it allows us to make very clear exactly what properties in >the annotations are guiding the selection of which conversion functions. > >...mikeb > >Mike Beckerle >STSM, Architect, Scalable Computing >IBM Software Group >Information Integration Solutions >Westborough, MA 01581 >voice and FAX 508-599-7148 >home/mobile office 508-915-4767 > > > > "Westhead, Martin (Martin)" > <westhead(a)avaya.com> > Sent by: owner-dfdl-wg(a)ggf.org > To > <dfdl-wg@ggf > 01/23/2006 04:58 PM .org> > cc > > Subject > [dfdl-wg] > More > documents > > > > > > > > > > > > > > >These are documents 3 and 4 of a set of 3 :). > >The documents are: >- bundles - a reusable block of DFDL statements >- Conversions - a proposal for the semantic details as to how >conversions are chosen during the operation of a DFDL parser. These >semantics include a description as to how the conversions can be extended. > >These documents round off the current set of proposals from the >extensibility design team. The team has reasonable consensus on these >documents, with the (relatively trivial) exception of bundles where there >may be some outstanding issues. We believe that with these two documents >there is enough material to support the necessary extensibility. However, >we have not yet worked through enough examples of round tripping to >understand whether that is sufficiently well covered and we are expecting >that all the ideas in these documents will require refining by the group as >a whole. > >I would like to discuss these two documents on Wednesday (if possible). > >Thanks, > >Martin[attachment "DFDL_Conversions_4.doc" deleted by Mike >Beckerle/Worcester/IBM] [attachment "DFDL_Bundles_2.doc" deleted by Mike >Beckerle/Worcester/IBM] > >(See attached file: DFDL_Conversions_4.doc)

4 5

Notes on telcon 2006-31-1
by Westhead, Martin (Martin) 01 Feb '06

01 Feb '06

Present Martin Westhead Bob McGrath Mike Beckerle Geoff Judd Jim Myers Suman Kali F2F Agenda DFDL underlying mechanisms Conversions/context/hidden elements - Relationship to variables Scoping document Choices and wildcards Null and default values Actions to lead to a final document set. - What APIs to we expect DFDL implementations to provide to users - Which XML Schema constructs are supported - The set of DFDL properties we provide - Review of trackers Conversions discussion It was proposed that the document is split to try to separate those aspects of the document which are clearly "extensions" from those which are defining underlying semantics. There is clearly a need to provide one or more detailed worked examples which go through all the operations of conversion choice and context. (Martin will undertake to try to do at least one for next week, Bob has also volunteered to contribute material here)

1 0

Telcon agenda for tomorrow
by Westhead, Martin (Martin) 01 Feb '06

01 Feb '06

Preliminary Agenda ------------------ 1. Agenda for F2F 2. Reports from trackers 3. Ongoing review "Conversions" document 4. Review "Bundles" document Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference Reference: 158672 Security Code : 785343

1 0

More documents
by Westhead, Martin (Martin) 31 Jan '06

31 Jan '06

These are documents 3 and 4 of a set of 3 :). The documents are: - bundles - a reusable block of DFDL statements - Conversions - a proposal for the semantic details as to how conversions are chosen during the operation of a DFDL parser. These semantics include a description as to how the conversions can be extended. These documents round off the current set of proposals from the extensibility design team. The team has reasonable consensus on these documents, with the (relatively trivial) exception of bundles where there may be some outstanding issues. We believe that with these two documents there is enough material to support the necessary extensibility. However, we have not yet worked through enough examples of round tripping to understand whether that is sufficiently well covered and we are expecting that all the ideas in these documents will require refining by the group as a whole. I would like to discuss these two documents on Wednesday (if possible). Thanks, Martin

4 4