- dfdl-wg - lists.ogf.org

RE: [dfdl-wg] How to handle multi-dimensional arrays
by Myers, James D 02 Mar '05

02 Mar '05

Mike, Some random quick comments - hopefully not to terse or stream-of-conciousness to be incomprehensible: This approach seems to dictate how the user will represent the array in XML (dictating their schema) rather than just describing how to pick up the right content. Which I think we agreed is a bad thing. That's not to say this isn't a reasonable way to represent a multidim array in XML, just that having DFDL go look at the attributes outside the annotation and requiring this formatting in the output XML doesn't seem right. In <dfdl:dataFormat arrayStorageOrder="@y @x"> it seems that DFDL only needs the dimension sizes for its own purposes and we could probably use our referencing mechanism to get them (i.e. if I read the two dimension ints earlier and want to reference them for the array sizes) - maybe some kind of <dfdl:runtimeoccurs> elements for the n dimensions? To allow the kind of output XML you propose, we probably need something new to allow you to loop. If the only place we need looping is for multidimensional arrays (and the special case of one dim), perhaps we can do something similar to what you propose and essentially have the array mechanism define some loop variables that can be referenced (a dfdl layer?). I don't have a full proposal thought out, but imagine defining an array as a stream that can be referenced using multiple dimensions rather than a single cursor, and having a mechanism so that the current value of the cursor(s) are available to the user. So, from the Reference.xsd example, we might want to have an attribute that shows the x value of the xdata elements analogous to the multidim example: <xs:element name="xdata" type="xs:float" maxOccurs="unbounded"> <xs:annotation> <xs:appinfo> <dfdl:runtimeoccurs>../x</dfdl:runtimeoccurs> </xs:appinfo> </xs:annotation> <xs:attribute name="x"> <dfdl:runtimevalue = #currentcursorvalue#/> </xs:attribute> </xs:element> where #currentcursorvalue# is something we have not yet made available for output (or is this available via xpath - the position of the current element in a sequence?). This would change the Reference.xml example output to have elements like <xdata x="1">2.78</xdata> <xdata x="2">3.14</xdata> So, if I can summarize/rephrase, I think we should keep the mechanism for single or multidim arrays separate from how the output is displayed, but I like the idea of making the current cursor(s) available for use, which I don't think we've done yet. And having a real multidimension construct rather than calculating them from a flat cursor is probably a requirement for scientific use, so some multidim analog of dfdl:runtimeoccurs is needed. Jim -----Original Message----- From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of mike.beckerle(a)ascentialsoftware.com Sent: Friday, February 18, 2005 4:22 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] How to handle multi-dimensional arrays We have come up with an approach to how to represent multi-dimensional arrays within XSD-described XML. The attached test file (.xml) and DFDL Schema (.dfdl.xsd) illustrate the proposed solution. The proposal does not require any changes to XSD, XML or any other special constructs outside of a single dfdl annotation to specify the storage order of the representation. I'm pretty happy with how this works out. We can handle arrays with different storage orders, like fortran style column-major vs. more common row-major, and it dovetails nicely with XPath expressions and the XSD data model. Schema validation can really do something for you, like tell you if you have all the elements of the array (if it's fixed size), and that you don't have multiple elements occupying the same array location. Those interested in multi-dimensional array support please give this some consideration. That said, I'm departing on vacation for a week, so I'll toss this out there for people to look at, but I won't be able to interact with you all on it until I get back. ...mikeb

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays
by Mike Beckerle 02 Mar '05

02 Mar '05

Thanks for the supportive comments Steve, Anyone else? Particularly those of you closer to the scientific-computing community. Would you care to comment on the multi-dimensional arrays proposal that I sent out a bit more than a week ago? ...mikeb > -----Original Message----- > From: Steve Hanson [mailto:smh@uk.ibm.com] > Sent: Wednesday, March 02, 2005 9:23 AM > To: mike.beckerle(a)ascentialsoftware.com > Cc: dfdl-wg(a)gridforum.org; owner-dfdl-wg(a)ggf.org > Subject: Re: [dfdl-wg] How to handle multi-dimensional arrays > > Hi Mike > > I will confess to having not given a great deal of thought to > multi-dim arrays in the past, but what you propose looks to be sound. > > Regards, Steve > > Steve Hanson > WebSphere Business Integration Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 --------------- > > We have come up with an approach to how to represent > multi-dimensional arrays within XSD-described XML. The > attached test file (.xml) and DFDL Schema (.dfdl.xsd) > illustrate the proposed solution. > > The proposal does not require any changes to XSD, XML or any > other special constructs outside of a single dfdl annotation > to specify the storage order of the representation. > > I'm pretty happy with how this works out. We can handle > arrays with different storage orders, like fortran style > column-major vs. more common row-major, and it dovetails > nicely with XPath expressions and the XSD data model. Schema > validation can really do something for you, like tell you if > you have all the elements of the array (if it's fixed size), > and that you don't have multiple elements occupying the same > array location. > > Those interested in multi-dimensional array support please > give this some consideration. > > That said, I'm departing on vacation for a week, so I'll toss > this out there for people to look at, but I won't be able to > interact with you all on it until I get back. > > ...mikeb > > [attachment "test2dArray.xml" deleted by Steve > Hanson/UK/IBM] [attachment "test2dArray.dfdl.xsd" deleted by > Steve Hanson/UK/IBM] >

1 0

RE: [dfdl-wg] alternate syntax for DFDL annotations
by Mike Beckerle 02 Mar '05

02 Mar '05

Sorry for missing our call today, and thanks to everyone for the rapid feedback on the non-native attributes approach. I think there's a consensus here that non-native attributes are going too far, and that if we can just make the appinfo annotations a bit more dense we'll be happy. So , for now, we're not going to do non-native attributes in our DFDL prototype, I'll make a note about the possiblility in an appendix of the doc though. None of the tools I have will validate them, and that's a big drawback. Right now, our prototype handles small examples with annotations that look like this: <xs:element name="foo" type="xs:int"> <xs:annotation> <xs:appinfo source="http://dataformat.org/"> <dfdl:dataFormat byteOrder="bigEndian" charset="UTF-8">  <dfdl:separator>,</dfdl:separator>  </dfdl:dataFormat> </xs:appinfo> </xs:annotation> </xs:element> Note that if you use appinfo annotations then you don't need namespace qualification on the attribute names. We also allow for "extension" declarations in that the dfdl:dataFormat element is the head of a XSD substitution group, So you can also define an extension like ascl:binaryIntegerFormat below: <xs:element name="foo" type="xs:int"> <xs:annotation> <xs:appinfo source="http://dataformat.org/"> <ascl:binaryIntegerFormat byteOrder="bigEndian"/> </xs:appinfo> </xs:annotation> </xs:element> The ascl:binaryIntegerFormat is a narrowing. It doesn't allow any properties to be specified that don't make sense for a binary integer. It's in a different namespace to illustrate that these could be defined by a DFDL user in their DFDL schema, not just by the dfdl implementation itself. The basic dfdl:dataFormat element allows any and all representation properties. Whether we supply standard library elements which are narrowings of it, or let those be defined by end-users is something we can discuss. This all comes for free by just using XSD properly. It adds almost nothing to the implementation. ...mikeb > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of Steve Hanson > Sent: Wednesday, March 02, 2005 1:08 PM > To: Myers, James D > Cc: dfdl-wg(a)gridforum.org; owner-dfdl-wg(a)ggf.org > Subject: RE: [dfdl-wg] alternate syntax for DFDL annotations > > > > > > I can't say I am over keen on this, because > - you will end up with odd rules where some properties can be > attributes and some can't, which will confuse users. > - you have to invent errors and/or overide behaviour when you > get a clash between attribute and annotation > - (minor) it makes the job of any software that instantiates > a dfdl memory model harder > > If we do adopt this then #2 is preferable. > > Regards, Steve > > Steve Hanson > WebSphere Business Integration Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 > > > > > "Myers, James D" > > <jim.myers(a)pnl.go > > v> > To > Sent by: dfdl-wg(a)gridforum.org > > owner-dfdl-wg@ggf > cc > .org > > > Subject > RE: [dfdl-wg] > alternate syntax for > 02/03/2005 17:35 DFDL annotations > > > > > > > > > > > > > > > > > > > I agree - I think I like #2. I know at one point in the past > we had complaints when we tried to use attributes with > namespaces - it may just have been that parsers weren't > dealing with them correctly early on. > Whatever it was, option 2 would provide a work around as well > as helping us keep a single 'canonical' form for DFDL. > > Jim > > > -----Original Message----- > > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] > On Behalf > > Of Martin Westhead > > Sent: Wednesday, March 02, 2005 12:16 PM > > To: Mike Beckerle > > Cc: dfdl-wg(a)gridforum.org > > Subject: Re: [dfdl-wg] alternate syntax for DFDL annotations > > > > > > Hi Mike, > > > > This is interesting I wasn't aware you could do this. My > concern would > > be if it turned out the schema tools couldn't handle it but I tried > > XML Spy which seems to just ignore the extra annotations. > > > > So I'd be happy to use them. As you point out we will need > both these > > and annotations so we are left with a choice. Either we: > > > > 1. Specify some (most) dfdl additions in this non-native > attribute > > form and use annotations when required or > > > > 2. We specify everything as annotations but allow this form as a > > systematic short hand. > > > > I think I prefer (2). The sort of thing I have in mind is > making the > > annotation syntax look like: > > > > <xs:element name="foo" type="xs:string"> > > <annotation> > > <appinfo> > > <dfdl:attributes dfdl:repType="text" > > dfdl:charset="UTF-8" > > dfdl:repLength="10"/> > > </appinfo> > > <annotation > > </xs:element> > > > > And then this be directly equivalent (and trivially transformed > > into/from) what you had before: > > > > <xs:element name="foo" > > type="xs:string" > > dfdl:repType="text" > > dfdl:charset="UTF-8" > > dfdl:repLength="10"/> > > > > I'm not sure if it matters that much but it seems like it might be > > neater if everything could be represented in the same form. > > > > Also there may be times when you would like to use the > annotation form > > because you have a bock of stuff that could be reused > multiple times. > > Once your annotations have become attributes in the main > schema they > > are more intimately tied to it. > > > > Cheers, > > > > Martin > > > > > > > > Mike Beckerle wrote: > > > Up til now we've considered DFDL annotations only as inside the > > > appinfo context. However, we should consider whether we should use > > non-native > > > attributes as well or > > > > > > as an alternative: E.g., simple DFDL rep properties could also be > > > expressed like this: > > > > > > <xs:element name="foo" > > > > > > type="xs:string" > > > > > > dfdl:repType="text" > > > > > > dfdl:charset="UTF-8" > > > > > > dfdl:repLength="10"/> > > > > > > > > > > > > This has the advantages of compactness, and is a fully > > supported way > > > of > > > extending XML Schema. That is, using non-native attributes is a > > > supported extension idiom. This won't handle things that > > really need the > > > syntactic support of element structure to express their > complexity, > > > e.g., things like specifying text delimiters with quoting and > > > escape-sequence specifications. For those we'll still need > > to open an > > > appinfo annotation up. However, for basic things like > > byteOrder and such > > > it is far more attractive syntactically to use non-native > > attributes > > > than appinfo annotations. > > > > > > > > > Comments? > > > > > > ...mikeb > > > > > > > > > >

1 0

RE: [dfdl-wg] Conference Call Today - notes
by Myers, James D 02 Mar '05

02 Mar '05

Sorry for the late FYI on this: If we have a meeting in Toronto (anywhere outside the US), PNNL staff will need 45-60 days to get clearance for the foreign travel. That would mean making decisions about dates and specific location as soon as possible for us to be involved. (I would hope to have at least one of us and probably two of us there.) Jim > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of mike.beckerle(a)ascentialsoftware.com > Sent: Wednesday, February 16, 2005 12:45 PM > To: dfdl-wg(a)gridforum.org > Subject: [dfdl-wg] Conference Call Today - notes > > > > DFDL-WG Conference call 2005-02-16 > > Attendees: Suman Kalia, Steve Hanson, Susan Malaika - IBM, > Tara Talbot - PNNL, Mike Beckerle - Ascential Software > > IBM offered to host a F2F meeting in mid to late April, or > early May at their Toronto facility. Seems like a good time > frame. Gives Ascential folks time time to get prototype "in > play", and leaves time before the June GGF > (Chicago) to get some things written up. > > MikeB and Suman to schedule time to work on combined model > and list of representation properties: week of Feb 28th. > > Ascential prototype: hoping to have simplest examples working > by end of this week. (No promises). Things like 2 binary > integers. MikeB to post current XSD for dfdl properties and a > few of our test files for the prototype. > > Susan asked about whether implementations of things like BFD > are available. Tara to investigate whether BFD is publicly > available now. > > We discussed when to start recruiting Oracle, Microsoft, and > other participants, since we will need them. We agreed that > our hope would be the June GGF though it really depends on > having an initial document representing some sort of > converged consensus of the current team on current directions. > > > Mike Beckerle > Ascential > >

2 1

Fw: [dfdl-wg] alternate syntax for DFDL annotations
by Suman Kalia 02 Mar '05

02 Mar '05

Hi Mike -- We will be creating XML schema to model our annotations and the attributes applicable to the representation of a particular element/type will be grouped within the complex type of the annotation schema. My preference will be use annotation approach consistently in DFDL. Moreover the attributes defined in the compact form are not easily accessible from the XML Schema APIs, you have to look into the DOMNode associated with the XSDElementDeclaration to get access to such attributes; it makes the programming task a little harder. Having said that SOAP encoded schemas use this compact form to identify the type and dimensions of the array types; some folks like this approach and some hate it.. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 03/02/2005 12:38 PM ----- Mike Beckerle <mike.beckerle(a)ascentialsoftware.com> Sent by: owner-dfdl-wg(a)ggf.org 03/01/2005 03:53 PM To dfdl-wg(a)gridforum.org cc Subject [dfdl-wg] alternate syntax for DFDL annotations Up til now we've considered DFDL annotations only as inside the appinfo context. However, we should consider whether we should use non-native attributes as well or as an alternative: E.g., simple DFDL rep properties could also be expressed like this: <xs:element name="foo" type="xs:string" dfdl:repType="text" dfdl:charset="UTF-8" dfdl:repLength="10"/> This has the advantages of compactness, and is a fully supported way of extending XML Schema. That is, using non-native attributes is a supported extension idiom. This won't handle things that really need the syntactic support of element structure to express their complexity, e.g., things like specifying text delimiters with quoting and escape-sequence specifications. For those we'll still need to open an appinfo annotation up. However, for basic things like byteOrder and such it is far more attractive syntactically to use non-native attributes than appinfo annotations. Comments? ...mikeb

1 0

RE: [dfdl-wg] alternate syntax for DFDL annotations
by Myers, James D 02 Mar '05

02 Mar '05

I agree - I think I like #2. I know at one point in the past we had complaints when we tried to use attributes with namespaces - it may just have been that parsers weren't dealing with them correctly early on. Whatever it was, option 2 would provide a work around as well as helping us keep a single 'canonical' form for DFDL. Jim > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of Martin Westhead > Sent: Wednesday, March 02, 2005 12:16 PM > To: Mike Beckerle > Cc: dfdl-wg(a)gridforum.org > Subject: Re: [dfdl-wg] alternate syntax for DFDL annotations > > > Hi Mike, > > This is interesting I wasn't aware you could do this. My > concern would > be if it turned out the schema tools couldn't handle it but I > tried XML > Spy which seems to just ignore the extra annotations. > > So I'd be happy to use them. As you point out we will need both these > and annotations so we are left with a choice. Either we: > > 1. Specify some (most) dfdl additions in this non-native attribute > form and use annotations when required or > > 2. We specify everything as annotations but allow this form as a > systematic short hand. > > I think I prefer (2). The sort of thing I have in mind is making the > annotation syntax look like: > > <xs:element name="foo" type="xs:string"> > <annotation> > <appinfo> > <dfdl:attributes dfdl:repType="text" > dfdl:charset="UTF-8" > dfdl:repLength="10"/> > </appinfo> > <annotation > </xs:element> > > And then this be directly equivalent (and trivially transformed > into/from) what you had before: > > <xs:element name="foo" > type="xs:string" > dfdl:repType="text" > dfdl:charset="UTF-8" > dfdl:repLength="10"/> > > I'm not sure if it matters that much but it seems like it might be > neater if everything could be represented in the same form. > > Also there may be times when you would like to use the > annotation form > because you have a bock of stuff that could be reused multiple times. > Once your annotations have become attributes in the main > schema they are > more intimately tied to it. > > Cheers, > > Martin > > > > Mike Beckerle wrote: > > Up til now we've considered DFDL annotations only as inside the > > appinfo > > context. However, we should consider whether we should use > non-native > > attributes as well or > > > > as an alternative: E.g., simple DFDL rep properties could also be > > expressed like this: > > > > <xs:element name="foo" > > > > type="xs:string" > > > > dfdl:repType="text" > > > > dfdl:charset="UTF-8" > > > > dfdl:repLength="10"/> > > > > > > > > This has the advantages of compactness, and is a fully > supported way > > of > > extending XML Schema. That is, using non-native attributes is a > > supported extension idiom. This won't handle things that > really need the > > syntactic support of element structure to express their complexity, > > e.g., things like specifying text delimiters with quoting and > > escape-sequence specifications. For those we'll still need > to open an > > appinfo annotation up. However, for basic things like > byteOrder and such > > it is far more attractive syntactically to use non-native > attributes > > than appinfo annotations. > > > > > > Comments? > > > > ...mikeb > > > >

2 1

DFDL call notes
by Myers, James D 02 Mar '05

02 Mar '05

FYI - quick notes about a short call today - Steve, Tara, and I hung out on the DFDL call today for about 20 minutes to see if we'd get a quorum. We didn't. We noted that the availability of the prototype from Ascential seems to be a pre-requisite for further discussions about the reader/writer/filter/function ideas and resolving the issues around transform versus sub-type centric approaches. (I'm glad its not me in the critical path this time!) In any case, I think we agreed that we need the prototype, or some description of it, soon to get the discussion moving again and make another productive f-2-f possible. Jim

1 0

alternate syntax for DFDL annotations
by Mike Beckerle 02 Mar '05

02 Mar '05

Up til now we've considered DFDL annotations only as inside the appinfo context. However, we should consider whether we should use non-native attributes as well or as an alternative: E.g., simple DFDL rep properties could also be expressed like this: <xs:element name="foo" type="xs:string" dfdl:repType="text" dfdl:charset="UTF-8" dfdl:repLength="10"/> This has the advantages of compactness, and is a fully supported way of extending XML Schema. That is, using non-native attributes is a supported extension idiom. This won't handle things that really need the syntactic support of element structure to express their complexity, e.g., things like specifying text delimiters with quoting and escape-sequence specifications. For those we'll still need to open an appinfo annotation up. However, for basic things like byteOrder and such it is far more attractive syntactically to use non-native attributes than appinfo annotations. Comments? ...mikeb

2 1

How to handle multi-dimensional arrays
by mike.beckerle＠ascentialsoftware.com 02 Mar '05

02 Mar '05

We have come up with an approach to how to represent multi-dimensional arrays within XSD-described XML. The attached test file (.xml) and DFDL Schema (.dfdl.xsd) illustrate the proposed solution. The proposal does not require any changes to XSD, XML or any other special constructs outside of a single dfdl annotation to specify the storage order of the representation. I'm pretty happy with how this works out. We can handle arrays with different storage orders, like fortran style column-major vs. more common row-major, and it dovetails nicely with XPath expressions and the XSD data model. Schema validation can really do something for you, like tell you if you have all the elements of the array (if it's fixed size), and that you don't have multiple elements occupying the same array location. Those interested in multi-dimensional array support please give this some consideration. That said, I'm departing on vacation for a week, so I'll toss this out there for people to look at, but I won't be able to interact with you all on it until I get back. ...mikeb

2 1

BFD Parser Engine
by Talbott, Tara D 22 Feb '05

22 Feb '05

During the last meeting Susan inquired about the implementation of the BFD parser. Here is a link to the BFD homepage which describes the project and specifications, http://collaboratory.emsl.pnl.gov/sam/bfd/. The parser engine is available from that page, but we are still investigating the status of the open source license so the source is not yet available, I will notifiy you when it is. Thanks, Tara

1 0