- dfdl-wg - lists.ogf.org

RE: [dfdl-wg] How to handle multi-dimensional arrays
by Mike Beckerle 03 Mar '05

03 Mar '05

Thanks for the supportive comments Steve, Anyone else? Particularly those of you closer to the scientific-computing community. Would you care to comment on the multi-dimensional arrays proposal that I sent out a bit more than a week ago? ...mikeb > -----Original Message----- > From: Steve Hanson [mailto:smh@uk.ibm.com] > Sent: Wednesday, March 02, 2005 9:23 AM > To: mike.beckerle(a)ascentialsoftware.com > Cc: dfdl-wg(a)gridforum.org; owner-dfdl-wg(a)ggf.org > Subject: Re: [dfdl-wg] How to handle multi-dimensional arrays > > Hi Mike > > I will confess to having not given a great deal of thought to > multi-dim arrays in the past, but what you propose looks to be sound. > > Regards, Steve > > Steve Hanson > WebSphere Business Integration Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 --------------- > > We have come up with an approach to how to represent > multi-dimensional arrays within XSD-described XML. The > attached test file (.xml) and DFDL Schema (.dfdl.xsd) > illustrate the proposed solution. > > The proposal does not require any changes to XSD, XML or any > other special constructs outside of a single dfdl annotation > to specify the storage order of the representation. > > I'm pretty happy with how this works out. We can handle > arrays with different storage orders, like fortran style > column-major vs. more common row-major, and it dovetails > nicely with XPath expressions and the XSD data model. Schema > validation can really do something for you, like tell you if > you have all the elements of the array (if it's fixed size), > and that you don't have multiple elements occupying the same > array location. > > Those interested in multi-dimensional array support please > give this some consideration. > > That said, I'm departing on vacation for a week, so I'll toss > this out there for people to look at, but I won't be able to > interact with you all on it until I get back. > > ...mikeb > > [attachment "test2dArray.xml" deleted by Steve > Hanson/UK/IBM] [attachment "test2dArray.dfdl.xsd" deleted by > Steve Hanson/UK/IBM] >

1 0

RE: [dfdl-wg] alternate syntax for DFDL annotations
by Mike Beckerle 03 Mar '05

03 Mar '05

Sorry for missing our call today, and thanks to everyone for the rapid feedback on the non-native attributes approach. I think there's a consensus here that non-native attributes are going too far, and that if we can just make the appinfo annotations a bit more dense we'll be happy. So , for now, we're not going to do non-native attributes in our DFDL prototype, I'll make a note about the possiblility in an appendix of the doc though. None of the tools I have will validate them, and that's a big drawback. Right now, our prototype handles small examples with annotations that look like this: <xs:element name="foo" type="xs:int"> <xs:annotation> <xs:appinfo source="http://dataformat.org/"> <dfdl:dataFormat byteOrder="bigEndian" charset="UTF-8">  <dfdl:separator>,</dfdl:separator>  </dfdl:dataFormat> </xs:appinfo> </xs:annotation> </xs:element> Note that if you use appinfo annotations then you don't need namespace qualification on the attribute names. We also allow for "extension" declarations in that the dfdl:dataFormat element is the head of a XSD substitution group, So you can also define an extension like ascl:binaryIntegerFormat below: <xs:element name="foo" type="xs:int"> <xs:annotation> <xs:appinfo source="http://dataformat.org/"> <ascl:binaryIntegerFormat byteOrder="bigEndian"/> </xs:appinfo> </xs:annotation> </xs:element> The ascl:binaryIntegerFormat is a narrowing. It doesn't allow any properties to be specified that don't make sense for a binary integer. It's in a different namespace to illustrate that these could be defined by a DFDL user in their DFDL schema, not just by the dfdl implementation itself. The basic dfdl:dataFormat element allows any and all representation properties. Whether we supply standard library elements which are narrowings of it, or let those be defined by end-users is something we can discuss. This all comes for free by just using XSD properly. It adds almost nothing to the implementation. ...mikeb > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of Steve Hanson > Sent: Wednesday, March 02, 2005 1:08 PM > To: Myers, James D > Cc: dfdl-wg(a)gridforum.org; owner-dfdl-wg(a)ggf.org > Subject: RE: [dfdl-wg] alternate syntax for DFDL annotations > > > > > > I can't say I am over keen on this, because > - you will end up with odd rules where some properties can be > attributes and some can't, which will confuse users. > - you have to invent errors and/or overide behaviour when you > get a clash between attribute and annotation > - (minor) it makes the job of any software that instantiates > a dfdl memory model harder > > If we do adopt this then #2 is preferable. > > Regards, Steve > > Steve Hanson > WebSphere Business Integration Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 > > > > > "Myers, James D" > > <jim.myers(a)pnl.go > > v> > To > Sent by: dfdl-wg(a)gridforum.org > > owner-dfdl-wg@ggf > cc > .org > > > Subject > RE: [dfdl-wg] > alternate syntax for > 02/03/2005 17:35 DFDL annotations > > > > > > > > > > > > > > > > > > > I agree - I think I like #2. I know at one point in the past > we had complaints when we tried to use attributes with > namespaces - it may just have been that parsers weren't > dealing with them correctly early on. > Whatever it was, option 2 would provide a work around as well > as helping us keep a single 'canonical' form for DFDL. > > Jim > > > -----Original Message----- > > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] > On Behalf > > Of Martin Westhead > > Sent: Wednesday, March 02, 2005 12:16 PM > > To: Mike Beckerle > > Cc: dfdl-wg(a)gridforum.org > > Subject: Re: [dfdl-wg] alternate syntax for DFDL annotations > > > > > > Hi Mike, > > > > This is interesting I wasn't aware you could do this. My > concern would > > be if it turned out the schema tools couldn't handle it but I tried > > XML Spy which seems to just ignore the extra annotations. > > > > So I'd be happy to use them. As you point out we will need > both these > > and annotations so we are left with a choice. Either we: > > > > 1. Specify some (most) dfdl additions in this non-native > attribute > > form and use annotations when required or > > > > 2. We specify everything as annotations but allow this form as a > > systematic short hand. > > > > I think I prefer (2). The sort of thing I have in mind is > making the > > annotation syntax look like: > > > > <xs:element name="foo" type="xs:string"> > > <annotation> > > <appinfo> > > <dfdl:attributes dfdl:repType="text" > > dfdl:charset="UTF-8" > > dfdl:repLength="10"/> > > </appinfo> > > <annotation > > </xs:element> > > > > And then this be directly equivalent (and trivially transformed > > into/from) what you had before: > > > > <xs:element name="foo" > > type="xs:string" > > dfdl:repType="text" > > dfdl:charset="UTF-8" > > dfdl:repLength="10"/> > > > > I'm not sure if it matters that much but it seems like it might be > > neater if everything could be represented in the same form. > > > > Also there may be times when you would like to use the > annotation form > > because you have a bock of stuff that could be reused > multiple times. > > Once your annotations have become attributes in the main > schema they > > are more intimately tied to it. > > > > Cheers, > > > > Martin > > > > > > > > Mike Beckerle wrote: > > > Up til now we've considered DFDL annotations only as inside the > > > appinfo context. However, we should consider whether we should use > > non-native > > > attributes as well or > > > > > > as an alternative: E.g., simple DFDL rep properties could also be > > > expressed like this: > > > > > > <xs:element name="foo" > > > > > > type="xs:string" > > > > > > dfdl:repType="text" > > > > > > dfdl:charset="UTF-8" > > > > > > dfdl:repLength="10"/> > > > > > > > > > > > > This has the advantages of compactness, and is a fully > > supported way > > > of > > > extending XML Schema. That is, using non-native attributes is a > > > supported extension idiom. This won't handle things that > > really need the > > > syntactic support of element structure to express their > complexity, > > > e.g., things like specifying text delimiters with quoting and > > > escape-sequence specifications. For those we'll still need > > to open an > > > appinfo annotation up. However, for basic things like > > byteOrder and such > > > it is far more attractive syntactically to use non-native > > attributes > > > than appinfo annotations. > > > > > > > > > Comments? > > > > > > ...mikeb > > > > > > > > > >

1 0

RE: [dfdl-wg] Conference Call Today - notes
by Myers, James D 03 Mar '05

03 Mar '05

Sorry for the late FYI on this: If we have a meeting in Toronto (anywhere outside the US), PNNL staff will need 45-60 days to get clearance for the foreign travel. That would mean making decisions about dates and specific location as soon as possible for us to be involved. (I would hope to have at least one of us and probably two of us there.) Jim > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of mike.beckerle(a)ascentialsoftware.com > Sent: Wednesday, February 16, 2005 12:45 PM > To: dfdl-wg(a)gridforum.org > Subject: [dfdl-wg] Conference Call Today - notes > > > > DFDL-WG Conference call 2005-02-16 > > Attendees: Suman Kalia, Steve Hanson, Susan Malaika - IBM, > Tara Talbot - PNNL, Mike Beckerle - Ascential Software > > IBM offered to host a F2F meeting in mid to late April, or > early May at their Toronto facility. Seems like a good time > frame. Gives Ascential folks time time to get prototype "in > play", and leaves time before the June GGF > (Chicago) to get some things written up. > > MikeB and Suman to schedule time to work on combined model > and list of representation properties: week of Feb 28th. > > Ascential prototype: hoping to have simplest examples working > by end of this week. (No promises). Things like 2 binary > integers. MikeB to post current XSD for dfdl properties and a > few of our test files for the prototype. > > Susan asked about whether implementations of things like BFD > are available. Tara to investigate whether BFD is publicly > available now. > > We discussed when to start recruiting Oracle, Microsoft, and > other participants, since we will need them. We agreed that > our hope would be the June GGF though it really depends on > having an initial document representing some sort of > converged consensus of the current team on current directions. > > > Mike Beckerle > Ascential > >

2 1

Fw: [dfdl-wg] alternate syntax for DFDL annotations
by Suman Kalia 03 Mar '05

03 Mar '05

Hi Mike -- We will be creating XML schema to model our annotations and the attributes applicable to the representation of a particular element/type will be grouped within the complex type of the annotation schema. My preference will be use annotation approach consistently in DFDL. Moreover the attributes defined in the compact form are not easily accessible from the XML Schema APIs, you have to look into the DOMNode associated with the XSDElementDeclaration to get access to such attributes; it makes the programming task a little harder. Having said that SOAP encoded schemas use this compact form to identify the type and dimensions of the array types; some folks like this approach and some hate it.. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 03/02/2005 12:38 PM ----- Mike Beckerle <mike.beckerle(a)ascentialsoftware.com> Sent by: owner-dfdl-wg(a)ggf.org 03/01/2005 03:53 PM To dfdl-wg(a)gridforum.org cc Subject [dfdl-wg] alternate syntax for DFDL annotations Up til now we've considered DFDL annotations only as inside the appinfo context. However, we should consider whether we should use non-native attributes as well or as an alternative: E.g., simple DFDL rep properties could also be expressed like this: <xs:element name="foo" type="xs:string" dfdl:repType="text" dfdl:charset="UTF-8" dfdl:repLength="10"/> This has the advantages of compactness, and is a fully supported way of extending XML Schema. That is, using non-native attributes is a supported extension idiom. This won't handle things that really need the syntactic support of element structure to express their complexity, e.g., things like specifying text delimiters with quoting and escape-sequence specifications. For those we'll still need to open an appinfo annotation up. However, for basic things like byteOrder and such it is far more attractive syntactically to use non-native attributes than appinfo annotations. Comments? ...mikeb

1 0

RE: [dfdl-wg] alternate syntax for DFDL annotations
by Myers, James D 03 Mar '05

03 Mar '05

I agree - I think I like #2. I know at one point in the past we had complaints when we tried to use attributes with namespaces - it may just have been that parsers weren't dealing with them correctly early on. Whatever it was, option 2 would provide a work around as well as helping us keep a single 'canonical' form for DFDL. Jim > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of Martin Westhead > Sent: Wednesday, March 02, 2005 12:16 PM > To: Mike Beckerle > Cc: dfdl-wg(a)gridforum.org > Subject: Re: [dfdl-wg] alternate syntax for DFDL annotations > > > Hi Mike, > > This is interesting I wasn't aware you could do this. My > concern would > be if it turned out the schema tools couldn't handle it but I > tried XML > Spy which seems to just ignore the extra annotations. > > So I'd be happy to use them. As you point out we will need both these > and annotations so we are left with a choice. Either we: > > 1. Specify some (most) dfdl additions in this non-native attribute > form and use annotations when required or > > 2. We specify everything as annotations but allow this form as a > systematic short hand. > > I think I prefer (2). The sort of thing I have in mind is making the > annotation syntax look like: > > <xs:element name="foo" type="xs:string"> > <annotation> > <appinfo> > <dfdl:attributes dfdl:repType="text" > dfdl:charset="UTF-8" > dfdl:repLength="10"/> > </appinfo> > <annotation > </xs:element> > > And then this be directly equivalent (and trivially transformed > into/from) what you had before: > > <xs:element name="foo" > type="xs:string" > dfdl:repType="text" > dfdl:charset="UTF-8" > dfdl:repLength="10"/> > > I'm not sure if it matters that much but it seems like it might be > neater if everything could be represented in the same form. > > Also there may be times when you would like to use the > annotation form > because you have a bock of stuff that could be reused multiple times. > Once your annotations have become attributes in the main > schema they are > more intimately tied to it. > > Cheers, > > Martin > > > > Mike Beckerle wrote: > > Up til now we've considered DFDL annotations only as inside the > > appinfo > > context. However, we should consider whether we should use > non-native > > attributes as well or > > > > as an alternative: E.g., simple DFDL rep properties could also be > > expressed like this: > > > > <xs:element name="foo" > > > > type="xs:string" > > > > dfdl:repType="text" > > > > dfdl:charset="UTF-8" > > > > dfdl:repLength="10"/> > > > > > > > > This has the advantages of compactness, and is a fully > supported way > > of > > extending XML Schema. That is, using non-native attributes is a > > supported extension idiom. This won't handle things that > really need the > > syntactic support of element structure to express their complexity, > > e.g., things like specifying text delimiters with quoting and > > escape-sequence specifications. For those we'll still need > to open an > > appinfo annotation up. However, for basic things like > byteOrder and such > > it is far more attractive syntactically to use non-native > attributes > > than appinfo annotations. > > > > > > Comments? > > > > ...mikeb > > > >

2 1

DFDL call notes
by Myers, James D 03 Mar '05

03 Mar '05

FYI - quick notes about a short call today - Steve, Tara, and I hung out on the DFDL call today for about 20 minutes to see if we'd get a quorum. We didn't. We noted that the availability of the prototype from Ascential seems to be a pre-requisite for further discussions about the reader/writer/filter/function ideas and resolving the issues around transform versus sub-type centric approaches. (I'm glad its not me in the critical path this time!) In any case, I think we agreed that we need the prototype, or some description of it, soon to get the discussion moving again and make another productive f-2-f possible. Jim

1 0

alternate syntax for DFDL annotations
by Mike Beckerle 03 Mar '05

03 Mar '05

Up til now we've considered DFDL annotations only as inside the appinfo context. However, we should consider whether we should use non-native attributes as well or as an alternative: E.g., simple DFDL rep properties could also be expressed like this: <xs:element name="foo" type="xs:string" dfdl:repType="text" dfdl:charset="UTF-8" dfdl:repLength="10"/> This has the advantages of compactness, and is a fully supported way of extending XML Schema. That is, using non-native attributes is a supported extension idiom. This won't handle things that really need the syntactic support of element structure to express their complexity, e.g., things like specifying text delimiters with quoting and escape-sequence specifications. For those we'll still need to open an appinfo annotation up. However, for basic things like byteOrder and such it is far more attractive syntactically to use non-native attributes than appinfo annotations. Comments? ...mikeb

2 1

How to handle multi-dimensional arrays
by mike.beckerle＠ascentialsoftware.com 02 Mar '05

02 Mar '05

We have come up with an approach to how to represent multi-dimensional arrays within XSD-described XML. The attached test file (.xml) and DFDL Schema (.dfdl.xsd) illustrate the proposed solution. The proposal does not require any changes to XSD, XML or any other special constructs outside of a single dfdl annotation to specify the storage order of the representation. I'm pretty happy with how this works out. We can handle arrays with different storage orders, like fortran style column-major vs. more common row-major, and it dovetails nicely with XPath expressions and the XSD data model. Schema validation can really do something for you, like tell you if you have all the elements of the array (if it's fixed size), and that you don't have multiple elements occupying the same array location. Those interested in multi-dimensional array support please give this some consideration. That said, I'm departing on vacation for a week, so I'll toss this out there for people to look at, but I won't be able to interact with you all on it until I get back. ...mikeb

2 1

BFD Parser Engine
by Talbott, Tara D 23 Feb '05

23 Feb '05

During the last meeting Susan inquired about the implementation of the BFD parser. Here is a link to the BFD homepage which describes the project and specifications, http://collaboratory.emsl.pnl.gov/sam/bfd/. The parser engine is available from that page, but we are still investigating the status of the open source license so the source is not yet available, I will notifiy you when it is. Thanks, Tara

1 0

RE: [dfdl-wg] DFDL Schema and test case format for forthcoming pr ototype
by Mike Beckerle 17 Feb '05

17 Feb '05

>From the one test case I included you can't see this, but the <data kind=...> element allows whitespace insertion and ignores all whitespace. E.g., <data kind="bits"> 0 1000000 10000000 00000000 00000000 0 1000001 01010100 00000000 00000000 0 0111111 11001100 11001100 11001100 </data> Is accepted as is: <data kind="hex"> 0095 35 0000000000000048 222038109f e2e4d5d940d3c5c140c9d540 000f 0095 35 0000000951500501 088322722f d2e4d9e3e94b4bf0d9f64b4b 002f </data> We did this since most hex dump tools will space things out a bit, and the spaces help clarity. There's also a <data kind="text"> variant which is different since it is whitespace preserving. ...mikeb > -----Original Message----- > From: mike.beckerle(a)ascentialsoftware.com > [mailto:mike.beckerle@ascentialsoftware.com] > Sent: Wednesday, February 16, 2005 2:06 PM > To: dfdl-wg(a)gridforum.org > Subject: [dfdl-wg] DFDL Schema and test case format for > forthcoming prototype > > > I posted schemas and a single test case file up at grid forge > as a way of letting people know a little more about what > we're up to at Ascential. > Here's the zip file of them: > > http://forge.gridforum.org/projects/dfdl-wg/document/ascl-info > -impl-2005-02- > 16.zip/en/1. > > It's a grid forge thing that this will download as a > ".zip.unknown" file, requiring you to rename it. > > We've written a few more test files than are included here, > covering Jim Myers's examples from last summer, and then > some, and they validate as XML under a few different > validators, but nothing is running yet to process them. The > enclosed example in the zip file is the one we're hoping to > make work this week. > > So the simple way to understand our prototype is that it runs > tests like these, and is otherwise intended to be easy to > change as the specs evolve, and is focused on simplicity and > clarity, not speed. > > Our goal with these testCase files is that you can get a hex > dump of some interesting data, paste it into a test case > document like this, and thereby easily create self-contained tests. > > Here's the README.txt contents: > > This directory contains test files in the xml/tests > subdirectory, and common schemas in the xsd subdirectory. > > xml/tests/testSimpleBinaryIntegers1.dfdl.xsd - DFDL annotated > XML schema for very simple data. 2 binary integers > xml/tests/testSimpleBinaryIntegers1.xml - test case file. > References the schema, includes hexified binary data, and > result data for comparison. > > xsd/dfdl.xsd and .html - the schema for the DFDL annotations > xsd/testCase.xsd and .html - the schema for the test files > > The html documentation is generated by a tool we use called > Oxygen XML. > > XMLSchema.xsd - These four are used in validating DFDL > annotated schemas to insure they are valid XSD. > XMLSchema.dtd - These are all available from the web, but are > included here for speed reasons. > datatypes.xsd > datatypes.dtd >

1 0