February 2005 - dfdl-wg - lists.ogf.org

How to handle multi-dimensional arrays
by mike.beckerle＠ascentialsoftware.com 02 Mar '05

02 Mar '05

We have come up with an approach to how to represent multi-dimensional arrays within XSD-described XML. The attached test file (.xml) and DFDL Schema (.dfdl.xsd) illustrate the proposed solution. The proposal does not require any changes to XSD, XML or any other special constructs outside of a single dfdl annotation to specify the storage order of the representation. I'm pretty happy with how this works out. We can handle arrays with different storage orders, like fortran style column-major vs. more common row-major, and it dovetails nicely with XPath expressions and the XSD data model. Schema validation can really do something for you, like tell you if you have all the elements of the array (if it's fixed size), and that you don't have multiple elements occupying the same array location. Those interested in multi-dimensional array support please give this some consideration. That said, I'm departing on vacation for a week, so I'll toss this out there for people to look at, but I won't be able to interact with you all on it until I get back. ...mikeb

2 1

BFD Parser Engine
by Talbott, Tara D 22 Feb '05

22 Feb '05

During the last meeting Susan inquired about the implementation of the BFD parser. Here is a link to the BFD homepage which describes the project and specifications, http://collaboratory.emsl.pnl.gov/sam/bfd/. The parser engine is available from that page, but we are still investigating the status of the open source license so the source is not yet available, I will notifiy you when it is. Thanks, Tara

1 0

RE: [dfdl-wg] DFDL Schema and test case format for forthcoming pr ototype
by Mike Beckerle 16 Feb '05

16 Feb '05

>From the one test case I included you can't see this, but the <data kind=...> element allows whitespace insertion and ignores all whitespace. E.g., <data kind="bits"> 0 1000000 10000000 00000000 00000000 0 1000001 01010100 00000000 00000000 0 0111111 11001100 11001100 11001100 </data> Is accepted as is: <data kind="hex"> 0095 35 0000000000000048 222038109f e2e4d5d940d3c5c140c9d540 000f 0095 35 0000000951500501 088322722f d2e4d9e3e94b4bf0d9f64b4b 002f </data> We did this since most hex dump tools will space things out a bit, and the spaces help clarity. There's also a <data kind="text"> variant which is different since it is whitespace preserving. ...mikeb > -----Original Message----- > From: mike.beckerle(a)ascentialsoftware.com > [mailto:mike.beckerle@ascentialsoftware.com] > Sent: Wednesday, February 16, 2005 2:06 PM > To: dfdl-wg(a)gridforum.org > Subject: [dfdl-wg] DFDL Schema and test case format for > forthcoming prototype > > > I posted schemas and a single test case file up at grid forge > as a way of letting people know a little more about what > we're up to at Ascential. > Here's the zip file of them: > > http://forge.gridforum.org/projects/dfdl-wg/document/ascl-info > -impl-2005-02- > 16.zip/en/1. > > It's a grid forge thing that this will download as a > ".zip.unknown" file, requiring you to rename it. > > We've written a few more test files than are included here, > covering Jim Myers's examples from last summer, and then > some, and they validate as XML under a few different > validators, but nothing is running yet to process them. The > enclosed example in the zip file is the one we're hoping to > make work this week. > > So the simple way to understand our prototype is that it runs > tests like these, and is otherwise intended to be easy to > change as the specs evolve, and is focused on simplicity and > clarity, not speed. > > Our goal with these testCase files is that you can get a hex > dump of some interesting data, paste it into a test case > document like this, and thereby easily create self-contained tests. > > Here's the README.txt contents: > > This directory contains test files in the xml/tests > subdirectory, and common schemas in the xsd subdirectory. > > xml/tests/testSimpleBinaryIntegers1.dfdl.xsd - DFDL annotated > XML schema for very simple data. 2 binary integers > xml/tests/testSimpleBinaryIntegers1.xml - test case file. > References the schema, includes hexified binary data, and > result data for comparison. > > xsd/dfdl.xsd and .html - the schema for the DFDL annotations > xsd/testCase.xsd and .html - the schema for the test files > > The html documentation is generated by a tool we use called > Oxygen XML. > > XMLSchema.xsd - These four are used in validating DFDL > annotated schemas to insure they are valid XSD. > XMLSchema.dtd - These are all available from the web, but are > included here for speed reasons. > datatypes.xsd > datatypes.dtd >

1 0

DFDL Schema and test case format for forthcoming prototype
by mike.beckerle＠ascentialsoftware.com 16 Feb '05

16 Feb '05

I posted schemas and a single test case file up at grid forge as a way of letting people know a little more about what we're up to at Ascential. Here's the zip file of them: http://forge.gridforum.org/projects/dfdl-wg/document/ascl-info-impl-2005-02- 16.zip/en/1. It's a grid forge thing that this will download as a ".zip.unknown" file, requiring you to rename it. We've written a few more test files than are included here, covering Jim Myers's examples from last summer, and then some, and they validate as XML under a few different validators, but nothing is running yet to process them. The enclosed example in the zip file is the one we're hoping to make work this week. So the simple way to understand our prototype is that it runs tests like these, and is otherwise intended to be easy to change as the specs evolve, and is focused on simplicity and clarity, not speed. Our goal with these testCase files is that you can get a hex dump of some interesting data, paste it into a test case document like this, and thereby easily create self-contained tests. Here's the README.txt contents: This directory contains test files in the xml/tests subdirectory, and common schemas in the xsd subdirectory. xml/tests/testSimpleBinaryIntegers1.dfdl.xsd - DFDL annotated XML schema for very simple data. 2 binary integers xml/tests/testSimpleBinaryIntegers1.xml - test case file. References the schema, includes hexified binary data, and result data for comparison. xsd/dfdl.xsd and .html - the schema for the DFDL annotations xsd/testCase.xsd and .html - the schema for the test files The html documentation is generated by a tool we use called Oxygen XML. XMLSchema.xsd - These four are used in validating DFDL annotated schemas to insure they are valid XSD. XMLSchema.dtd - These are all available from the web, but are included here for speed reasons. datatypes.xsd datatypes.dtd

1 0

Reposting of minutes from Dec 2004 F2F
by Mike Beckerle 16 Feb '05

16 Feb '05

Susan said she couldn't find these minutes on gridforge. When I doublechecked, I couldn't find them either so I reposted them. This URL works for me. http://forge.gridforum.org/projects/dfdl-wg/document/ggf-dfdl-wg-2004-12-06- meeting-summary002_/en/1.

1 0

Conference Call Today - notes
by mike.beckerle＠ascentialsoftware.com 16 Feb '05

16 Feb '05

DFDL-WG Conference call 2005-02-16 Attendees: Suman Kalia, Steve Hanson, Susan Malaika - IBM, Tara Talbot - PNNL, Mike Beckerle - Ascential Software IBM offered to host a F2F meeting in mid to late April, or early May at their Toronto facility. Seems like a good time frame. Gives Ascential folks time time to get prototype "in play", and leaves time before the June GGF (Chicago) to get some things written up. MikeB and Suman to schedule time to work on combined model and list of representation properties: week of Feb 28th. Ascential prototype: hoping to have simplest examples working by end of this week. (No promises). Things like 2 binary integers. MikeB to post current XSD for dfdl properties and a few of our test files for the prototype. Susan asked about whether implementations of things like BFD are available. Tara to investigate whether BFD is publicly available now. We discussed when to start recruiting Oracle, Microsoft, and other participants, since we will need them. We agreed that our hope would be the June GGF though it really depends on having an initial document representing some sort of converged consensus of the current team on current directions. Mike Beckerle Ascential

1 0

Conference Call Today
by mike.beckerle＠ascentialsoftware.com 16 Feb '05

16 Feb '05

Our semi-weekly conference call is today at Noon EST, 9PST, 17Z. If people get this in advance can you please send whether you can attend. I can share status of Ascential's forthcoming "Informational Implementation". ...mikeb

2 1

RE: [dfdl-wg] GGF13
by Mike Beckerle 07 Feb '05

07 Feb '05

>From the conference calls and the F2F in december, it seems nobody but me was planning for Seoul, and I've recently cancelled my plans for Seoul. I think a F2F in late march or April could be productive. I hope to have a working prototype distributed by then, so we'll be able to really dig into resolving issues. ...mikeb > -----Original Message----- > From: Steve Hanson [mailto:smh@uk.ibm.com] > Sent: Monday, February 07, 2005 9:51 AM > To: dfdl-wg(a)gridforum.org > Subject: [dfdl-wg] GGF13 > > > > > > Are folk in the DFDL WG planning on attending GGF 13 in Seoul? > > Or would it be more productive to have a separate DFDL WG > face-to-face in the US? > > Regards, Steve > > Steve Hanson > WebSphere Business Integration Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 >

1 0

GGF13
by Steve Hanson 07 Feb '05

07 Feb '05

Are folk in the DFDL WG planning on attending GGF 13 in Seoul? Or would it be more productive to have a separate DFDL WG face-to-face in the US? Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

RE: [dfdl-wg] Open source DFDL implementation?
by mike.beckerle＠ascentialsoftware.com 02 Feb '05

02 Feb '05

Our implementation will be open source. Ascential open source is a "dual use" kind of license. Basically means Ascential can do what it wants with the software including offer commercial terms, for everyone else it's like the GPL with the additional requirement that any mods you make to the software are relicensed back to Ascential. There are other licenses like this e.g., mySql and cygwin, I'm told there are others as well. This particular implementation is not intended to be production quality so I'm not sure the commercial terms matter in the above license. This implementation is being designed from the ground up to be simple, clear, and easily modified so that as we iron out the exact semantics of DFDL we can do so with a real implementation. These goals are in tension with performance, which we're punting on entirely. So, for example, almost all the Java object classes in this implementation are immutable objects. This makes the code dead-simple, but implies lots of object allocation and garbage collection during execution. We expect several objects to be allocated and discarded for every bit of input that is touched when reading data, for example. The GGF doesn't do "reference implementations", so I'm calling this thing an "Informational Implementation", but the spirit is the same. Our DFDL spec documents can be far clearer if we have an implementation that is easy and quick to modify, and one we can run real tests against readily as we debate the fine points.Our goal at Ascential is for this implementation to exactly match the standard. Other minor details: We're writing it in pure Java in the latest Java dialect, Java 1.5, making heavy use of generic types to simplify the code. ...mikeb _____ From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of Susan Malaika Sent: Wednesday, February 02, 2005 3:45 AM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Open source DFDL implementation? People have mentioned various implementations of DFDL on this mailing list. Will any of them be open source? Susan Malaika

1 0