- dfdl-wg - lists.ogf.org

Fw: [dfdl-wg] Discussion forums - try them out
by Steve Hanson 01 Dec '04

01 Dec '04

Mike - working now. It looks a very useful facility. It might be worth posting some of our existing discussion threads up there. I'll see if I get time to do that tomorrow. I added one append on the F2F. Suman, Susan - suggest you add yourselves as DFDL group members via the web page http://forge.gridforum.org/projects/dfdl-wg/ - I think that was what gave me the correct access in the end. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 01/12/2004 16:16 ----- Steve Hanson/UK/IBM To 01/12/2004 10:07 mike.beckerle(a)ascentialsoftware.com cc Subject Fw: [dfdl-wg] Discussion forums - try them out Mike - I get this error when I follow the link Error Permission denied. Exiting with Error Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 01/12/2004 10:06 ----- mike.beckerle@asc entialsoftware.co m To Sent by: dfdl-wg(a)gridforum.org owner-dfdl-wg@ggf cc .org Subject [dfdl-wg] Discussion forums - try 01/12/2004 00:42 them out https://forge.gridforum.org/forum/forum.php?forum_id=455 We have discussion forums now. I added one about the agenda for our F2F. Please post some comments and we'll see how it works. ...mikeb

1 0

Discussion forums - try them out
by mike.beckerle＠ascentialsoftware.com 01 Dec '04

01 Dec '04

https://forge.gridforum.org/forum/forum.php?forum_id=455 We have discussion forums now. I added one about the agenda for our F2F. Please post some comments and we'll see how it works. ...mikeb

1 0

RE: [SPAM] RE: [dfdl-wg] BSDF
by mike.beckerle＠ascentialsoftware.com 24 Nov '04

24 Nov '04

Good find Jim, One thing we can look at from these guys is their handling of bit-packed fields. I've kind of been hand-waving the whole bit-level thing recently as less important than "hard issues" of compositionality and layering, but we do have to face this issue eventually. Other than that, from my skim I don't see how we don't end up just subsuming what they do. I expect there's quite a few efforts out there to add some sort of dense-packed format description capability to XML. Until DFDL really "comes out" I think we'll see more and more of this. In some sense this bodes well for the eventual success of DFDL. In every case I suggest we reach out to them to be sure they are aware of the DFDL effort, and then they can decide to contribute or not, but at least they won't be working in a vaccum. At some point we'll badly need poeple to review specs and consider if DFDL meets their needs. ...mikeb _____ From: Myers, James D [mailto:jim.myers@pnl.gov] Sent: Tuesday, November 23, 2004 10:39 AM To: dfdl-wg(a)gridforum.org Subject: [SPAM] RE: [dfdl-wg] BSDF I ran across another community working in the same space as DFDL: BSDL as part of MPEG21 (someone at the NARA conference mentioned it). From a quick look at what they have, it appears to be inverse DFDL with plans to make DFDL next, i.e. they go from model to output binary format, but not vice versa. Similar schema approach, etc. with some comments on things in schema that they don't allow. I haven't found a spec doc yet, just a paper and some presentations you can google for. Jim <http://www2002.org/CDROM/alternate/334/> Bitstream Syntax Description Language: Application of XML-Schema ... ... Language (BSDL) that embraces such XML bitstream descriptions. ... 2/; [13] IEEE Standard for Binary Floating-Point ... Final Draft Amendment 4: Streaming video profile ... www2002.org/CDROM/alternate/334/

1 0

RE: [dfdl-wg] Transform examples
by mike.beckerle＠ascentialsoftware.com 24 Nov '04

24 Nov '04

To this set I would add: - Parse (or Parser, with the output-direction to be called unparse/unparser) > -----Original Message----- > From: Steve Hanson [mailto:smh@uk.ibm.com] > Sent: Wednesday, November 24, 2004 6:34 AM > To: dfdl-wg(a)gridforum.org > Subject: Fw: [dfdl-wg] Transform examples > > > > > > Here's a candidate list of words. > > - Transform > - Map > - Converter > - Conversion > - Translation > - Render > - Representation > - Formatter > - Cast > > We could also consider prefixing a word with 'Physical', eg, > 'PhysicalRender' > > Regards, Steve > > > ----- Forwarded by Steve Hanson/UK/IBM on 24/11/2004 11:08 ----- > > > Martin Westhead > > <martinwesthead@y > > ahoo.co.uk> > To > Sent by: "Chappell, Alan R" > > owner-dfdl-wg@ggf > <chappella(a)BATTELLE.ORG>, > .org dfdl-wg(a)gridforum.org > > > cc > > > 22/11/2004 17:31 > Subject > Re: [dfdl-wg] > Transform examples > > > > > > > > > > > > > > > > > Hi Alan, > > Thanks for this. Some thoughts/questions/comments that > occurred to me looking at your example. > > First a general point: > > - On one of the calls which I think was just me and the > IBMers they pointed out quite strongly that the ther > "transformation" has other connotations that make it very > confusing here, Mike also agrees. I think we need to choose a > different name to avoid such confusion. Mike suggested > "converters"/"convertions" which is suitably general. I also > thought about "casting" but I think that's perhaps too > specific. Either way it would be nice to choose something or > our discussions will be rather difficult to follow. > > (I'll continue to use transform for this email...) > > On the example: > - I was expecting the definition of the input and output > types to the transform should have equal status. Here it > seems that the input is specified implicitly in the parameter > set it will accept. > > - Why does asciiToInt read byte[]? I would expect it to > read at least char[] (or probably xs:string, since there is > no xs:char). The intent would be that you could then compose > it with a character converter of your choice. > > - In general I think the typing and compositional > properties of the transformations are the nut that requires > cracking here. My proposal is that the types should be > specified in XML data model at all levels so instead of > byte[] you would have: > > <xs:sequence> > <xs:element name="anon" type="xs:byte" minOccurs="0" > maxOccurs="unbounded"/> > </xs:sequence> > > that's all for now... > > Cheers, > > Martin > > > > > > > > > Chappell, Alan R wrote: > > Third try... No zip, just the 3 files important to the simple > > transform example.... > > > > > ---------------------------------------------------------------------- > > -- > > *From:* Chappell, Alan R > > *Sent:* Friday, November 19, 2004 1:39 PM > > *To:* dfdl-wg(a)gridforum.org > > *Subject:* *MJ-REJECTED* Transform examples > > > > Second try on sending these examples. I've cut the set down > to the 3 > > important files so hopefully it will get through this time. > > > > > ---------------------------------------------------------------------- > > -- > > *From:* Chappell, Alan R > > *Sent:* Thursday, November 18, 2004 8:47 AM > > *To:* dfdl-wg(a)gridforum.org > > *Subject:* *MJ-REJECTED* Transform examples > > > > Here is the example I mentioned yesterday. Look particularly at > > dfdltransforms.xsd, BasicAsciiIntExp.xsd, and > BasicBinIntExp.xsd. Note > > the "Exp" on those last two files indicate that they are > expansions of > > the information in the original versions of those files. > These make a > > first stab at giving a fully verbose description of the > structure and > > the transforms, i.e., it's working towards the canonical > > representation Martin talked about yesterday. The "dfdltransforms" > > gives the definitions of transforms and their components. > > > > There are lots of things that can be improved here. > > <<dfdl-examples.zip>> > > Alan R. Chappell > > chappella(a)battelle.org > > > > Pacific Northwest National Laboratory > > Battelle Seattle Research Center > > (206) 528-3228 > > > > >

1 0

Problems with access to Grid Forge Docs
by mike.beckerle＠ascentialsoftware.com 24 Nov '04

24 Nov '04

Steve reported some difficulty getting at my recently posted grid-forge docs. I've reposted these. Turns out that you can't delete a document then post another document of the same name. https://forge.gridforum.org/docman2/ViewCategory.php?group_id=113&category_i d=803&language_id=1&sort0=ddate&dir0=1 Gets you the latest list of docs sorted in decreasing chronological order. I hate portals like Grid Forge. I've never seen a good one. A simple file server or FTP site would work better. ...mikeb

1 0

Fw: One more time....RE: Updated - RE: [dfdl-wg] DFDL Properties
by Steve Hanson 24 Nov '04

24 Nov '04

Mike, I am getting this when I try to access the document using the latest URL ?? DCMN-000065: Version 1 of document ggf-dfdl-rep-properties-proposal-full-set-002.pdf has been deleted. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 24/11/2004 13:11 ----- mike.beckerle@asc entialsoftware.co m To Sent by: dfdl-wg(a)gridforum.org owner-dfdl-wg@gri cc dforum.org Subject One more time....RE: Updated - RE: 17/11/2004 20:58 [dfdl-wg] DFDL Properties Ok, that URL was stale, I had to update the doc one tweek (update TOC), so here's the right URL: https://forge.gridforum.org/docman2/ViewProperties.php?group_id=113&categor… From: mike.beckerle(a)ascentialsoftware.com [mailto:mike.beckerle@ascentialsoftware.com] Sent: Wednesday, November 17, 2004 3:50 PM To: dfdl-wg(a)gridforum.org Subject: Updated - RE: [dfdl-wg] DFDL Properties Importance: High Ignore what I just sent. Here's an updated full-set of rep properties document that I just uploaded.... https://forge.gridforum.org/docman2/ViewProperties.php?group_id=113&categor… The previous "basic set" document is subsumed by this, along with all the rep properties in the other document I mentioned. ...mikeb From: mike.beckerle(a)ascentialsoftware.com [mailto:mike.beckerle@ascentialsoftware.com] Sent: Wednesday, November 17, 2004 2:47 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] DFDL Properties Suman, Steve, (and everyone) This document on our gridforge site, is an earlier formulation of DFDL before we all rallied around the XSD with annotations concept, but in section 5.3 and subsequent sections of the document it discusses the set of representation properties. https://forge.gridforum.org/docman2/ViewProperties.php?group_id=113&categor… I have since set out to split the discussion and description of these properties from the discussion of the overall DFDL proposal since I think the properties and the details of their semantics are quite orthogonal to many of the other details of DFDL. A first attempt at this also involved subsetting the set of properties to a basic and advanced set. The basic set excludes many of the things needed for Cobol. You'll find this document on gridforge about the "basic set" https://forge.gridforum.org/docman2/ViewProperties.php?group_id=113&categor… I've since thought better of this, and I have in preparation a document that includes all the basic and advanced properties. Splitting them up like this doesn't seem that helpful and creates the issue of "which document is the property I want described in.". I'll try to get the revised properties document posted so we have the complete set for your review. Reconciling this with the OMG rep model material is something we will need to do. Folks at Ascential have reviewed the set of properties here and believe there is no format they've seen that this can't describe including the complete X.12 EDI set, Swift, etc. etc. ...mikeb From: Chappell, Alan R [mailto:chappella@BATTELLE.ORG] Sent: Monday, November 15, 2004 8:55 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Interim call scheduled Turned out it was best to have our interim call at our usual time of the week so: DFDL Working Group conference call Wed. 11/17 9am PST To join the teleconference, please dial 1-800-664-0771, press 0, and ask for the "Daffodil meeting". International callers can use 1-509-376-7411 Alan R. Chappell chappella(a)battelle.org Pacific Northwest National Laboratory Battelle Seattle Research Center (206) 528-3228

1 0

Fw: [dfdl-wg] Transform examples
by Steve Hanson 24 Nov '04

24 Nov '04

Here's a candidate list of words. - Transform - Map - Converter - Conversion - Translation - Render - Representation - Formatter - Cast We could also consider prefixing a word with 'Physical', eg, 'PhysicalRender' Regards, Steve ----- Forwarded by Steve Hanson/UK/IBM on 24/11/2004 11:08 ----- Martin Westhead <martinwesthead@y ahoo.co.uk> To Sent by: "Chappell, Alan R" owner-dfdl-wg@ggf <chappella(a)BATTELLE.ORG>, .org dfdl-wg(a)gridforum.org cc 22/11/2004 17:31 Subject Re: [dfdl-wg] Transform examples Hi Alan, Thanks for this. Some thoughts/questions/comments that occurred to me looking at your example. First a general point: - On one of the calls which I think was just me and the IBMers they pointed out quite strongly that the ther "transformation" has other connotations that make it very confusing here, Mike also agrees. I think we need to choose a different name to avoid such confusion. Mike suggested "converters"/"convertions" which is suitably general. I also thought about "casting" but I think that's perhaps too specific. Either way it would be nice to choose something or our discussions will be rather difficult to follow. (I'll continue to use transform for this email...) On the example: - I was expecting the definition of the input and output types to the transform should have equal status. Here it seems that the input is specified implicitly in the parameter set it will accept. - Why does asciiToInt read byte[]? I would expect it to read at least char[] (or probably xs:string, since there is no xs:char). The intent would be that you could then compose it with a character converter of your choice. - In general I think the typing and compositional properties of the transformations are the nut that requires cracking here. My proposal is that the types should be specified in XML data model at all levels so instead of byte[] you would have: <xs:sequence> <xs:element name="anon" type="xs:byte" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> that's all for now... Cheers, Martin Chappell, Alan R wrote: > Third try... No zip, just the 3 files important to the simple transform > example.... > > ------------------------------------------------------------------------ > *From:* Chappell, Alan R > *Sent:* Friday, November 19, 2004 1:39 PM > *To:* dfdl-wg(a)gridforum.org > *Subject:* *MJ-REJECTED* Transform examples > > Second try on sending these examples. I've cut the set down to the 3 > important files so hopefully it will get through this time. > > ------------------------------------------------------------------------ > *From:* Chappell, Alan R > *Sent:* Thursday, November 18, 2004 8:47 AM > *To:* dfdl-wg(a)gridforum.org > *Subject:* *MJ-REJECTED* Transform examples > > Here is the example I mentioned yesterday. Look particularly at > dfdltransforms.xsd, BasicAsciiIntExp.xsd, and BasicBinIntExp.xsd. Note > the "Exp" on those last two files indicate that they are expansions of > the information in the original versions of those files. These make a > first stab at giving a fully verbose description of the structure and > the transforms, i.e., it’s working towards the canonical representation > Martin talked about yesterday. The "dfdltransforms" gives the > definitions of transforms and their components. > > There are lots of things that can be improved here. > <<dfdl-examples.zip>> > Alan R. Chappell > chappella(a)battelle.org > > Pacific Northwest National Laboratory > Battelle Seattle Research Center > (206) 528-3228 > >

1 0

Fw: [dfdl-wg] Transform examples
by Steve Hanson 24 Nov '04

24 Nov '04

I will second Mike's points here. As discussed on one of our calls, DFDL 'transforms' are the equivalent to a set of IBM model properties. The key information needed to correctly and efficiently parse a string are, in our model: - Physical type (fixed length, length prefixed, null terminated, etc - same as Mike's length protocols) - Length count (if fixed length) - Length reference (name of another field in the message giving the length - alternative to length count) - Ccsid - Length units (handles the character set issue Mike noted below, in conjunction with the ccsid) - Justification - Padding character (these last two used because we will auto-strip/add padding chars on read/write) Regards, Steve ----- Forwarded by Steve Hanson/UK/IBM on 24/11/2004 10:52 ----- mike.beckerle@asc entialsoftware.co m To Sent by: jim.myers(a)pnl.gov, owner-dfdl-wg@ggf dfdl-wg(a)gridforum.org .org cc Subject 22/11/2004 17:26 RE: [dfdl-wg] Transform examples I'm just trying to avoid having DFDL's design preclude the creation of an efficient implementation. It's easy to do this if we're not careful. E.g., if we ignore length issues then I can guarantee we won't be able to meet our previously-stated goals of making efficient random access possible. I can speak from experience here in that I have a mature implementation of a system like DFDL, just not an industry standard one. In the implementation one of the most critical things is the ability to rapidly get to the ends of records, or fields, without having to process the intervening data. Much effort in the implementation is devoted to these kinds of length machinations, and we use a plug in architecture where each data type and all converters on them can be extended and plugged in anew. For example we didn't originally have any date/time or dateTime types, but added them afterwards. All types and converters obey a number of different protocols associated with how length works. I can't see any other way to do it. That said, we want the minimum complexity in DFDL at the interface to the transformations that enables efficient implementation. ...mikeb From: Myers, James D [mailto:jim.myers@pnl.gov] Sent: Monday, November 22, 2004 11:51 AM To: dfdl-wg(a)gridforum.org Subject: RE: [dfdl-wg] Transform examples Just a high level comment (not sure I'd split things up the same way you have but I won't comment on that now ...): I'm not averse to putting something in DFDL about the expectations about length of fields, but is this useful in practice versus simply coding this in the parser you build? The parser is really the thing that will use it to calculate offsets, so some methods related to getting offsets on the transform classes are really what's needed - is that made simpler if there's info in the DFDL transform description? (And can I safely ignore this info in a dumb reference implementation that can only calculate offsets by actually parsing and then counting?) Jim -----Original Message----- From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of mike.beckerle(a)ascentialsoftware.com Sent: Monday, November 22, 2004 11:01 AM To: Chappell, Alan R; dfdl-wg(a)gridforum.org Subject: RE: [dfdl-wg] Transform examples Alan, I looked at these examples. There's one thing I think you've overlooked in the way transforms are specified here. This is the fact that intFromBinary knows that it will pick exactly 4 bytes off the input stream, and could advertise that property to the DFDL "system" in some way, whereas intFromAscii might take anything from 1 to however many characters. E.g., it might be able to tolerate whitespace of any size, leading zeros, etc. So as a transform it needs to advertise that the length of data being consumed requires that you run the transform. Where I'm coming from is this. It is very important that a DFDL description of data enable processing the data efficiently. To me that means that if data is all fixed width, then one should be able to randomly access fields in the data in constant time. Even if the data is variable width, one should be able to efficiently skip through it to find the boundaries without necessarily having to process all the data, convert to common format, etc. To achieve this, transformations must support determining length and determining value separately when possible. There are these things I call "length protocols" 1) FIXED_LENGTH: the length is static in the meaning of the type. E.g., 4 byte length is implicit in the type "int" 2) STATIC_LENGTH: the length is static as part of the element definition. E.g., 12 digit packed decimal known from the Cobol FD. Or a string with exactly 12 characters. (note that we ignore implications of variable-width chracter encodings like UTF-8 here on purpose more on that below). 3) OUTSIDE_LENGTH: the length is dynamic, and comes from elsewhere. I.e., consider a stored length prefix field. We probably don't have to touch the data to skip past it, for example, though we did have to read the length field someplace to know how far to skip. 4) PARSE_LENGTH: the length is dynamic, and computing the length of the element is as hard as computing the value, so you might as well do them both simultaneously (e.g., delimited text situation) Now the character set issue. If the character set is fixed width, like ascii, ebcdic, or UTF-16, then the above apply as defined. If the data format is text and the character set is variable width, like UTF-8, or Shift-jis, then 1, 2, and 3 all collapse into 4. I.e., all lengths require you to parse the characters one by one. However, I'd like this detail to be pushed down into the DFDL implementation because there are different ways to do it. E.g., you could do like Java and convert everything to UTF-16 first and eliminate the whole issue, or you can try to be more clever. I think transforms must advertise the protocols they support. E.g., intFromBinary in your example supports only FIXED_LENGTH protocol, and it should say the length is exactly 4. intFromAscii should support protocols 2, 3, and 4. Only protocol 4 supports delimiters and their attendant complexities like how embedded delimiters might be quoted or escaped. This "transform" function must compute both an integer value, and also compute the length of consumed data in the underlying stream, or by-side-effect advance the stream to the new position. The point is not to take a position on whether we manage lengths, or have a stateful cursor on the stream, the point is that there are 3 functions to provide. One is parameterized by a static length, One is parameterized by a dynamic length, and the third is parameterized by delimiters, escape sequence specifications, etc. All share the numbase parameter. This all adds baggage, but I think it is necessary or things just can't be efficient. ...mikeb From: Chappell, Alan R [mailto:chappella@BATTELLE.ORG] Sent: Friday, November 19, 2004 4:44 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Transform examples Third try... No zip, just the 3 files important to the simple transform example.... From: Chappell, Alan R Sent: Friday, November 19, 2004 1:39 PM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Second try on sending these examples. I've cut the set down to the 3 important files so hopefully it will get through this time. From: Chappell, Alan R Sent: Thursday, November 18, 2004 8:47 AM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Here is the example I mentioned yesterday. Look particularly at dfdltransforms.xsd, BasicAsciiIntExp.xsd, and BasicBinIntExp.xsd. Note the "Exp" on those last two files indicate that they are expansions of the information in the original versions of those files. These make a first stab at giving a fully verbose description of the structure and the transforms, i.e., it’s working towards the canonical representation Martin talked about yesterday. The "dfdltransforms" gives the definitions of transforms and their components. There are lots of things that can be improved here. <<dfdl-examples.zip>> Alan R. Chappell chappella(a)battelle.org Pacific Northwest National Laboratory Battelle Seattle Research Center (206) 528-3228

1 0

RE: [dfdl-wg] BSDF
by Myers, James D 23 Nov '04

23 Nov '04

I ran across another community working in the same space as DFDL: BSDL as part of MPEG21 (someone at the NARA conference mentioned it). From a quick look at what they have, it appears to be inverse DFDL with plans to make DFDL next, i.e. they go from model to output binary format, but not vice versa. Similar schema approach, etc. with some comments on things in schema that they don't allow. I haven't found a spec doc yet, just a paper and some presentations you can google for. Jim Bitstream Syntax Description Language: Application of XML-Schema ... <http://www2002.org/CDROM/alternate/334/> ... Language (BSDL) that embraces such XML bitstream descriptions. ... 2/; [13] IEEE Standard for Binary Floating-Point ... Final Draft Amendment 4: Streaming video profile ... www2002.org/CDROM/alternate/334/

1 0

Fw: [dfdl-wg] DFDL and binary XML again
by Suman Kalia 23 Nov '04

23 Nov '04

Mike -- This would work well in the simple case where a schema does not include/import another schema. What if the included/imported schema has a conflicting or different annotation - we are then back to precedence rules.. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 11/23/2004 08:32 AM ----- mike.beckerle(a)ascentialsoftware.com Sent by: owner-dfdl-wg(a)ggf.org 11/22/2004 06:38 PM To dfdl-wg(a)gridforum.org cc Subject [dfdl-wg] DFDL and binary XML again Folks, If you just take any XSD, and add a "top level" DFDL annotation including: repType="binary" byteOrder="bigEndian" .... And as well you must provide other top-level defaults for date formats, how nulls are represented by default and so forth. Doing this and then writing out the data should get you a quite standard binary format for XML transmission which ought to satisfy much of the needs of the binaryXML community. I found discussion of DFDL on public-xml-binary@w3c this week, and this notion crossed my mind that while binaryXML formats can't solve the DFDL problem, DFDL *CAN* solve the binaryXML problem. ...mikeb

1 0