- dfdl-wg - lists.ogf.org

Fw: [dfdl-wg] Transform examples
by Steve Hanson 24 Nov '04

24 Nov '04

I will second Mike's points here. As discussed on one of our calls, DFDL 'transforms' are the equivalent to a set of IBM model properties. The key information needed to correctly and efficiently parse a string are, in our model: - Physical type (fixed length, length prefixed, null terminated, etc - same as Mike's length protocols) - Length count (if fixed length) - Length reference (name of another field in the message giving the length - alternative to length count) - Ccsid - Length units (handles the character set issue Mike noted below, in conjunction with the ccsid) - Justification - Padding character (these last two used because we will auto-strip/add padding chars on read/write) Regards, Steve ----- Forwarded by Steve Hanson/UK/IBM on 24/11/2004 10:52 ----- mike.beckerle@asc entialsoftware.co m To Sent by: jim.myers(a)pnl.gov, owner-dfdl-wg@ggf dfdl-wg(a)gridforum.org .org cc Subject 22/11/2004 17:26 RE: [dfdl-wg] Transform examples I'm just trying to avoid having DFDL's design preclude the creation of an efficient implementation. It's easy to do this if we're not careful. E.g., if we ignore length issues then I can guarantee we won't be able to meet our previously-stated goals of making efficient random access possible. I can speak from experience here in that I have a mature implementation of a system like DFDL, just not an industry standard one. In the implementation one of the most critical things is the ability to rapidly get to the ends of records, or fields, without having to process the intervening data. Much effort in the implementation is devoted to these kinds of length machinations, and we use a plug in architecture where each data type and all converters on them can be extended and plugged in anew. For example we didn't originally have any date/time or dateTime types, but added them afterwards. All types and converters obey a number of different protocols associated with how length works. I can't see any other way to do it. That said, we want the minimum complexity in DFDL at the interface to the transformations that enables efficient implementation. ...mikeb From: Myers, James D [mailto:jim.myers@pnl.gov] Sent: Monday, November 22, 2004 11:51 AM To: dfdl-wg(a)gridforum.org Subject: RE: [dfdl-wg] Transform examples Just a high level comment (not sure I'd split things up the same way you have but I won't comment on that now ...): I'm not averse to putting something in DFDL about the expectations about length of fields, but is this useful in practice versus simply coding this in the parser you build? The parser is really the thing that will use it to calculate offsets, so some methods related to getting offsets on the transform classes are really what's needed - is that made simpler if there's info in the DFDL transform description? (And can I safely ignore this info in a dumb reference implementation that can only calculate offsets by actually parsing and then counting?) Jim -----Original Message----- From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of mike.beckerle(a)ascentialsoftware.com Sent: Monday, November 22, 2004 11:01 AM To: Chappell, Alan R; dfdl-wg(a)gridforum.org Subject: RE: [dfdl-wg] Transform examples Alan, I looked at these examples. There's one thing I think you've overlooked in the way transforms are specified here. This is the fact that intFromBinary knows that it will pick exactly 4 bytes off the input stream, and could advertise that property to the DFDL "system" in some way, whereas intFromAscii might take anything from 1 to however many characters. E.g., it might be able to tolerate whitespace of any size, leading zeros, etc. So as a transform it needs to advertise that the length of data being consumed requires that you run the transform. Where I'm coming from is this. It is very important that a DFDL description of data enable processing the data efficiently. To me that means that if data is all fixed width, then one should be able to randomly access fields in the data in constant time. Even if the data is variable width, one should be able to efficiently skip through it to find the boundaries without necessarily having to process all the data, convert to common format, etc. To achieve this, transformations must support determining length and determining value separately when possible. There are these things I call "length protocols" 1) FIXED_LENGTH: the length is static in the meaning of the type. E.g., 4 byte length is implicit in the type "int" 2) STATIC_LENGTH: the length is static as part of the element definition. E.g., 12 digit packed decimal known from the Cobol FD. Or a string with exactly 12 characters. (note that we ignore implications of variable-width chracter encodings like UTF-8 here on purpose more on that below). 3) OUTSIDE_LENGTH: the length is dynamic, and comes from elsewhere. I.e., consider a stored length prefix field. We probably don't have to touch the data to skip past it, for example, though we did have to read the length field someplace to know how far to skip. 4) PARSE_LENGTH: the length is dynamic, and computing the length of the element is as hard as computing the value, so you might as well do them both simultaneously (e.g., delimited text situation) Now the character set issue. If the character set is fixed width, like ascii, ebcdic, or UTF-16, then the above apply as defined. If the data format is text and the character set is variable width, like UTF-8, or Shift-jis, then 1, 2, and 3 all collapse into 4. I.e., all lengths require you to parse the characters one by one. However, I'd like this detail to be pushed down into the DFDL implementation because there are different ways to do it. E.g., you could do like Java and convert everything to UTF-16 first and eliminate the whole issue, or you can try to be more clever. I think transforms must advertise the protocols they support. E.g., intFromBinary in your example supports only FIXED_LENGTH protocol, and it should say the length is exactly 4. intFromAscii should support protocols 2, 3, and 4. Only protocol 4 supports delimiters and their attendant complexities like how embedded delimiters might be quoted or escaped. This "transform" function must compute both an integer value, and also compute the length of consumed data in the underlying stream, or by-side-effect advance the stream to the new position. The point is not to take a position on whether we manage lengths, or have a stateful cursor on the stream, the point is that there are 3 functions to provide. One is parameterized by a static length, One is parameterized by a dynamic length, and the third is parameterized by delimiters, escape sequence specifications, etc. All share the numbase parameter. This all adds baggage, but I think it is necessary or things just can't be efficient. ...mikeb From: Chappell, Alan R [mailto:chappella@BATTELLE.ORG] Sent: Friday, November 19, 2004 4:44 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Transform examples Third try... No zip, just the 3 files important to the simple transform example.... From: Chappell, Alan R Sent: Friday, November 19, 2004 1:39 PM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Second try on sending these examples. I've cut the set down to the 3 important files so hopefully it will get through this time. From: Chappell, Alan R Sent: Thursday, November 18, 2004 8:47 AM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Here is the example I mentioned yesterday. Look particularly at dfdltransforms.xsd, BasicAsciiIntExp.xsd, and BasicBinIntExp.xsd. Note the "Exp" on those last two files indicate that they are expansions of the information in the original versions of those files. These make a first stab at giving a fully verbose description of the structure and the transforms, i.e., it’s working towards the canonical representation Martin talked about yesterday. The "dfdltransforms" gives the definitions of transforms and their components. There are lots of things that can be improved here. <<dfdl-examples.zip>> Alan R. Chappell chappella(a)battelle.org Pacific Northwest National Laboratory Battelle Seattle Research Center (206) 528-3228

1 0

RE: [dfdl-wg] BSDF
by Myers, James D 23 Nov '04

23 Nov '04

I ran across another community working in the same space as DFDL: BSDL as part of MPEG21 (someone at the NARA conference mentioned it). From a quick look at what they have, it appears to be inverse DFDL with plans to make DFDL next, i.e. they go from model to output binary format, but not vice versa. Similar schema approach, etc. with some comments on things in schema that they don't allow. I haven't found a spec doc yet, just a paper and some presentations you can google for. Jim Bitstream Syntax Description Language: Application of XML-Schema ... <http://www2002.org/CDROM/alternate/334/> ... Language (BSDL) that embraces such XML bitstream descriptions. ... 2/; [13] IEEE Standard for Binary Floating-Point ... Final Draft Amendment 4: Streaming video profile ... www2002.org/CDROM/alternate/334/

1 0

Fw: [dfdl-wg] DFDL and binary XML again
by Suman Kalia 23 Nov '04

23 Nov '04

Mike -- This would work well in the simple case where a schema does not include/import another schema. What if the included/imported schema has a conflicting or different annotation - we are then back to precedence rules.. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 11/23/2004 08:32 AM ----- mike.beckerle(a)ascentialsoftware.com Sent by: owner-dfdl-wg(a)ggf.org 11/22/2004 06:38 PM To dfdl-wg(a)gridforum.org cc Subject [dfdl-wg] DFDL and binary XML again Folks, If you just take any XSD, and add a "top level" DFDL annotation including: repType="binary" byteOrder="bigEndian" .... And as well you must provide other top-level defaults for date formats, how nulls are represented by default and so forth. Doing this and then writing out the data should get you a quite standard binary format for XML transmission which ought to satisfy much of the needs of the binaryXML community. I found discussion of DFDL on public-xml-binary@w3c this week, and this notion crossed my mind that while binaryXML formats can't solve the DFDL problem, DFDL *CAN* solve the binaryXML problem. ...mikeb

1 0

Fw: [dfdl-wg] Annotation complexity
by Steve Hanson 23 Nov '04

23 Nov '04

One more thought in this area. XML Schema itself does not have the notion of inheriting properties from a parent object. For example, if I have a complex type that contains only integer elements, there is no facility to factor out the xsd:type attribute from all the elements and instead set it as an attribute of the complex type. Was this a deliberate decision? I'll see if I can find out. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/11/2004 09:27 ----- Steve Hanson/UK/IBM To 22/11/2004 11:24 dfdl-wg(a)gridforum.org cc Subject RE: [dfdl-wg] Annotation complexity (Document link: Steve Hanson) > Another approach would be to allow annotation locations to be > specified using XPath. Since XPath can specify both specific > locations and large groups of locations this could be quite > powerful. We used to have a similar concept in the IBM broker message model, we called it 'element qualifier'. A top level global element could carry a reference (XPATH) to an element it ultimately contained, for the purposes of overriding certain properties, including default value. We found that: a) Their use made the model much harder to understand b) The code we had to write was considerably more complex c) No customer ever admitted to using the capability We withdrew the facility with no deprecation warning in V5 (June 2003) and so far nobody has shouted. Was this the kind of thing you are proposing? Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Myers, James D" <jim.myers(a)pnl.go v> To Sent by: dfdl-wg(a)gridforum.org owner-dfdl-wg@ggf cc .org Subject RE: [dfdl-wg] Annotation complexity 19/11/2004 21:03 > > Myers, James D wrote: > > I guess I'm not sure how restricting annotations to elements solves > > things. > > Sorry, I am finding it more difficult to be precise than I > thought with > this. What I mean here is _leaf_ elements/attributes - which > I think can > be defined as elements with simple type descriptions. I think if you > stick to these its unambiguous...no? I agree, I think - if anotations are only on simple types. That would eliminate the element hierarchy, though - does it eliminate the type hierarchy - simple element in base type has an annotation that gets inherited by the same simple element in a derived type. I think this would be unambiguous, but perhaps hard to find (surprise the global dfdl default for endianess doesn't apply because some subtype three levels back had an annotation) > > I think Mike has examples of annotations that he would like higher in > the tree but with this extreme position there would be no inheritance. Except through type derivation...? > > Two other thoughts occur: > - I think definitions applied to a data source are > orthogonal to this > scoping issue. We still need to understand precedence but that seems > relatively easy to resolve. Definitions on sources: I know we've talked about this (!!! - the whole outside-in or inside out representations...), but I think the only mechanism for this in the examples is to assign a source to an element and annotate that element with, for example, endianness. For some types of sources, e.g. other layers, that have an internal structure, any annotation of the source itself would essentially become an inheritable annotation on the elements within it. So - we could try to define such a mechanism to apply to sources, but I'm not sure it would be applicable to all without recreating some of the inheritance issues. > > - Another approach would be to allow annotation locations to be > specified using XPath. Since XPath can specify both specific > locations > and large groups of locations this could be quite powerful. Hmmm. Yes, as long as xpaths can only be defined from a single root, i.e. not from within type definitions relative to the type (not talking about layers here). Otherwise we don't get rid of inheritance/precedence issues. At a minimum, this approach might help collect all of the defaults together at the top level for readability, but it might separate defaults from reusable types. Jim

1 0

DFDL and binary XML again
by mike.beckerle＠ascentialsoftware.com 22 Nov '04

22 Nov '04

Folks, If you just take any XSD, and add a "top level" DFDL annotation including: repType="binary" byteOrder="bigEndian" .... And as well you must provide other top-level defaults for date formats, how nulls are represented by default and so forth. Doing this and then writing out the data should get you a quite standard binary format for XML transmission which ought to satisfy much of the needs of the binaryXML community. I found discussion of DFDL on public-xml-binary@w3c this week, and this notion crossed my mind that while binaryXML formats can't solve the DFDL problem, DFDL *CAN* solve the binaryXML problem. ...mikeb

1 0

Transform examples
by Chappell, Alan R 22 Nov '04

22 Nov '04

Third try... No zip, just the 3 files important to the simple transform example.... _____ From: Chappell, Alan R Sent: Friday, November 19, 2004 1:39 PM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Second try on sending these examples. I've cut the set down to the 3 important files so hopefully it will get through this time. _____ From: Chappell, Alan R Sent: Thursday, November 18, 2004 8:47 AM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Here is the example I mentioned yesterday. Look particularly at dfdltransforms.xsd, BasicAsciiIntExp.xsd, and BasicBinIntExp.xsd. Note the "Exp" on those last two files indicate that they are expansions of the information in the original versions of those files. These make a first stab at giving a fully verbose description of the structure and the transforms, i.e., it's working towards the canonical representation Martin talked about yesterday. The "dfdltransforms" gives the definitions of transforms and their components. There are lots of things that can be improved here. <<dfdl-examples.zip>> Alan R. Chappell chappella(a)battelle.org Pacific Northwest National Laboratory Battelle Seattle Research Center (206) 528-3228

2 1

RE: [dfdl-wg] Transform examples
by mike.beckerle＠ascentialsoftware.com 22 Nov '04

22 Nov '04

I'm just trying to avoid having DFDL's design preclude the creation of an efficient implementation. It's easy to do this if we're not careful. E.g., if we ignore length issues then I can guarantee we won't be able to meet our previously-stated goals of making efficient random access possible. I can speak from experience here in that I have a mature implementation of a system like DFDL, just not an industry standard one. In the implementation one of the most critical things is the ability to rapidly get to the ends of records, or fields, without having to process the intervening data. Much effort in the implementation is devoted to these kinds of length machinations, and we use a plug in architecture where each data type and all converters on them can be extended and plugged in anew. For example we didn't originally have any date/time or dateTime types, but added them afterwards. All types and converters obey a number of different protocols associated with how length works. I can't see any other way to do it. That said, we want the minimum complexity in DFDL at the interface to the transformations that enables efficient implementation. ...mikeb _____ From: Myers, James D [mailto:jim.myers@pnl.gov] Sent: Monday, November 22, 2004 11:51 AM To: dfdl-wg(a)gridforum.org Subject: RE: [dfdl-wg] Transform examples Just a high level comment (not sure I'd split things up the same way you have but I won't comment on that now ...): I'm not averse to putting something in DFDL about the expectations about length of fields, but is this useful in practice versus simply coding this in the parser you build? The parser is really the thing that will use it to calculate offsets, so some methods related to getting offsets on the transform classes are really what's needed - is that made simpler if there's info in the DFDL transform description? (And can I safely ignore this info in a dumb reference implementation that can only calculate offsets by actually parsing and then counting?) Jim -----Original Message----- From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of mike.beckerle(a)ascentialsoftware.com Sent: Monday, November 22, 2004 11:01 AM To: Chappell, Alan R; dfdl-wg(a)gridforum.org Subject: RE: [dfdl-wg] Transform examples Alan, I looked at these examples. There's one thing I think you've overlooked in the way transforms are specified here. This is the fact that intFromBinary knows that it will pick exactly 4 bytes off the input stream, and could advertise that property to the DFDL "system" in some way, whereas intFromAscii might take anything from 1 to however many characters. E.g., it might be able to tolerate whitespace of any size, leading zeros, etc. So as a transform it needs to advertise that the length of data being consumed requires that you run the transform. Where I'm coming from is this. It is very important that a DFDL description of data enable processing the data efficiently. To me that means that if data is all fixed width, then one should be able to randomly access fields in the data in constant time. Even if the data is variable width, one should be able to efficiently skip through it to find the boundaries without necessarily having to process all the data, convert to common format, etc. To achieve this, transformations must support determining length and determining value separately when possible. There are these things I call "length protocols" 1) FIXED_LENGTH: the length is static in the meaning of the type. E.g., 4 byte length is implicit in the type "int" 2) STATIC_LENGTH: the length is static as part of the element definition. E.g., 12 digit packed decimal known from the Cobol FD. Or a string with exactly 12 characters. (note that we ignore implications of variable-width chracter encodings like UTF-8 here on purpose more on that below). 3) OUTSIDE_LENGTH: the length is dynamic, and comes from elsewhere. I.e., consider a stored length prefix field. We probably don't have to touch the data to skip past it, for example, though we did have to read the length field someplace to know how far to skip. 4) PARSE_LENGTH: the length is dynamic, and computing the length of the element is as hard as computing the value, so you might as well do them both simultaneously (e.g., delimited text situation) Now the character set issue. If the character set is fixed width, like ascii, ebcdic, or UTF-16, then the above apply as defined. If the data format is text and the character set is variable width, like UTF-8, or Shift-jis, then 1, 2, and 3 all collapse into 4. I.e., all lengths require you to parse the characters one by one. However, I'd like this detail to be pushed down into the DFDL implementation because there are different ways to do it. E.g., you could do like Java and convert everything to UTF-16 first and eliminate the whole issue, or you can try to be more clever. I think transforms must advertise the protocols they support. E.g., intFromBinary in your example supports only FIXED_LENGTH protocol, and it should say the length is exactly 4. intFromAscii should support protocols 2, 3, and 4. Only protocol 4 supports delimiters and their attendant complexities like how embedded delimiters might be quoted or escaped. This "transform" function must compute both an integer value, and also compute the length of consumed data in the underlying stream, or by-side-effect advance the stream to the new position. The point is not to take a position on whether we manage lengths, or have a stateful cursor on the stream, the point is that there are 3 functions to provide. One is parameterized by a static length, One is parameterized by a dynamic length, and the third is parameterized by delimiters, escape sequence specifications, etc. All share the numbase parameter. This all adds baggage, but I think it is necessary or things just can't be efficient. ...mikeb _____ From: Chappell, Alan R [mailto:chappella@BATTELLE.ORG] Sent: Friday, November 19, 2004 4:44 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Transform examples Third try... No zip, just the 3 files important to the simple transform example.... _____ From: Chappell, Alan R Sent: Friday, November 19, 2004 1:39 PM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Second try on sending these examples. I've cut the set down to the 3 important files so hopefully it will get through this time. _____ From: Chappell, Alan R Sent: Thursday, November 18, 2004 8:47 AM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Here is the example I mentioned yesterday. Look particularly at dfdltransforms.xsd, BasicAsciiIntExp.xsd, and BasicBinIntExp.xsd. Note the "Exp" on those last two files indicate that they are expansions of the information in the original versions of those files. These make a first stab at giving a fully verbose description of the structure and the transforms, i.e., it's working towards the canonical representation Martin talked about yesterday. The "dfdltransforms" gives the definitions of transforms and their components. There are lots of things that can be improved here. <<dfdl-examples.zip>> Alan R. Chappell chappella(a)battelle.org Pacific Northwest National Laboratory Battelle Seattle Research Center (206) 528-3228

1 0

DFDL in the news
by Myers, James D 22 Nov '04

22 Nov '04

FYI: Looks like the 10 minute talk I gave at the NARA meeting ended up in the news (see below). Looks like only a minor distortion (other than one slide in ppt with animation, I didn't demonstrate anything...). I think the slides will be put up at NARA and there may be a streaming video of the whole conference put up at some point. James Myers, Computational Sciences and Mathematics, was quoted in Government Computer News (GCN): <http://gcn.com/vol1_no1/daily-updates/27942-1.html>

1 0

RE: [dfdl-wg] Transform examples
by Myers, James D 22 Nov '04

22 Nov '04

Just a high level comment (not sure I'd split things up the same way you have but I won't comment on that now ...): I'm not averse to putting something in DFDL about the expectations about length of fields, but is this useful in practice versus simply coding this in the parser you build? The parser is really the thing that will use it to calculate offsets, so some methods related to getting offsets on the transform classes are really what's needed - is that made simpler if there's info in the DFDL transform description? (And can I safely ignore this info in a dumb reference implementation that can only calculate offsets by actually parsing and then counting?) Jim -----Original Message----- From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of mike.beckerle(a)ascentialsoftware.com Sent: Monday, November 22, 2004 11:01 AM To: Chappell, Alan R; dfdl-wg(a)gridforum.org Subject: RE: [dfdl-wg] Transform examples Alan, I looked at these examples. There's one thing I think you've overlooked in the way transforms are specified here. This is the fact that intFromBinary knows that it will pick exactly 4 bytes off the input stream, and could advertise that property to the DFDL "system" in some way, whereas intFromAscii might take anything from 1 to however many characters. E.g., it might be able to tolerate whitespace of any size, leading zeros, etc. So as a transform it needs to advertise that the length of data being consumed requires that you run the transform. Where I'm coming from is this. It is very important that a DFDL description of data enable processing the data efficiently. To me that means that if data is all fixed width, then one should be able to randomly access fields in the data in constant time. Even if the data is variable width, one should be able to efficiently skip through it to find the boundaries without necessarily having to process all the data, convert to common format, etc. To achieve this, transformations must support determining length and determining value separately when possible. There are these things I call "length protocols" 1) FIXED_LENGTH: the length is static in the meaning of the type. E.g., 4 byte length is implicit in the type "int" 2) STATIC_LENGTH: the length is static as part of the element definition. E.g., 12 digit packed decimal known from the Cobol FD. Or a string with exactly 12 characters. (note that we ignore implications of variable-width chracter encodings like UTF-8 here on purpose more on that below). 3) OUTSIDE_LENGTH: the length is dynamic, and comes from elsewhere. I.e., consider a stored length prefix field. We probably don't have to touch the data to skip past it, for example, though we did have to read the length field someplace to know how far to skip. 4) PARSE_LENGTH: the length is dynamic, and computing the length of the element is as hard as computing the value, so you might as well do them both simultaneously (e.g., delimited text situation) Now the character set issue. If the character set is fixed width, like ascii, ebcdic, or UTF-16, then the above apply as defined. If the data format is text and the character set is variable width, like UTF-8, or Shift-jis, then 1, 2, and 3 all collapse into 4. I.e., all lengths require you to parse the characters one by one. However, I'd like this detail to be pushed down into the DFDL implementation because there are different ways to do it. E.g., you could do like Java and convert everything to UTF-16 first and eliminate the whole issue, or you can try to be more clever. I think transforms must advertise the protocols they support. E.g., intFromBinary in your example supports only FIXED_LENGTH protocol, and it should say the length is exactly 4. intFromAscii should support protocols 2, 3, and 4. Only protocol 4 supports delimiters and their attendant complexities like how embedded delimiters might be quoted or escaped. This "transform" function must compute both an integer value, and also compute the length of consumed data in the underlying stream, or by-side-effect advance the stream to the new position. The point is not to take a position on whether we manage lengths, or have a stateful cursor on the stream, the point is that there are 3 functions to provide. One is parameterized by a static length, One is parameterized by a dynamic length, and the third is parameterized by delimiters, escape sequence specifications, etc. All share the numbase parameter. This all adds baggage, but I think it is necessary or things just can't be efficient. ...mikeb ________________________________ From: Chappell, Alan R [mailto:chappella@BATTELLE.ORG] Sent: Friday, November 19, 2004 4:44 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Transform examples Third try... No zip, just the 3 files important to the simple transform example.... ________________________________ From: Chappell, Alan R Sent: Friday, November 19, 2004 1:39 PM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Second try on sending these examples. I've cut the set down to the 3 important files so hopefully it will get through this time. ________________________________ From: Chappell, Alan R Sent: Thursday, November 18, 2004 8:47 AM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Here is the example I mentioned yesterday. Look particularly at dfdltransforms.xsd, BasicAsciiIntExp.xsd, and BasicBinIntExp.xsd. Note the "Exp" on those last two files indicate that they are expansions of the information in the original versions of those files. These make a first stab at giving a fully verbose description of the structure and the transforms, i.e., it's working towards the canonical representation Martin talked about yesterday. The "dfdltransforms" gives the definitions of transforms and their components. There are lots of things that can be improved here. <<dfdl-examples.zip>> Alan R. Chappell chappella(a)battelle.org Pacific Northwest National Laboratory Battelle Seattle Research Center (206) 528-3228

1 0

RE: [dfdl-wg] Transform examples
by mike.beckerle＠ascentialsoftware.com 22 Nov '04

22 Nov '04

Alan, I looked at these examples. There's one thing I think you've overlooked in the way transforms are specified here. This is the fact that intFromBinary knows that it will pick exactly 4 bytes off the input stream, and could advertise that property to the DFDL "system" in some way, whereas intFromAscii might take anything from 1 to however many characters. E.g., it might be able to tolerate whitespace of any size, leading zeros, etc. So as a transform it needs to advertise that the length of data being consumed requires that you run the transform. Where I'm coming from is this. It is very important that a DFDL description of data enable processing the data efficiently. To me that means that if data is all fixed width, then one should be able to randomly access fields in the data in constant time. Even if the data is variable width, one should be able to efficiently skip through it to find the boundaries without necessarily having to process all the data, convert to common format, etc. To achieve this, transformations must support determining length and determining value separately when possible. There are these things I call "length protocols" 1) FIXED_LENGTH: the length is static in the meaning of the type. E.g., 4 byte length is implicit in the type "int" 2) STATIC_LENGTH: the length is static as part of the element definition. E.g., 12 digit packed decimal known from the Cobol FD. Or a string with exactly 12 characters. (note that we ignore implications of variable-width chracter encodings like UTF-8 here on purpose more on that below). 3) OUTSIDE_LENGTH: the length is dynamic, and comes from elsewhere. I.e., consider a stored length prefix field. We probably don't have to touch the data to skip past it, for example, though we did have to read the length field someplace to know how far to skip. 4) PARSE_LENGTH: the length is dynamic, and computing the length of the element is as hard as computing the value, so you might as well do them both simultaneously (e.g., delimited text situation) Now the character set issue. If the character set is fixed width, like ascii, ebcdic, or UTF-16, then the above apply as defined. If the data format is text and the character set is variable width, like UTF-8, or Shift-jis, then 1, 2, and 3 all collapse into 4. I.e., all lengths require you to parse the characters one by one. However, I'd like this detail to be pushed down into the DFDL implementation because there are different ways to do it. E.g., you could do like Java and convert everything to UTF-16 first and eliminate the whole issue, or you can try to be more clever. I think transforms must advertise the protocols they support. E.g., intFromBinary in your example supports only FIXED_LENGTH protocol, and it should say the length is exactly 4. intFromAscii should support protocols 2, 3, and 4. Only protocol 4 supports delimiters and their attendant complexities like how embedded delimiters might be quoted or escaped. This "transform" function must compute both an integer value, and also compute the length of consumed data in the underlying stream, or by-side-effect advance the stream to the new position. The point is not to take a position on whether we manage lengths, or have a stateful cursor on the stream, the point is that there are 3 functions to provide. One is parameterized by a static length, One is parameterized by a dynamic length, and the third is parameterized by delimiters, escape sequence specifications, etc. All share the numbase parameter. This all adds baggage, but I think it is necessary or things just can't be efficient. ...mikeb _____ From: Chappell, Alan R [mailto:chappella@BATTELLE.ORG] Sent: Friday, November 19, 2004 4:44 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] Transform examples Third try... No zip, just the 3 files important to the simple transform example.... _____ From: Chappell, Alan R Sent: Friday, November 19, 2004 1:39 PM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Second try on sending these examples. I've cut the set down to the 3 important files so hopefully it will get through this time. _____ From: Chappell, Alan R Sent: Thursday, November 18, 2004 8:47 AM To: dfdl-wg(a)gridforum.org Subject: *MJ-REJECTED* Transform examples Here is the example I mentioned yesterday. Look particularly at dfdltransforms.xsd, BasicAsciiIntExp.xsd, and BasicBinIntExp.xsd. Note the "Exp" on those last two files indicate that they are expansions of the information in the original versions of those files. These make a first stab at giving a fully verbose description of the structure and the transforms, i.e., it's working towards the canonical representation Martin talked about yesterday. The "dfdltransforms" gives the definitions of transforms and their components. There are lots of things that can be improved here. <<dfdl-examples.zip>> Alan R. Chappell chappella(a)battelle.org Pacific Northwest National Laboratory Battelle Seattle Research Center (206) 528-3228

1 0