September 2005 - dfdl-wg

Telcon today
by Martin Westhead 07 Sep '05

07 Sep '05

Sorry this did not appear yesterday I sent it out but I sent it from the wrong address and it bounced. Preliminary Agenda ------------------ - Plans for F2F at GGF - Issues list review comments/suggestions - Choosing topics/issues for discussion - Additional types (Bob's email) Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call.

1 0

Re: [dfdl-wg] Issues: additional data types
by Martin Westhead 07 Sep '05

07 Sep '05

Hi Robert, As I said in my other reply to you, one of the features that we decided upon with DFDL was that the model that we were translating into was the XML data model. So any file described by DFDL should be translatable into a well formed XML document. The XML data model may not be ideal but it is the standard data model. The reason that the types you describe are not in DFDL is that they are not in XML. That is not to say they are not important or should not be addressed. My opinion on these is that they can be built out of the existing DFDL/XML components and that this is the correct way of handle them. The standard should provide a document that describes one or more ways in which these types can be achieved. More inline... Robert E. McGrath wrote: ..snip... > *1. Enum* > > This type has a set of <name, value> pairs, e.g., <�Red�, 0>, <�Blue�, > 1>, etc. The values are stored in the data, with the name-value pairs > stored in metadata. > > > > Note: one use is for localization, using different maps to give > localized strings. > > > > *Difficulty*: Low > > *Priority:*�� Low I'm confused about what you want to achieve. If you only store the value (an integer) what is the function of the name, is it just for human beings reading the file, or is there some way it is used programatically? One way to approach this type would be a choice over a series of tags with appropriate aattribute types constrained to hardwired values: <enum name="Red" value="0"/> (Complex type with two attributes, one a string constrained to the single value "Red" the other an integer constrained to the value 0). You could then use the DFDL annotations to ensure that this tag gets picked when zero occurs in the file. > *2. Opaque (tagged)* > > This is some kind of non-numeric bit string, with a length and some kind > of tag. > > This might be used, for example, for 1024-bit encryption keys.� The type > means �just pass through the bits�. > > Generally, can be used to store any kind of �blob�, which can be objects > that are meaningful to specific software. > > > > This can be simulated with unsigned integers, but it may be useful to > know that it is not really an integer, or whatever. > > > > *Difficulty*: Low > > *Priority:*�� Low This is just a sequence of bytes... it may need to be hidden (the layering introduces a requirement for it to be possible to be explicit about what is visible at a particular layer). (I don't know what the current favourite way to do this is). > *3. �Code�* > > > > How should �code� be marked up?� It is usually stored in blobs, but it > needs a tag so you know how to interpret it. > > > > This is actually a special case of �opaque�. > > > > *Difficulty*: Low > > *Priority:*�� Low I don't understand what you mean. > > *4. Bitfield / packed* > > > > This type is bits packed into bytes. > > > > *Difficulty*: Low > > *Priority:*�� Low Agreed. I think we can do this but it should be easier. > *5. Pointer� * > > > > Many times there will be pointers within the data, e.g., to offsets in > the file, or to indexes in an array.� This will be critical for storing > objects such as lists or trees. > > > > URL�s� and XPATHS are not especially well suited for this. > > > > This can be simulated with unsigned integers, but they need to be > �swizzled� when translating, so they need to be tagged. > > > > Note that there might be several types of addressing within the data: > > � Offset from zero > > � Offset relative to �foo� > > The offsets might be in different increments:� bits, bytes, words, > elements, etc. > > > > There could be multi-part addresses, e.g., page + offset in page. > > > > *Difficulty*:�� Medium > > *Priority: *��High I spent a long time thinking about pointers at one time. I was unable to come up with anything I felt covered the bases. Perhaps you have enough experience with pointer representations to help us out here. An interesting problem that comes to mind though is what is the XML representation of the pointer value. If it is a tag like: <pointer offset="20" offsetType="bytes" index="5" indexType="float32"/> then all that is needed is to define the metadata conventions that allow that to be correctly interpreted. This is a little unsatisfactory though... > *6. Array* > > > > This is a critical type, must be supported. > > > > There are a lot of issues. > > > > I am preparing a separate memo. > > > > *Difficulty*:�� High > > *Priority: *��Very High We have talked a lot about arrays. A big issue that there are several ways you may want to represent an array within your XML data model. IMO the right way will depend on how you want to use the data. I think the right way to do this is to have a series of recipes for users to capture array semantics in their DFDL files. Cheers, Martin

3 3

Re: [dfdl-wg] Issues: additional data types
by Robert E. McGrath 06 Sep '05

06 Sep '05

On Tuesday 06 September 2005 15:40, Mike Beckerle wrote: > I don't draw a distinction between a sequence of N bytes and an N-byte > represenational-size object of undetermined type. Or rather, the > difference is just of the intent of the person using it. The key role for DFDL is to convey the intent of the producer to the ultimate consumers (without knowing who may consume) > Do you want a type where it is simply an error to access individual bytes? > I.e., so we can express the length in bytes but without expressing that > you should/can access it byte by byte? This is exactly the point of the blob. The tag means "if you don't know what this tag means, don't touch this blob." The case I'm concerned with is a generic grid service that will do something to an array of bytes. E.g., make a JPEG, or run a text mining algorithm. It needs to know the difference between "array of bytes" and "blob".

1 0

split into multiple topics - Re: [dfdl-wg] Issues: additional data types
by Mike Beckerle 06 Sep '05

06 Sep '05

I'd like to split this topic into several distinct ones: Arrays - I have a placeholder for this in the doc. Opaque and "code" types are separate. This is related also to the concept of "open content". Enums Bitfields Pointers Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> Sent by: owner-dfdl-wg(a)ggf.org 09/02/2005 03:13 PM To dfdl-wg(a)gridforum.org cc Subject [dfdl-wg] Issues: additional data types Greetings, Here is an "issue" for the DFDL: additional data types that should be considered. Please see attached. --- Robert E. McGrath National Center for Supercomputing Applications University of Illinois, Urbana-Champaign Champaign, Illinois 61820 (217)-333-6549 mcgrath(a)ncsa.uiuc.edu

4 14

Fw: Enums - Re: split into multiple topics - Re: [dfdl-wg] Issues: additional data types
by Steve Hanson 06 Sep '05

06 Sep '05

You can consider the MRM model as an IBM proprietary equivalent to DFDL. The logical model is XML schema, the physical model is our equivalent of DFDL annotations. I don't see the real world case for having xs:enums on the schema simple types whose values are the names of the 88s. The only requirement I have seen is for level 88 values to participate in validation of the byte stream, which means the xs:enums are the values of the 88s, not the names. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/09/2005 19:43 ----- "Robert E. McGrath" <mcgrath(a)ncsa.uiu To c.edu> dfdl-wg(a)gridforum.org Sent by: cc owner-dfdl-wg@ggf .org Subject Re: Enums - Re: split into multiple topics - Re: [dfdl-wg] Issues: 06/09/2005 18:49 additional data types On Tuesday 06 September 2005 12:21, Steve Hanson wrote: > The MRM model COBOL importer gives the user an option to create > xs:enumeration values from level 88 values, and that is all. This is so > the MRM parser can validate a byte stream against the model. This is in > keeping with the rule that values supplied with metadata are part of the > logical model, not the physical model. What's the motivation for doing > anything more than this? Sorry, I don't know what "the logical model" vs. "the physical model" means. IMO, the goal is to have a way for a producer to generate data (e.g., with 88s as discussed below), which a consumer can interpret as either 88 or symbols, depending on the purpose of the consumer. > > Using your example that gives: > > <xs:element name="AS-COM-TRAN-ID"> > <xs:simpleType> > <xs:restriction base="xs:string"> > <xs:enumeration value="CP80"/> > <xs:enumeration value="IC40"/> > ... > </xs:restriction> > </xs:simpleType> > </xs:element> > > Regards, Steve > > Steve Hanson > WebSphere Business Integration Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 > > > > Mike Beckerle > <beckerle(a)us.ibm. > com> To > Sent by: dfdl-wg(a)gridforum.org > owner-dfdl-wg@ggf cc > .org > Subject > Enums - Re: split into multiple > 06/09/2005 17:40 topics - Re: [dfdl-wg] Issues: > additional data types > > > > > > > > > > > > About enums. Here's starting thoughts: > > Here's a real-world example from COBOL: > > 01 AS-CPST-REC. > 06 AS-CPCOM. > 09 AS-COM-STORE-TYPE PIC X. > 09 AS-COM-STORE-NO PIC 9(05). > 09 AS-COM-TRAN-ID PIC X(04). > 88 TRAN-COUPON VALUE 'CP80'. > 88 TRAN-REVENUE VALUE 'IC40' > 'RA40'. > 88 TRAN-SALES VALUE 'IC40'. > 88 TRAN-DELIVER VALUE 'IC44'. > 88 TRAN-RENTS VALUE 'RA40' > 'RA42'. > 88 TRAN-RENT-RETURN VALUE 'RA41'. > 09 AS-COM-QUANTITY PIC S9(05). > 09 AS-COM-PART-NO PIC 9(06). > .... more fields elided .... > > Those "88" entries in there are enumerated constants. Note that for the > TRAN-REVENUE and TRAN-RENTS constants, multiple values are associated with > the same name. On reference this means that the constant matches either > value. When written, this means the first value is used. Cobol doesn't > strongly associate these enumerated values with the field to which they can > be assigned, but usually it's obvious. In this case it is the > AS-COM-TRAN-ID field which is the 4-character-long string which has the > string constants associated with it. > > This particular example is a common one. The record has variant structure > (not shown above) depending on the tag field which is this AS-COM-TRAN-ID > field. > > Working out how we want this example to work in XSD is the first step. > > I like using a hidden field here. For example: Here's a possible idea for > how this is represented in XSD: > > > <xs:element name="AS-COM-TRAN-ID">  > <xs:simpleType> > <xs:restriction base="xs:NCName"> > <xs:enumeration value="TRAN-COUPON"/> > <xs:enumeration value="TRAN-REVENUE"/> > <xs:enumeration value="TRAN-SALES"/> > <xs:enumeration value="TRAN-DELIVER"/> > <xs:enumeration value="TRAN-RENTS"/> > </xs:restriction> > </xs:simpleType> > </xs:element> > <xs:sequence> > <xs:annotation> > <xs:appinfo source="http://dataformat.org/"> > <xs:layer name="rep" type="AS-COM-TRAN-ID-repType"/>  > </xs:appinfo> > </xs:annotation> > </xs:sequence> > > <xs:simpleType name="AS-COM-TRAN-ID-repType"> > <xs:restriction base="xs:string> > <xs:enumeration value="CP80"/> > <xs:enumeration value="IC40"/>  > .... > </xs:restriction> > </xs:simpleType> > > Now in an annotation (not shown above) on the element AS-COM-TRAN-ID there > would be a dfdl:valueCalc property which would compute the value of > AS-COM-TRAN-ID based on the value of the hidden field. Symmetrically, the > 'rep' hidden field would have a dfdl:repCalc property which would give the > inverse formula for output. > > One difficulty I have with this is the notion that we're projecting into > the string type. I.e., these symbolic constants aren't names for integers, > but rather we're expressing operations on strings. In the above example the > enumerated constants actually are strings, but in other examples they would > be integers. The next tier of interpretation, i.e., where we're decidng the > variant based on the value of AS-COM-TRAN-ID would be expressed as string > comparisons which is potentially inefficient. This is part of the problem > with using XSD as our type system basis. XSD doesn't have a notion of > symbolic named constant. > > Alternative: There is the DTD named entity stuff. Does anybody want to > propose that? > > Mike Beckerle > Architect, Scalable Computing > IBM Software Group > Information Integration Solutions > Westborough, MA > > > Mike > Beckerle/Worcester/IBM@IBMUS > > Sent by: To > owner-dfdl-wg(a)ggf.org "Robert E. McGrath" > <mcgrath(a)ncsa.uiuc.edu> > cc > 09/02/2005 04:34 PM dfdl-wg(a)gridforum.org, > owner-dfdl-wg(a)ggf.org > Subject > split into multiple topics - Re: > [dfdl-wg] Issues: additional > data types > > > > > > > > > > > > > > I'd like to split this topic into several distinct ones: > > Arrays - I have a placeholder for this in the doc. > > Opaque and "code" types are separate. This is related also to the concept > of "open content". > > Enums > > Bitfields > > Pointers > > > Mike Beckerle > Architect, Scalable Computing > IBM Software Group > Information Integration Solutions > Westborough, MA > > "Robert E. McGrath" > <mcgrath(a)ncsa.uiuc.edu> > Sent by: owner-dfdl-wg(a)ggf.org > To > dfdl-wg(a)gridforum.org > 09/02/2005 03:13 PM cc > > Subject > [dfdl-wg] Issues: > additional data types > > > > > > > > > > > > > > > Greetings, > > Here is an "issue" for the DFDL: additional data types that should > be considered. > > Please see attached. > > --- > Robert E. McGrath > National Center for Supercomputing Applications > University of Illinois, Urbana-Champaign > Champaign, Illinois 61820 > (217)-333-6549 > > mcgrath(a)ncsa.uiuc.edu (See attached file: DT.htm) -- --- Robert E. McGrath National Center for Supercomputing Applications University of Illinois, Urbana-Champaign Champaign, Illinois 61820 (217)-333-6549 mcgrath(a)ncsa.uiuc.edu

1 0

Plumbing document
by Martin Westhead 06 Sep '05

06 Sep '05

Hi Folks, In the discussion at the end of the meeting we mentioned a document I wrote following the discussion at the December f2f. I don't think it made it as far a Gridforge so I have just added it at: https://forge.gridforum.org/docman2/ViewCategory.php?group_id=113&category_… It is an attempt at starting to describe compositional features of DFDL in a formal/semi-formal way. I think that as we add layering something like this is going to be essential to keep everything clear. Cheers, Martin

3 4

Re: Arrays issue - Re: [dfdl-wg] Issues: additional data types
by Jim Myers 06 Sep '05

06 Sep '05

1 0

Agenda items for tomorrow's meeting?
by Martin Westhead 06 Sep '05

06 Sep '05

Hi Folks, Does anyone have any items for tomorrow's agenda? I have: - Additional types (Bob's email) - Initial issues list (I will put this out later today) - Issues with draft (perhaps the authors or those commenting could pick some issues to discuss? If we can put them our in advance then others can review and prepare for discussion) Please send me (or the list) any topics, on the draft or otherwise, you'd like adding. Cheers, Martin

2 1

Comments on DFDL draft spec
by Steve Hanson 04 Sep '05

04 Sep '05

The draft is a big step forward for DFDL. Here are the initial comments from the IBM Hursley team. We have not reviewed all the Appendices yet, including the detailed properties list, but I am on a course all this week so I think it makes sense to send what we have so far to the group. (See attached file: IBM Hursley MRM team comments on DFDL draft specification Aug 2005.doc) Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0