- dfdl-wg - lists.ogf.org

Issues list update
by Martin Westhead 07 Sep '05

07 Sep '05

Since the issues list seems to be useful. I have updated it after today's discussion. I have also tidied up a few formating glitches. Please let me know if there are any further problems (strange characters etc.) with the way it is displayed. I have added an Issues List Master document which is the Open Office version (.sxc) of this document. Unfortunately Gridforge adds a .unknown extension when you download it, this may need to be deleted to get open office to accept it. In future if you are constructing a list of comments for the group that you believe are likely to end up as issues it would be very helpful if you could arrange them in something close to this format. i.e. a table in some format (Excel, Word, HTML, Open Office, etc) with some of these columns. Then the relevant columns can be cut and pasted into place. Thanks, Martin

1 0

Fw: [dfdl-wg] Opaque/BLOB/Uninterpreted/Raw - also hexBinary - (was Re: split into multiple topics - Re: [dfdl-wg] Issues: additional data types)
by Steve Hanson 07 Sep '05

07 Sep '05

I prefer the hexBinary approach. The disadvantage Mike observes with the array of bytes approach is not limited to conversion to XML. It would also apply to any in-memory tree (SDO etc) built by the DFDL parser. And the element unnecessarily becomes eligible to take the dfdl:occurs annotation. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 07/09/2005 11:23 ----- Mike Beckerle <beckerle(a)us.ibm. com> To Sent by: Mike Beckerle <beckerle(a)us.ibm.com> owner-dfdl-wg@ggf cc .org dfdl-wg(a)gridforum.org, "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu>, owner-dfdl-wg(a)ggf.org 06/09/2005 21:21 Subject [dfdl-wg] Opaque/BLOB/Uninterpreted/Raw - also hexBinary - (was Re: split into multiple topics - Re: [dfdl-wg] Issues: additional data types) re: Opaque or uninterpreted or raw fields. These are sometimes called Blobs, though database people reserve that term for the acronym "BLOB" which stands for Binary Large Object, which has to do with size being too large for the smaller binary SQL type objects. I.e., there's no such thing as a small BLOB in databases. I think in our mailing list we've used blob to mean "opaque bytes" of any size at all. I believe use of the 'hexBinary' type is also probably this same topic. I.e., how to deal with data where you don't know its proper interpretation, though you can express how big it is so that we can at least copy it from place to place. I think there are two choices here. One is just use "occuring" bytes. E.g., here's uniterpreted data of length 1234 bytes: <element name="ignoreMe" type="byte" minOccurs="1234" maxOccurs="1234" dfdl:repType="binary"/> This is a basic binary byte array. I think this works fine as a blob/opaque type. I believe we do not need any other kind of raw/opaque type. If we had one, we'd have to have a way to express its length, and be specific about the units of that length, and the above accomplishes that with pretty much minimum baggage. You name it what you want, i.e, "unused" or "dummy" or "ignore" or whatever you want. We might want an annotation to indicate that this data should not be accessed, to distinguish this case from an actually array of bytes that you DO want to access, but I'm not sure that's worth it. Note that the OMG CAM model does have an access control attribute. Perhaps we can use that. However, I doubt it allows distinguishing copy from access. The alternative is to use the "hexBinary" type for this. In that case we need to express the size in the DFDL annotation: <element name="ignoreMe" type="hexBinary" dfdl:repLength="1234" dfdl:repType="binary"/> I can think of one advantage of hexBinary over the occuring bytes approach, which is suppose you do want to use DFDL in the obvious way to convert data into XML format. Never mind that DFDL is supposed to enable avoiding this, suppose it's what you want to do. Then my above byte array for the "ignoreMe" element ends up as: <ignoreMe>0</ignoreMe><ignoreMe>0</ignoreMe><ignoreMe>0</ignoreMe><ignoreMe>0</ignoreMe><ignoreMe>0</ignoreMe><ignoreMe>0</ignoreMe>....<ignoreMe>0</ignoreMe> Which is big compared to: <ignoreMe>000000000000...00</ignoreMe> which is what we'd get if we allow hexBinary as a type. Note that if we add the hexBinary type, you'll still be able to do it the other way, so the hexBinary notion is not strictly speaking necessary or minimalist. ...mikeb Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA Mike Beckerle/Worcester/IBM@IBMUS Sent by: To owner-dfdl-wg(a)ggf.org "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> cc 09/02/2005 04:34 PM dfdl-wg(a)gridforum.org, owner-dfdl-wg(a)ggf.org Subject split into multiple topics - Re: [dfdl-wg] Issues: additional data types I'd like to split this topic into several distinct ones: Arrays - I have a placeholder for this in the doc. Opaque and "code" types are separate. This is related also to the concept of "open content". Enums Bitfields Pointers Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> Sent by: owner-dfdl-wg(a)ggf.org To dfdl-wg(a)gridforum.org 09/02/2005 03:13 PM cc Subject [dfdl-wg] Issues: additional data types Greetings, Here is an "issue" for the DFDL: additional data types that should be considered. Please see attached. --- Robert E. McGrath National Center for Supercomputing Applications University of Illinois, Urbana-Champaign Champaign, Illinois 61820 (217)-333-6549 mcgrath(a)ncsa.uiuc.edu (See attached file: DT.htm)

1 0

Telcon Summary 2005-09-07
by Martin Westhead 07 Sep '05

07 Sep '05

Present ------- Mike Bekerle Robert McGrath Steve Hanson Tara Talbott Martin Westhead Geoff Judd Plans for F2F at GGF -------------------- - we should get the schedule for the DFDL sessions at GGF15 at end of this week. - we'll schedule DFDL WG F2F sessions around these time slots on the other days of the conference. - we have booked a conf. room (seats 10) at an IBM facility a short distance (walkable, quick cab ride) from the GGF15 event for DFDL WG F2F discussions on the other days. (I have all 3 days held now.) Trackers vs. issues list ------------------------ Discussed value of trackers against the issues list. It was thought that trackers should be used for tracking large scale actions, the list for small grained issues. Email vs. Forums ---------------- Some are still unhappy about the difficulty of reading unthreaded discussions. However, notification on Grid Forge is still broken on the Forums. The group was unable to come up with anything better than email. We will put pressure, GGF to fix this and progress with email for now. IBM will also look at using alternative mail readers that with display threads. Choosing topics/issues for discussion ------------------------------------- Ideally topics will be chosen in the previous meeting. However, for now an item by item review of the issues list seems to be our current process. Issues list review ------------------ Reviewed items 1-4. Set up new trackers: - Arrays [Bob McGrath] - Scoping (including Selector) [Mike Bekerle] - Layers [Tara Talbott] All to report by Face to Face Oct 3rd. Notes on trackers: - The assignments are not as above. I can only assign projects to people who are members of DFDL. To get membership you have to go to the DFDL page and click on "Request membership" in the top right. Can Tara and Bob, and anyone else who might lead a team do this please? - I made a mistake and submitted the Layers item twice. I have not worked out how to delete it so I moved it to "Dead items" tracker...

1 0

Fw: [dfdl-wg] Issues List
by Steve Hanson 07 Sep '05

07 Sep '05

The tracker system looks to be exactly what we want. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 07/09/2005 09:04 ----- Martin Westhead <martinwesthead@y ahoo.co.uk> To Sent by: Martin Westhead owner-dfdl-wg@ggf <martinwesthead(a)yahoo.co.uk> .org cc dfdl-wg(a)gridforum.org Subject 07/09/2005 07:32 Re: [dfdl-wg] Issues List Martin Westhead wrote: > - This is an excel file generated with Open Office please let me know > if there are problems with the document. It turns out this does not work well in Excel. I have replaced it with an HTML version that can be found here: https://forge.gridforum.org/projects/dfdl-wg/document/Issues_List/en/1 Let me know what you think: - is this too fine grained - should we be using trackers instead (e.g. see: https://forge.gridforum.org/projects/dais-wg) Cheers, Martin

1 0

Discussing DFDL spec issues via email
by Steve Hanson 07 Sep '05

07 Sep '05

I don't know what others think, but this is not working for me. My inbox is a sea of mail, some of which are replies with history, some of which are not; some have replies in-line, some have not. We definitely need an alternative place to post discussions. If the forum notification system does not work properly then maybe we need to get into the habit of checking the forums regularly? Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

Issues List
by Martin Westhead 07 Sep '05

07 Sep '05

Hi Folks, I have published an issues list at: https://forge.gridforum.org/projects/dfdl-wg/document/Issues_List/en/1 There are currently 69 issues taken from the draft IBM spec, the comments from IBM Hursley and Robert McGrath's recent email. This was (by necessity) a rough cut n' paste exercise so there will be bugs. Please take a look. I'd like to have a brief discussion about whether this is the correct format/approach etc. tomorrow. No doubt there will be some rationalization of the issues required please could those who generated them look them over and suggest modifications/duplicate issues etc. Notes: - At this point I have ignored issues associated with the appendices of the draft. - This is an excel file generated with Open Office please let me know if there are problems with the document. Cheers, Martin

1 1

Telcon today
by Martin Westhead 07 Sep '05

07 Sep '05

Sorry this did not appear yesterday I sent it out but I sent it from the wrong address and it bounced. Preliminary Agenda ------------------ - Plans for F2F at GGF - Issues list review comments/suggestions - Choosing topics/issues for discussion - Additional types (Bob's email) Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call.

1 0

Re: [dfdl-wg] Issues: additional data types
by Martin Westhead 07 Sep '05

07 Sep '05

Hi Robert, As I said in my other reply to you, one of the features that we decided upon with DFDL was that the model that we were translating into was the XML data model. So any file described by DFDL should be translatable into a well formed XML document. The XML data model may not be ideal but it is the standard data model. The reason that the types you describe are not in DFDL is that they are not in XML. That is not to say they are not important or should not be addressed. My opinion on these is that they can be built out of the existing DFDL/XML components and that this is the correct way of handle them. The standard should provide a document that describes one or more ways in which these types can be achieved. More inline... Robert E. McGrath wrote: ..snip... > *1. Enum* > > This type has a set of <name, value> pairs, e.g., <�Red�, 0>, <�Blue�, > 1>, etc. The values are stored in the data, with the name-value pairs > stored in metadata. > > > > Note: one use is for localization, using different maps to give > localized strings. > > > > *Difficulty*: Low > > *Priority:*�� Low I'm confused about what you want to achieve. If you only store the value (an integer) what is the function of the name, is it just for human beings reading the file, or is there some way it is used programatically? One way to approach this type would be a choice over a series of tags with appropriate aattribute types constrained to hardwired values: <enum name="Red" value="0"/> (Complex type with two attributes, one a string constrained to the single value "Red" the other an integer constrained to the value 0). You could then use the DFDL annotations to ensure that this tag gets picked when zero occurs in the file. > *2. Opaque (tagged)* > > This is some kind of non-numeric bit string, with a length and some kind > of tag. > > This might be used, for example, for 1024-bit encryption keys.� The type > means �just pass through the bits�. > > Generally, can be used to store any kind of �blob�, which can be objects > that are meaningful to specific software. > > > > This can be simulated with unsigned integers, but it may be useful to > know that it is not really an integer, or whatever. > > > > *Difficulty*: Low > > *Priority:*�� Low This is just a sequence of bytes... it may need to be hidden (the layering introduces a requirement for it to be possible to be explicit about what is visible at a particular layer). (I don't know what the current favourite way to do this is). > *3. �Code�* > > > > How should �code� be marked up?� It is usually stored in blobs, but it > needs a tag so you know how to interpret it. > > > > This is actually a special case of �opaque�. > > > > *Difficulty*: Low > > *Priority:*�� Low I don't understand what you mean. > > *4. Bitfield / packed* > > > > This type is bits packed into bytes. > > > > *Difficulty*: Low > > *Priority:*�� Low Agreed. I think we can do this but it should be easier. > *5. Pointer� * > > > > Many times there will be pointers within the data, e.g., to offsets in > the file, or to indexes in an array.� This will be critical for storing > objects such as lists or trees. > > > > URL�s� and XPATHS are not especially well suited for this. > > > > This can be simulated with unsigned integers, but they need to be > �swizzled� when translating, so they need to be tagged. > > > > Note that there might be several types of addressing within the data: > > � Offset from zero > > � Offset relative to �foo� > > The offsets might be in different increments:� bits, bytes, words, > elements, etc. > > > > There could be multi-part addresses, e.g., page + offset in page. > > > > *Difficulty*:�� Medium > > *Priority: *��High I spent a long time thinking about pointers at one time. I was unable to come up with anything I felt covered the bases. Perhaps you have enough experience with pointer representations to help us out here. An interesting problem that comes to mind though is what is the XML representation of the pointer value. If it is a tag like: <pointer offset="20" offsetType="bytes" index="5" indexType="float32"/> then all that is needed is to define the metadata conventions that allow that to be correctly interpreted. This is a little unsatisfactory though... > *6. Array* > > > > This is a critical type, must be supported. > > > > There are a lot of issues. > > > > I am preparing a separate memo. > > > > *Difficulty*:�� High > > *Priority: *��Very High We have talked a lot about arrays. A big issue that there are several ways you may want to represent an array within your XML data model. IMO the right way will depend on how you want to use the data. I think the right way to do this is to have a series of recipes for users to capture array semantics in their DFDL files. Cheers, Martin

3 3

Re: [dfdl-wg] Issues: additional data types
by Robert E. McGrath 06 Sep '05

06 Sep '05

On Tuesday 06 September 2005 15:40, Mike Beckerle wrote: > I don't draw a distinction between a sequence of N bytes and an N-byte > represenational-size object of undetermined type. Or rather, the > difference is just of the intent of the person using it. The key role for DFDL is to convey the intent of the producer to the ultimate consumers (without knowing who may consume) > Do you want a type where it is simply an error to access individual bytes? > I.e., so we can express the length in bytes but without expressing that > you should/can access it byte by byte? This is exactly the point of the blob. The tag means "if you don't know what this tag means, don't touch this blob." The case I'm concerned with is a generic grid service that will do something to an array of bytes. E.g., make a JPEG, or run a text mining algorithm. It needs to know the difference between "array of bytes" and "blob".

1 0

split into multiple topics - Re: [dfdl-wg] Issues: additional data types
by Mike Beckerle 06 Sep '05

06 Sep '05

I'd like to split this topic into several distinct ones: Arrays - I have a placeholder for this in the doc. Opaque and "code" types are separate. This is related also to the concept of "open content". Enums Bitfields Pointers Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> Sent by: owner-dfdl-wg(a)ggf.org 09/02/2005 03:13 PM To dfdl-wg(a)gridforum.org cc Subject [dfdl-wg] Issues: additional data types Greetings, Here is an "issue" for the DFDL: additional data types that should be considered. Please see attached. --- Robert E. McGrath National Center for Supercomputing Applications University of Illinois, Urbana-Champaign Champaign, Illinois 61820 (217)-333-6549 mcgrath(a)ncsa.uiuc.edu

4 14