- dfdl-wg - lists.ogf.org

Agenda for tomorrow's telcon
by Martin Westhead 17 Aug '05

17 Aug '05

Hi folks, Just collecting items for tomorrow's telcon, so far I have: - Update on IBM draft doc (will someone be on who can do this?) - Primer update, Jim Meyers Any other offers? Thanks, Martin

3 2

FW: Data files for DFDL
by Talbott, Tara D 04 Aug '05

04 Aug '05

Here are a few of the sample data files that we have worked on and their xml translations. I believe that these are examples of files where it would be difficult to achieve the desired translation without some form of layering. premix.bin - a binary file from the chemkin program. Contains three different sections, determined by a header value. In the solution section there is a sequence of x values, the values were combined with a sequence of points containing a temp value and massFrac values. test.prp - a text-based file based on a nwChem format. Contains some header info, then a list of field names, then a series of frames with values that should correspond with the field names. When translating, the list of field names is read as a layer and later referenced from the frames. clayers.bin - same as example of layers shown at last f2f, creates a sequence of 100 complex numbers from a file containing a sequence of 100 real floats then 100 imaginary floats. Tara > The raw data files and their xml translations > <<premix.bin>> <<chemkinbin.xml>> > <<test.prp>> <<prp.xml>> > <<clayers.bin>> <<clayer.xml>>

1 0

today's call - nobody on
by Mike Beckerle 20 Jul '05

20 Jul '05

ok, so it's summer time and the livin's easy. I was on the call, nobody else came on. Quick status of IBM Westborough DFDL project: 1) we have data-source indirection kicking over in our prototype. We're pretty happy with how this is coming out. The feature seems simple enough to explain and the implementation is of acceptable complexity. We're still pushing to have a proposal for the group by end of july. 2) we are working on a proposal for annotation syntax which will reconcile differing viewpoints within IBM. Still on track for a proposal to the broader community by end of july. ...mikeb

1 0

example of MS Biztalk dfdl-like annotations
by Mike Beckerle 14 Jul '05

14 Jul '05

I found this link to the Microsoft biztalk style of annotations. Thought some of you might like to surf it. There's no news here. Just indicates a style. http://www.winterdom.com/dev/bts/BTSFFSchemaTut2.html

2 2

Layering in DFDL (for our FAQ and docs)
by Mike Beckerle 12 Jul '05

12 Jul '05

I had action items from GGF14 to begin building up an FAQ. Here's something about layering. ------------------------- Layering in DFDL The book "ASN.1 Complete" by Larmouth (ISBN 0-12-233435-3 and available online as a pdf) discusses the importance of layer support in format descriptions. (Begin Excerpt) The layering concept is perhaps most commonly associated with the International Standards Organization (ISO) and International Telecommunications Union (ITU) "architecture" or "7-layer model" for Open Systems Interconnection (OSI) shown in Figure 3. While many of the protocols developed within this framework are not greatly used today, it remains an interesting academic study for approaches to protocol specification. In the original OSI concept in the late 1970s, there would be just 6 layers providing (progressively richer) carrier services, with a final "application layer" where each specification supported a single endapplication, with no "holes". It became apparent, however, over the next decade, that even in the "application layer" people wanted to leave "holes" in their specification for later extensions, or to provide a means of tailoring their protocol to specific needs. For example, one of the more recent and important protocols - Secure Electronic Transactions (SET) - contains a wealth of fully-defined message semantics, but also provides for a number of "holes" which can transfer "merchant details" which are not specified in the SET specification itself. So we have basic messages for purchase requests and responses, inquiry requests and responses, authorization requests and responses, and so on, but within those messages there are ?holes? for ?message extensions? - additional information specific to a particular merchant. It is thus important that any mechanism or notation for specifying a protocol should be able to cater well for the inclusion of "holes". This has been one of the more important developments in ASN.1 in the last decade, and will be a subject of much further discussion in this book. "Catering well" for the inclusion of "holes" implies that the notation must have defined mechanisms (preferably uniformly applied to all specifications written using that notation) to identify the contents of a hole at communications time. (In lower layers, this is sometimes referred to as the "protocol id" problem). Equally important, however, are notational means to clearly identify that a specification is incomplete (contains a hole), together with well-defined mechanisms to relate the (perhaps later in time) specification of the contents of holes to the location of the holes themselves. (End Excerpt) The argument made here is equally true for DFDL. We need the ability to describe a data format containing a hole or payload which another DFDL schema can then describe the format of. DFDL actually encounters some data formats where there are discontiguous holes. Consider the nonVSAM VS format. (see "IBM OS/390 DFSMS: Using Data Sets" IBM publication SC26-7339-01, Second Edition, December 2000. (online at: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/DGT1D411/CCONTEN TS?SHELF=EZ239126&DN=SC26-7339-01&DT=20001014144419) In this format, data records with actual data fields of interest are broken up into segments. The segments are of variable size, and a record can fit in a single segment or can span multiple segments. The segments are of 3 types, initial, middle (there can be zero or more of these), and final. The hole that the record fits in is assembled by putting together the partial holes from each of the segments. Adding minor additional complexity is that in the actual format the segments are then grouped into variable-sized blocks as an I/O transfer-unit efficiency optimization. A further wrinkle on holes in DFDL is the notion of encoding. Modern data formats often contain holes (or we'll also call them payloads) which have been encoded to allow data transfer in text-only mediums, or to compress to save space, or to encrypt, or for various other reasons. The encoding must be decoded and the resulting data is the payload where we then want to describe the format. There are many examples of this, but email messages using MIME encapsulated attachments are a classic example. We'd like to describe a file of email messages each containing MIME encapsulated attachments where the attachments are compressed binary data where the data is a binary data format. We'd like to describe this file and expose the logical structure of the data that is inside the MIME encapsulated attachments.

1 0

posting to our forum
by Mike Beckerle 12 Jul '05

12 Jul '05

I posted an article on layering to our DFDL forum on gridforge. Try out the forum on grid forge to see if you can find it, or if you tried to setup monitoring, then you should get a message from gridforge notifying you of the posting (if you don't it's not working!) The posting is related to an FAQ-topic that came up at GGF14. (I.e., one of the pieces of material that we need to accumulate for a DFDL FAQ) Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 1

Bit/byte order
by Martin Westhead 11 Jul '05

11 Jul '05

Hi all, This is a discussion topic that came up in last weeks telcon. Mike asserted that only byte order was needed. I was not completely comfortable with this and said I'd investigate. After talking it over with various folks and looking a couple of things up, I have reached the conclusion that I agree with Mike. The key observation is that microprocessors handle bytes in parallel, there is no "bit pointer". So, within the context of data on a machine the only ordering over bits is their with respect to significance. Hence, to all intents and purposes, as Mike observes, bit-order==byte-order. It is the case that hardware representations of binary data will have a particular bit-ordering - Ethernet for example is LSB first, representations of data on tapes (or disks) will have a bit ordering. However the conclusion I reached from discussions here was that this is a red-herring for DFDL. Bit-order in this context is one of a number of physical aspects of the data representation which would include frequencies, signal strengths etc. There may be some archival value in recording all that stuff, and you could put it in DFDL, but I think that stuff is safely out of scope for where we are now. As soon as you are looking at your data from software you have bytes. Cheers, Martin

1 0

Nomination of Jim Myers to replace Alan Chappell as co-chair
by Mike Beckerle 06 Jul '05

06 Jul '05

Folks, Alan Chappell has stepped down as co-chair of the DFDL-WG. Jim Myers has been nominated to take his place. Please send any comments on this nomination (e.g., in favor, opposed, don't care) to myself and Martin Westhead, (co-chair) directly. Martin and my emails are on the CC of this email message. Thanks Mike Beckerle

1 0

new forum created
by Mike Beckerle 06 Jul '05

06 Jul '05

I created a "DFDL General" forum for structured discussion. The forum is here: https://forge.gridforum.org/forum/forum.php?forum_id=547 If you are like me you want to receive an email (or email digest) about activity in the forum. To do this you set your monitoring preferences here: https://forge.gridforum.org/monitoring/monitorPreference.php (dont' forget to login first) ...mikeb Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 0

Today's call
by Martin Westhead 06 Jul '05

06 Jul '05

I will be on the call today. Could someone send me the number and access details though - I don't have them here. Thanks, Martin

1 0