- dfdl-wg - lists.ogf.org

example of MS Biztalk dfdl-like annotations
by Mike Beckerle 14 Jul '05

14 Jul '05

I found this link to the Microsoft biztalk style of annotations. Thought some of you might like to surf it. There's no news here. Just indicates a style. http://www.winterdom.com/dev/bts/BTSFFSchemaTut2.html

2 2

Layering in DFDL (for our FAQ and docs)
by Mike Beckerle 12 Jul '05

12 Jul '05

I had action items from GGF14 to begin building up an FAQ. Here's something about layering. ------------------------- Layering in DFDL The book "ASN.1 Complete" by Larmouth (ISBN 0-12-233435-3 and available online as a pdf) discusses the importance of layer support in format descriptions. (Begin Excerpt) The layering concept is perhaps most commonly associated with the International Standards Organization (ISO) and International Telecommunications Union (ITU) "architecture" or "7-layer model" for Open Systems Interconnection (OSI) shown in Figure 3. While many of the protocols developed within this framework are not greatly used today, it remains an interesting academic study for approaches to protocol specification. In the original OSI concept in the late 1970s, there would be just 6 layers providing (progressively richer) carrier services, with a final "application layer" where each specification supported a single endapplication, with no "holes". It became apparent, however, over the next decade, that even in the "application layer" people wanted to leave "holes" in their specification for later extensions, or to provide a means of tailoring their protocol to specific needs. For example, one of the more recent and important protocols - Secure Electronic Transactions (SET) - contains a wealth of fully-defined message semantics, but also provides for a number of "holes" which can transfer "merchant details" which are not specified in the SET specification itself. So we have basic messages for purchase requests and responses, inquiry requests and responses, authorization requests and responses, and so on, but within those messages there are ?holes? for ?message extensions? - additional information specific to a particular merchant. It is thus important that any mechanism or notation for specifying a protocol should be able to cater well for the inclusion of "holes". This has been one of the more important developments in ASN.1 in the last decade, and will be a subject of much further discussion in this book. "Catering well" for the inclusion of "holes" implies that the notation must have defined mechanisms (preferably uniformly applied to all specifications written using that notation) to identify the contents of a hole at communications time. (In lower layers, this is sometimes referred to as the "protocol id" problem). Equally important, however, are notational means to clearly identify that a specification is incomplete (contains a hole), together with well-defined mechanisms to relate the (perhaps later in time) specification of the contents of holes to the location of the holes themselves. (End Excerpt) The argument made here is equally true for DFDL. We need the ability to describe a data format containing a hole or payload which another DFDL schema can then describe the format of. DFDL actually encounters some data formats where there are discontiguous holes. Consider the nonVSAM VS format. (see "IBM OS/390 DFSMS: Using Data Sets" IBM publication SC26-7339-01, Second Edition, December 2000. (online at: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/DGT1D411/CCONTEN TS?SHELF=EZ239126&DN=SC26-7339-01&DT=20001014144419) In this format, data records with actual data fields of interest are broken up into segments. The segments are of variable size, and a record can fit in a single segment or can span multiple segments. The segments are of 3 types, initial, middle (there can be zero or more of these), and final. The hole that the record fits in is assembled by putting together the partial holes from each of the segments. Adding minor additional complexity is that in the actual format the segments are then grouped into variable-sized blocks as an I/O transfer-unit efficiency optimization. A further wrinkle on holes in DFDL is the notion of encoding. Modern data formats often contain holes (or we'll also call them payloads) which have been encoded to allow data transfer in text-only mediums, or to compress to save space, or to encrypt, or for various other reasons. The encoding must be decoded and the resulting data is the payload where we then want to describe the format. There are many examples of this, but email messages using MIME encapsulated attachments are a classic example. We'd like to describe a file of email messages each containing MIME encapsulated attachments where the attachments are compressed binary data where the data is a binary data format. We'd like to describe this file and expose the logical structure of the data that is inside the MIME encapsulated attachments.

1 0

posting to our forum
by Mike Beckerle 12 Jul '05

12 Jul '05

I posted an article on layering to our DFDL forum on gridforge. Try out the forum on grid forge to see if you can find it, or if you tried to setup monitoring, then you should get a message from gridforge notifying you of the posting (if you don't it's not working!) The posting is related to an FAQ-topic that came up at GGF14. (I.e., one of the pieces of material that we need to accumulate for a DFDL FAQ) Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 1

Bit/byte order
by Martin Westhead 11 Jul '05

11 Jul '05

Hi all, This is a discussion topic that came up in last weeks telcon. Mike asserted that only byte order was needed. I was not completely comfortable with this and said I'd investigate. After talking it over with various folks and looking a couple of things up, I have reached the conclusion that I agree with Mike. The key observation is that microprocessors handle bytes in parallel, there is no "bit pointer". So, within the context of data on a machine the only ordering over bits is their with respect to significance. Hence, to all intents and purposes, as Mike observes, bit-order==byte-order. It is the case that hardware representations of binary data will have a particular bit-ordering - Ethernet for example is LSB first, representations of data on tapes (or disks) will have a bit ordering. However the conclusion I reached from discussions here was that this is a red-herring for DFDL. Bit-order in this context is one of a number of physical aspects of the data representation which would include frequencies, signal strengths etc. There may be some archival value in recording all that stuff, and you could put it in DFDL, but I think that stuff is safely out of scope for where we are now. As soon as you are looking at your data from software you have bytes. Cheers, Martin

1 0

Nomination of Jim Myers to replace Alan Chappell as co-chair
by Mike Beckerle 06 Jul '05

06 Jul '05

Folks, Alan Chappell has stepped down as co-chair of the DFDL-WG. Jim Myers has been nominated to take his place. Please send any comments on this nomination (e.g., in favor, opposed, don't care) to myself and Martin Westhead, (co-chair) directly. Martin and my emails are on the CC of this email message. Thanks Mike Beckerle

1 0

new forum created
by Mike Beckerle 06 Jul '05

06 Jul '05

I created a "DFDL General" forum for structured discussion. The forum is here: https://forge.gridforum.org/forum/forum.php?forum_id=547 If you are like me you want to receive an email (or email digest) about activity in the forum. To do this you set your monitoring preferences here: https://forge.gridforum.org/monitoring/monitorPreference.php (dont' forget to login first) ...mikeb Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 0

Today's call
by Martin Westhead 06 Jul '05

06 Jul '05

I will be on the call today. Could someone send me the number and access details though - I don't have them here. Thanks, Martin

1 0

Call-in information : There are two GGF DFDL Sessions on Thursday ( 9am and 11am Chicago time)
by Mike Beckerle 06 Jul '05

06 Jul '05

There are two GGF DFDL Sessions on Thursday ( 9am and 11am Chicago time) Please call into this number at 9:00am Chicago on Thursday .... 1-877-950-2442 or 1-203-480-6948 - passcode 575542

2 2

materials for possible discussion during session 2
by Mike Beckerle 30 Jun '05

30 Jun '05

Two slides about DFDL I presented at the Data area summary - Session 1 pre-slides - before the presentation. Session 2 doesn't really have any slides (just a title slide) This represents work on the annotation syntax/model. We took a clean-slate approach to a model for the rep properties with an eye towards enabling annotations "something like" the OMG CAM-model instances approach. Very rough work, but since we might be able to have a conference call today I thought I'd forward these for consideration. Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

1 0

uploaded to our gridforge page
by Mike Beckerle 22 Jun '05

22 Jun '05

I've uploaded notes and slides from the F2F, and as promised a zip of the examples we walked through from the Ascl (now IBM) prototype my group is working on. In this zip is testCase.xsd, which is the schema of the test-case files that we're using. These are a common format that we could all share for tests of a variety of implementations. http://forge.gridforum.org /projects/dfdl-wg/document/F2F_2005_05_Examples_and_XSDs/en/1. is the URL for the zip file. Sorry for the delay in getting this all up on gridforge. Mike Beckerle Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA

2 2