- dfdl-wg - lists.ogf.org

Comments on draft 32 of DFDL spec
by RPost 10 Jun '08

10 Jun '08

Hi, I have been performing ETL since the early '80s when there were over 20 different floppy disk formats and we had to write products like Uniform to copy data from one format to another. At MicroPro Int'l (of WordStar fame) I was also involved in rewriting domestic versions of software to support Shift-JIS for display and input for the Japanese market. More recently I have written and supported ETL software for the telecommunications sector, using ATIS XML formats) and for banking which uses Automated Clearing House (ACH) XML standards. ACH uses a lot of files with a format: FileHeader, (BatchHeader, Detail+, BatchTrailer)+, FileTrailer. As you can imagine I have seen a lot of duplication of effort due to the lack of a standard way to define even the simplest of data formats, let alone the complex ones. Hence my interest in DFDL, which started with Defuddle. I have started to read the recent archives to get a sense of where the DFDL project stands now compared to where it was in 2003-2005 when the great DFDL ice-age began and everyone's projects (Defuddle, Virtual XML) froze in their tracks. Also, I have reviewed the recent Core-032.2 document, added comments to it and have a few general questions about the current state of all things DFDL. 1. Re Parsing (input) only - How complete is the current draft spec in terms of being able to create schemas and a parser for reading binary files? Would it support the most common delimited and header/detail/trailer types of files? >From my limited exposure to the drafts and emails, and my early use of Defuddle, it certainly seems like the parsing part is nearly complete and ready to have implementations created. 2. Will a conforming DFDL processor be required to support both parsing and unparsing? I have only needed the parsing direction for most of the ETL work I have been involved with in the last several years.Each company I worked for had their own custom file readers and parsers. Thus I am very interested in having a product like Defuddle that can read/parse the basics. 3. Can someone clarify the extent, if any, to which DFDL is expected to be used to validate data content as opposed to data structure. This isn't at all clear anywhere in the spec that I could find. I added a comment suggesting a statement in either the 'What DFDL is' or the 'What DFDL is not' section about this. My assumption is that if a DFDL schema is used to unparse an infoset the only guarantee is that the resulting physical structure will be correct and not the logical structure. Consider an example of a filed with records having two date fields: start_date, end_date. There are two input (parsing) operations that are potentially useful: 1. Physical - Read the date values into internal elements (or XML elements) and validate the presence/absence/nullable state of each 2. Business/Logical - Validate that the end_date is either null, or if not null is greater than or equal to the start_date. Naturally DFDL will support #1 but does it support #2? I wouldn't think so. The ETL work I do might not even know what the business rule is and even when we do we always have to deal with 'dirty' data. There are two corresponding output (unparsing) operations of interest: 1. Physical - Write the date values in the proper external physical format. 2. Business/Logical - validate that the date values in the infoset that are to be written meet the business rule stated above. Again, I would expect DFDL to support #1 but not #2. I suggest that some comment or description be added to the spec to make clear the extent to which DFDL supports the business/logical aspect of the data. My concern is that users will be misled into thinking they can arbitrarily populate an infoset and then, using a DFDL schema, create an external file that can be properly used by a native application. It's one thing to 'roundtrip' data that is sourced from a native application and quite another to produce valid native files using a non-native application. Unless, of course you have ideas about branching out to BRDL - Business Rule Definition Language? Rick Post

2 1

Alan cannot make call this week
by Alan Powell 02 Jun '08

02 Jun '08

I am away for the rest of this week so cannot make the call Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

Re: [DFDL-WG] available of DFDL use cases - sample data being used to refine spec
by Mike Beckerle 02 Jun '08

02 Jun '08

Actually we have a test rig but few test cases and they are getting stale. We have an XML test file format which lets one create self contained test files ie you can put some binary data as a hex dump, text data, or even binary bit dumps, along with the XML equivalent of it (the natural xml peojection of the data) as XML data. The inverse for output as well. The test file references a DFDL schema so that the schema is exactly as it would be for "real" use. I'll find the links. ...mikeb On May 31, 2008, at 9:08 PM, "RPost" <rp0428(a)pacbell.net> wrote: > Hi, > > > > I finally managed to connect to the DFDL site long enough to > download a copy of ogf-dfdl-v1.0-Core-030.doc, the draft 30 doc. > > > > In reviewing the doc and recent emails at the archives, http://www.ogf.org/pipermail/dfdl-wg/ > , there are references to sample data and use cases that are being > used to help refine the specification. > > > > Can I get copies of any use cases and/or specific sample date that > are being used? I would like to start creating a ‘DFDL sample’ > companion doc that would contain a richer set of examples than what > would be appropriate for the specification itself. > > > > For example there was a recent discussion from the archives about > Nulls and Defaults (Tue Apr 8 19:31:52 CDT 2008). Parts of the > discussion regard what properties must be specified and what the > allowable property/data combinations are. > > > > I would like to create actual sample data for some of these cases > (e.g. required field that is missing but has a default) and perhaps > a truth table that shows explicitly what combination of properties > must be specified to be valid. > > > > This doc would not be concerned with exactly how (or where in the > DFDL schema) a specific property is specified but only how the > property/properties can be used for a specific test case. > > > > Such a doc could help bring some clarity to the interaction of the > properties and, at some future time, assist implementers in creating > proper validation test sets. > > > > If someone could forward me some sample use cases, or data scenarios > (e.g. Cobol OCCURS data) and let me know which ones would be most > useful to have some more detailed property examples for I could > start the doc with those. > > > > Thanks for any info. > > > > > > Rick > > > > p.s. The link I have been largely unsuccessful using to access the > DFDL site is > > > > http://www.ggf.org/ggf_areas_data.htm > > > > Most of the links from this page, including the DFDL one, give proxy > or maintenance errors. > > > > I have emailed the webmaster about the problem but haven’t heard any > thing back. > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

Re: [DFDL-WG] available of DFDL use cases - sample data being used to refine spec
by RPost 01 Jun '08

01 Jun '08

Hi, I finally managed to connect to the DFDL site long enough to download a copy of ogf-dfdl-v1.0-Core-030.doc, the draft 30 doc. In reviewing the doc and recent emails at the archives, http://www.ogf.org/pipermail/dfdl-wg/, there are references to sample data and use cases that are being used to help refine the specification. Can I get copies of any use cases and/or specific sample date that are being used? I would like to start creating a 'DFDL sample' companion doc that would contain a richer set of examples than what would be appropriate for the specification itself. For example there was a recent discussion from the archives about Nulls and Defaults (Tue Apr 8 19:31:52 CDT 2008). Parts of the discussion regard what properties must be specified and what the allowable property/data combinations are. I would like to create actual sample data for some of these cases (e.g. required field that is missing but has a default) and perhaps a truth table that shows explicitly what combination of properties must be specified to be valid. This doc would not be concerned with exactly how (or where in the DFDL schema) a specific property is specified but only how the property/properties can be used for a specific test case. Such a doc could help bring some clarity to the interaction of the properties and, at some future time, assist implementers in creating proper validation test sets. If someone could forward me some sample use cases, or data scenarios (e.g. Cobol OCCURS data) and let me know which ones would be most useful to have some more detailed property examples for I could start the doc with those. Thanks for any info. Rick p.s. The link I have been largely unsuccessful using to access the DFDL site is http://www.ggf.org/ggf_areas_data.htm Most of the links from this page, including the DFDL one, give proxy or maintenance errors. I have emailed the webmaster about the problem but haven't heard anything back.

1 0

DFDL: New co-chair
by Mike Beckerle 31 May '08

31 May '08

I'd like to nominate Steve Hanson to be co-chair of this group. Anyone with either objections or statements of support, please write to me directly. Mike Beckerle DFDL co-chair

1 0

DFDL: Minutes from OGF WG call, 28 May 2008
by Ian W Parkinson 30 May '08

30 May '08

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 16:00 GMT, 28 May 2008 Attendees Mike Beckerle (Oco) Ian Parkinson (IBM) Alan Powell (IBM) 1. Next specification drafts Mike and Alan have independently made small changes to the current specification draft, and there are many changes to be made following last week's face-to-face meeting in Hursley. Alan will merge his changes with Mike's changes to form draft 32, and will also produce a 'point release' of this draft having removed all of the Microsoft Word comments and change annotations. This point release will be used as a basis for draft 33, which will contain changes from the face-to-face meeting. Mike and Alan will divide the work necessary to achieve this among the team; though the meeting covered some of these items: Alan will make 'simple' changes from the WTX list of issues Alan will document speculative parsing, and add a new section covering selectors We will ask Steve to produce necessary changes for variable markup The meeting recapped the conclusions regarding forward and backward references - arbitrary forward and backward references are allowed during unparse, but only arbitrary backward references are allowed on parse. Limited forward references will be allowed on parse, but no definition of the limit has yet been made. Alan will write this up. Mike will author an example showing the comination of choice and valueCalc. While thinking about this example, he has concluded that discriminators will need to be evaluated during unparse, as well as parse. He will discuss this during the next working group call. Mike will also author a new section in response to comments from Sandy Gao and Suman Kalia, explicitly connecting syntax with DFDL semantics. Meeting closed, 16:45 GMT Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL slides from f2f meeting - Hursley UK
by Mike Beckerle 21 May '08

21 May '08

Attached are the results from the F2F in Hursley. I also have captured images of the flip charts, and I'll send those along in addition shortly. .mike

1 0

DFDL: Calculated Values section rewrite
by Mike Beckerle 15 May '08

15 May '08

My deliverable for the next draft (32) was to revise the calculated values section. I have rewritten it and the draft is attached. Of note: I have removed the troublesome "outputLengthCalc" property, as I no longer see a critical need for it. One of the examples computes the number of occurrences for a byte array, and I think that mechanism along with alignment is sufficient to handle the troublesome cases I was considering where size of padding had to be dynamically computed. The section now consists of a short table of definition, followed by illustrative examples, however, I think this is fine. The semantics of inputValueCalc and outputValueCalc aren't really that complex. They can be described in a paragraph. It is the motivation for them that is complicated, so I think examples in the spec, while sometimes considered problematic, are OK in this case.

3 5

DFDL: Time change for weekly calls?
by Mike Beckerle 15 May '08

15 May '08

I believe all active participants are either ET.US time, or UK time. We chose Noon ET.US because we used to have active participants from PT.US, but currently we don't, so I for one would like to free up Wednesday lunch time. I suggest moving back either 1 hour, or 90 minutes to either 11am ET.US 4pm UK, or 10:30am ET.US, 3:30pm UK. Any day of the week is acceptable to me. Keeping it Wednesday would be my preference. Comments on feasibility? .mikeb

2 1

DFDL: Agenda for DFDL WG call - 2008-05-14
by Mike Beckerle 14 May '08

14 May '08

Proposed agenda for today's call. Please provide additional topics. - Draft 32 status - Unparsing property precedence (left over from agenda of last meeting) - plans for meetings in Hursley next week o topics and prep needed o logistics - Mike to go to what building, and ask for whom, etc.

1 1