June 2008 - dfdl-wg - lists.ogf.org

XSL-style SmartLengthString Sample
by RPost 18 Jun '08

18 Jun '08

Hi, Attached is a doc that describes a (syntax-deprived) version of Mike's SmartLengthString example that is done more in a style that combines XML and XSL. It would use XML for the visible infoset and element declarations but an XSL style for the transforms. It attempts to: 1. Separate input/parse considerations from output/unparse. 2. Provide for all physical elements without requiring them to be in the visible infoset 3. Provide for multiple/nested layers 4. Allow for multiple transforms to be performed even within a single layer. I still have scalability concerns with using one outputValueCalc (or input) expression if multiple or complex transformations need to be done. The idea is to use a simple structure that can be nested to get the complexity that is needed. Here is an excerpt from the doc but the formatting will probably get out of whack: <xs:complexType name="smartLengthString" ...other attributes...> <xs:sequence> <hiddenStructure> -- the annotations and transforms between the physical and logical layers <hiddenElements> . . . one or more elements . . . </hiddenElements> <inputLayer> . . . The transforms and lower layer elements needed to produce the higher-layer Elements and ultimately the infoset informational values. </inputLayer> <outputLayer> . . . The transforms and higher layer elements needed to produce the lower-layer elements and ultimately the physical layer values. </outputLayer> </hiddenStructure> <visibleElements> -- the infoset as experienced by the user </xs:sequence> </xs:complexType> As you can see I took a lot of liberties with the syntax to try to make it easier to see the structure I am using. The 'hiddenStructure' element is what contains the equivalent of the <xs:annotation><xs:appinfo> from the example. The hidden elements are equivalent to elements in the example that are not in the public infoset. The inputLayer contains whatever is needed (including nested inputLayers) for parsing. Likewise for the outputLayer and unparsing. Here is a skeleton inputLayer: <inputLayer> <layerElements>  </layerElements> <transforms>  <transforms> <inputLayer>  </inputLayer> </inputLayer> There is a section for elements specific to this layer; somewhat akin to intermediate working variables or intermediate calculations. Another section is for an ordered list of transforms. This allows for as many or as few as needed but provides clean separation between them. The first transform might create the value of the 'lengthFlag' hidden element. The next transform could then use the lengthFlag value to extract the logicalLength value and so on. It sure seems to flow ok but maybe that is just in my head. The nested 'inputLayer' allows for a chain of transforms to be created. The nested inputLayer could create values for this layer's layerElements and get it's input from either another nested layer or from the physical layer itself. If the doc does nothing more than help solidify your thinking about how to get where you want to go it will have done it's job. By the way, thanks for the excellent example Mike. Rick

2 1

DFDL: new meeting time
by Mike Beckerle 16 Jun '08

16 Jun '08

Can we officially change the weekly meeting time to 9am Wednesday ET.US (2pm UK Time) We discussed this before, but I am now running into many conflicts with our current meeting time, so it's no longer viable for me. Please chime in on whether this time will work for you. .mikeb Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451 | <mailto:mbeckerle.dfdl@gmail.com> mbeckerle.dfdl(a)gmail.com

3 4

Invitation: OGF DFDL Working Group weekly call (18 Jun 14:00 GDT in DE2K09)
by Steve Hanson 16 Jun '08

16 Jun '08

Calendar Entry ---------------------------------------------------------------------------------------------------------------------------- Subject Chair OGF DFDL Working Group weekly call Steve Hanson/UK/IBM Invitees Date Required (to) 18/06/2008 Alan Powell/UK/IBM, mbeckerle(a)oco-inc.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Time Kimber/UK/IBM 14:00 - 15:00 (1 hour) Optional (cc) dfdl-wg(a)ogf.org Proposed start -1 at Proposed end -1 at Original start 18/06/2008 at 14:00 Original end 18/06/2008 at 15:00 Repeats Show detailed repeat information This meeting is scheduled to repeat Where Location DE2K09 Participant Passcode: 8190865 Country Toll Numbers Freephone/Toll Free Number AUSTRIA 43-1-928-9652 BELGIUM 32-2-400-6791 0800-4-9612 DENMARK 45-7014-0237 FRANCE LYON: 33-4-26-69-12-74 080-511-3725 FRANCE MARSEILLE: 33-4-86-06-00-74 080-511-3725 FRANCE PARIS: 33-1-70-70-81-58 080-511-3725 GERMANY 49-69-2222-20363 0800-100-6504 IRELAND 353-1-247-7768 ITALY 39-02-3600-3648 800-985-679 LUXEMBOURG 352-27-000-1312 NETHERLANDS 31-20-713-2741 0800-020-3237 SPAIN 34-91-414-1542 800-099-667 SWEDEN 46-8-566-10-780 SWITZERLAND 41-44-580-3334 0800-000-516 UNITED KINGDOM BIRMINGHAM: 44-121-210-9014 0800-279-9193 UNITED KINGDOM GLASGOW: 44-141-202-3214 0800-279-9193 UNITED KINGDOM LEEDS: 44-113-301-2114 0800-279-9193 UNITED KINGDOM LONDON: 44-20-7019-0808 0800-279-9193 UNITED KINGDOM MANCHESTER: 44-161-601-1414 0800-279-9193 USA 866-505-4412

1 0

DFDL: gridforge up today
by Mike Beckerle 16 Jun '08

16 Jun '08

Gridforge appears to be alive again. See: http://forge.gridforum.org/sf/docman/do/listDocuments/projects.dfdl-wg/docma n.root.current_0?_sortby=documentList%28dateLastModified%29&_sorder=document List%28asc%29 .mikeb Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451 | <mailto:mbeckerle.dfdl@gmail.com> mbeckerle.dfdl(a)gmail.com

2 1

Gridforge Problems
by Mike Beckerle 11 Jun '08

11 Jun '08

Can we get status on gridforge? We've been unable to use it for some time now. It does seem to come up occasionally because a new group member has been able to download some docs, but we've found it unreliable enough that we can't depend on it. Is it being fixed? Thanks Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451 | <mailto:mbeckerle.dfdl@gmail.com> mbeckerle.dfdl(a)gmail.com

1 0

comments and concerns re choice-discriminator-example
by RPost 11 Jun '08

11 Jun '08

I have some questions and possible concerns re the sample xml file Mike posted. The attached document 'SmartlengthStringComments1 discusses them. I attached the other doc that I found from 2003 written by Martin Westhead. The current sample and an XSLT sample in the old doc bear an uncanny resemblance and thought it might be worthwhile to look at the two side-by-side since they are for the exact same use case. It raises the question of whether DFDL syntax could allow actual XSLT snippets to be included in the annotation section to use the power of XSLT to do these things. XSLT also supports extensions that could handle complex or unusual corner cases. A parser should be able to (but I haven't confirmed it) create a mini XSL stylesheet from the snippet and use a transform to generate the result value that is needed. If so this would allow an existing standard language to be used. See my item #5 in my doc since there I ask the question of how I might contribute without being disruptive. It is not my intention to disrupt your efforts but the resemblance between these two approaches is so similar I had to bring it to your attention. Rick

2 1

Re: [DFDL-WG] Comments on draft 32 of DFDL spec
by JimMyers 10 Jun '08

10 Jun '08

Rick, A quick update from NCSA on Defuddle: after we got things started at PNNL and Tara Talbott created the initial version, we did indeed have an 'ice age' with no direct funding for it. Last year we received a small amount of funding at NCSA (where I moved to) from NARA to start Defuddle moving again and to incorporate it into the EU SHAMAN project's digital preservation architecture (lots of things in SHAMAN but the relevant idea here is the iRODS storage broker calling Defuddle to map things to logical models and the Multivalent Browser viewing the logical model). Due in part to the lack of funding, we've dropped out of the DFDL discussion for a while. Probably the most significant issues where Defuddle doesn't match the draft spec are that I believe the idea of layers has been dropped from the version 1 spec plan (which we think is critical)and there has been a lot of work on the spec to deal with nilability, etc. (some of which I've argued can be avoided if you have layers). In the next year+, we're rebuilding Defuddle on the latest libraries, doing a bunch of scalability and stress testing, targetting some common file formats (.e.g PNG) to show it works beyond the relatively simple scientific formats we've done to date, exploring uses in the digital library/preservation communities and looking to extend from XML modeling to RDF/semantic modeling. The last one of these starts to go in the direction of your questions #2 - if you get to RDF, you can start using OWL/rule constraints to assure that the data coming out/going in is semantically what you want (The DFDL group has discussed the backward direction, but at least for Defuddle it is not yet in our actual development plans). Our intial thought for doing this is to just include a GRRDL annotation in the DFDL file which tells you how to map (via XSLT for example) from the XML created by the current Defuddle to the RDF you want - in essence going from binary to XML and XML to RDF logical model in two sequential steps. We haven't thought as deeply about semantic validation as the DFDL group has about XML-level validation (i.e. the discussion of errors that can be caught at the time of format definition, versus XML schema constraint violations on parsing versus parsing errors themselves, etc.), but I think getting to RDF will allow a lot of what you're talking about with standard semantic web tools. As we get going on Defuddle again, I hope to get reconnected with the DFDL effort - I've mostly been lurking on the list the past year+ - and see how we can help without being disruptive (perhaps looking at post 1.0 changes?). In any case, I wanted to respond to your question and give a 'what's new' report back to the group since I've been quiet a while. For Defuddle, while I still think we need to grow, things are looking up with a very interested sponsor, international collaboration, and some momentum after the 'ice age'. (As always, I'd be very happy to talk with anyone who'd like to get involved in Defuddle (software development or creating format descriptions and using it, etc.) - Defuddle is open source and I know everyone involved to date would really like to see it become community (versus project) driven.) Cheers, Jim James D. Myers, Ph.D. Associate Director, Cyberenvironments National Center for Supercomputing Applications University of Illinois at Urbana Champaign 1205 W Clark St. Urbana, IL 61801 217-244-1934 ----- "RPost" <rp0428(a)pacbell.net> wrote: > Hi, > > > > I have been performing ETL since the early '80s when there were over > 20 different floppy disk formats > > and we had to write products like Uniform to copy data from one format > to another. > > > > At MicroPro Int'l (of WordStar fame) I was also involved in rewriting > domestic versions of software > > to support Shift-JIS for display and input for the Japanese market. > > > > More recently I have written and supported ETL software for the > telecommunications sector, > > using ATIS XML formats) and for banking which uses Automated Clearing > House (ACH) XML standards. > > ACH uses a lot of files with a format: FileHeader, (BatchHeader, > Detail+, BatchTrailer)+, FileTrailer. > > > > As you can imagine I have seen a lot of duplication of effort due to > the lack of a standard way to > > define even the simplest of data formats, let alone the complex ones. > > > > Hence my interest in DFDL, which started with Defuddle. > > > > I have started to read the recent archives to get a sense of where the > DFDL project stands > > now compared to where it was in 2003-2005 when the great DFDL ice-age > began and everyone's > > projects (Defuddle, Virtual XML) froze in their tracks. > > > > Also, I have reviewed the recent Core-032.2 document, added comments > to it and have a few > > general questions about the current state of all things DFDL. > > > > 1. Re Parsing (input) only - How complete is the current draft spec in > terms of being able > > to create schemas and a parser for reading binary files? Would it > support the most common delimited > > and header/detail/trailer types of files? > > > > From my limited exposure to the drafts and emails, and my early use of > Defuddle, it certainly seems like > > the parsing part is nearly complete and ready to have implementations > created. > > > > 2. Will a conforming DFDL processor be required to support both > parsing and unparsing? > > > > I have only needed the parsing direction for most of the ETL work I > have been involved with in > > the last several years.Each company I worked for had their own custom > file readers and parsers. > > Thus I am very interested in having a product like Defuddle that can > read/parse the basics. > > > > 3. Can someone clarify the extent, if any, to which DFDL is expected > to be used to validate data > > content as opposed to data structure. This isn't at all clear anywhere > in the spec that I could find. > > I added a comment suggesting a statement in either the 'What DFDL is' > or the 'What DFDL is not' > > section about this. > > > > My assumption is that if a DFDL schema is used to unparse an infoset > the only guarantee is that the > > resulting physical structure will be correct and not the logical > structure. > > > > Consider an example of a filed with records having two date fields: > start_date, end_date. > > > > There are two input (parsing) operations that are potentially useful: > > > > 1. Physical - Read the date values into internal elements (or XML > elements) and validate the > > presence/absence/nullable state of each > > > > 2. Business/Logical - Validate that the end_date is either null, or if > not null is greater than > > or equal to the start_date. > > > > Naturally DFDL will support #1 but does it support #2? I wouldn't > think so. The ETL work I do might > > not even know what the business rule is and even when we do we always > have to deal with 'dirty' data. > > > > There are two corresponding output (unparsing) operations of interest: > > > > 1. Physical - Write the date values in the proper external physical > format. > > > > 2. Business/Logical - validate that the date values in the infoset > that are to be written meet > > the business rule stated above. > > > > Again, I would expect DFDL to support #1 but not #2. > > > > I suggest that some comment or description be added to the spec to > make clear the extent to which > > DFDL supports the business/logical aspect of the data. > > > > My concern is that users will be misled into thinking they can > arbitrarily populate an infoset and > > then, using a DFDL schema, create an external file that can be > properly used by a native application. > > > > It's one thing to 'roundtrip' data that is sourced from a native > application and quite another to > > produce valid native files using a non-native application. > > > > Unless, of course you have ideas about branching out to BRDL - > Business Rule Definition Language? > > > > Rick Post > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/dfdl-wg

2 1

Comments on draft 32 of DFDL spec
by RPost 10 Jun '08

10 Jun '08

Hi, I have been performing ETL since the early '80s when there were over 20 different floppy disk formats and we had to write products like Uniform to copy data from one format to another. At MicroPro Int'l (of WordStar fame) I was also involved in rewriting domestic versions of software to support Shift-JIS for display and input for the Japanese market. More recently I have written and supported ETL software for the telecommunications sector, using ATIS XML formats) and for banking which uses Automated Clearing House (ACH) XML standards. ACH uses a lot of files with a format: FileHeader, (BatchHeader, Detail+, BatchTrailer)+, FileTrailer. As you can imagine I have seen a lot of duplication of effort due to the lack of a standard way to define even the simplest of data formats, let alone the complex ones. Hence my interest in DFDL, which started with Defuddle. I have started to read the recent archives to get a sense of where the DFDL project stands now compared to where it was in 2003-2005 when the great DFDL ice-age began and everyone's projects (Defuddle, Virtual XML) froze in their tracks. Also, I have reviewed the recent Core-032.2 document, added comments to it and have a few general questions about the current state of all things DFDL. 1. Re Parsing (input) only - How complete is the current draft spec in terms of being able to create schemas and a parser for reading binary files? Would it support the most common delimited and header/detail/trailer types of files? >From my limited exposure to the drafts and emails, and my early use of Defuddle, it certainly seems like the parsing part is nearly complete and ready to have implementations created. 2. Will a conforming DFDL processor be required to support both parsing and unparsing? I have only needed the parsing direction for most of the ETL work I have been involved with in the last several years.Each company I worked for had their own custom file readers and parsers. Thus I am very interested in having a product like Defuddle that can read/parse the basics. 3. Can someone clarify the extent, if any, to which DFDL is expected to be used to validate data content as opposed to data structure. This isn't at all clear anywhere in the spec that I could find. I added a comment suggesting a statement in either the 'What DFDL is' or the 'What DFDL is not' section about this. My assumption is that if a DFDL schema is used to unparse an infoset the only guarantee is that the resulting physical structure will be correct and not the logical structure. Consider an example of a filed with records having two date fields: start_date, end_date. There are two input (parsing) operations that are potentially useful: 1. Physical - Read the date values into internal elements (or XML elements) and validate the presence/absence/nullable state of each 2. Business/Logical - Validate that the end_date is either null, or if not null is greater than or equal to the start_date. Naturally DFDL will support #1 but does it support #2? I wouldn't think so. The ETL work I do might not even know what the business rule is and even when we do we always have to deal with 'dirty' data. There are two corresponding output (unparsing) operations of interest: 1. Physical - Write the date values in the proper external physical format. 2. Business/Logical - validate that the date values in the infoset that are to be written meet the business rule stated above. Again, I would expect DFDL to support #1 but not #2. I suggest that some comment or description be added to the spec to make clear the extent to which DFDL supports the business/logical aspect of the data. My concern is that users will be misled into thinking they can arbitrarily populate an infoset and then, using a DFDL schema, create an external file that can be properly used by a native application. It's one thing to 'roundtrip' data that is sourced from a native application and quite another to produce valid native files using a non-native application. Unless, of course you have ideas about branching out to BRDL - Business Rule Definition Language? Rick Post

2 1

Alan cannot make call this week
by Alan Powell 02 Jun '08

02 Jun '08

I am away for the rest of this week so cannot make the call Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

Re: [DFDL-WG] available of DFDL use cases - sample data being used to refine spec
by Mike Beckerle 02 Jun '08

02 Jun '08

Actually we have a test rig but few test cases and they are getting stale. We have an XML test file format which lets one create self contained test files ie you can put some binary data as a hex dump, text data, or even binary bit dumps, along with the XML equivalent of it (the natural xml peojection of the data) as XML data. The inverse for output as well. The test file references a DFDL schema so that the schema is exactly as it would be for "real" use. I'll find the links. ...mikeb On May 31, 2008, at 9:08 PM, "RPost" <rp0428(a)pacbell.net> wrote: > Hi, > > > > I finally managed to connect to the DFDL site long enough to > download a copy of ogf-dfdl-v1.0-Core-030.doc, the draft 30 doc. > > > > In reviewing the doc and recent emails at the archives, http://www.ogf.org/pipermail/dfdl-wg/ > , there are references to sample data and use cases that are being > used to help refine the specification. > > > > Can I get copies of any use cases and/or specific sample date that > are being used? I would like to start creating a ‘DFDL sample’ > companion doc that would contain a richer set of examples than what > would be appropriate for the specification itself. > > > > For example there was a recent discussion from the archives about > Nulls and Defaults (Tue Apr 8 19:31:52 CDT 2008). Parts of the > discussion regard what properties must be specified and what the > allowable property/data combinations are. > > > > I would like to create actual sample data for some of these cases > (e.g. required field that is missing but has a default) and perhaps > a truth table that shows explicitly what combination of properties > must be specified to be valid. > > > > This doc would not be concerned with exactly how (or where in the > DFDL schema) a specific property is specified but only how the > property/properties can be used for a specific test case. > > > > Such a doc could help bring some clarity to the interaction of the > properties and, at some future time, assist implementers in creating > proper validation test sets. > > > > If someone could forward me some sample use cases, or data scenarios > (e.g. Cobol OCCURS data) and let me know which ones would be most > useful to have some more detailed property examples for I could > start the doc with those. > > > > Thanks for any info. > > > > > > Rick > > > > p.s. The link I have been largely unsuccessful using to access the > DFDL site is > > > > http://www.ggf.org/ggf_areas_data.htm > > > > Most of the links from this page, including the DFDL one, give proxy > or maintenance errors. > > > > I have emailed the webmaster about the problem but haven’t heard any > thing back. > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0