- dfdl-wg - lists.ogf.org

Confirmed: OGF DFDL Working Group weekly call (2 Jul 14:00 GDT in DE2K09)
by Steve Hanson 25 Jun '08

25 Jun '08

Calendar Entry ---------------------------------------------------------------------------------------------------------------------------- Subject Chair OGF DFDL Working Group weekly call Steve Hanson/UK/IBM Invitees Date Required (to) 02/07/2008 Alan Powell/UK/IBM, mbeckerle(a)oco-inc.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Time Kimber/UK/IBM 14:00 - 15:00 (1 hour) Optional (cc) dfdl-wg(a)ogf.org Proposed start -1 at Proposed end -1 at Original start 02/07/2008 at 14:00 Original end 02/07/2008 at 15:00 Repeats This meeting is scheduled to repeat Show detailed repeat information This meeting is scheduled to repeat Applies to This instance and all future instances Where Location DE2K09 Participant Passcode: 8190865 Country Toll Numbers Freephone/Toll Free Number AUSTRIA 43-1-928-9652 BELGIUM 32-2-400-6791 0800-4-9612 DENMARK 45-7014-0237 FRANCE LYON: 33-4-26-69-12-74 080-511-3725 FRANCE MARSEILLE: 33-4-86-06-00-74 080-511-3725 FRANCE PARIS: 33-1-70-70-81-58 080-511-3725 GERMANY 49-69-2222-20363 0800-100-6504 IRELAND 353-1-247-7768 ITALY 39-02-3600-3648 800-985-679 LUXEMBOURG 352-27-000-1312 NETHERLANDS 31-20-713-2741 0800-020-3237 SPAIN 34-91-414-1542 800-099-667 SWEDEN 46-8-566-10-780 SWITZERLAND 41-44-580-3334 0800-000-516 UNITED KINGDOM BIRMINGHAM: 44-121-210-9014 0800-279-9193 UNITED KINGDOM GLASGOW: 44-141-202-3214 0800-279-9193 UNITED KINGDOM LEEDS: 44-113-301-2114 0800-279-9193 UNITED KINGDOM LONDON: 44-20-7019-0808 0800-279-9193 UNITED KINGDOM MANCHESTER: 44-161-601-1414 0800-279-9193 USA 866-505-4412

1 0

Required encodings and testing
by RPost 25 Jun '08

25 Jun '08

Ian raises a couple of interesting issues. 1. Exactly how should encodings be specified? I like the idea of using http://www.iana.org/assignments/character-sets if possible. Even here the ICU *.ucm files don't always use the 'Name' specified in the standard and the standard does not include the 'x-' (experimental - meaning not a standard). Java mostly uses the standard 'Name' value but doesn't always match the ICU name. Even using the standard the 'Name' itself may not be the preferred MIME name. Name: Extended_UNIX_Code_Packed_Format_for_Japanese MIBenum: 18 Alias: csEUCPkdFmtJapanese Alias: EUC-JP (preferred MIME Name) Java supports the name and both aliases. ICU has several in their 'convrtrs.txt' file including X-EUC-JP. The standard says 'The MIBenum value is a unique value for use in MIBs to identify coded character sets.' Perhaps using the standard 'MIBenum' value for uniqueness and the 'Name' and any or all of the aliases would work. I'm guessing that an appendix will ultimately provide the list? 2. For DFDL, when there is a conflict, what is more important: adherance to a standard? Or providing schema writers to express requirements unambigously? This line in the rfc2781 link Ian provides caught my eye: '...addresses the issues of serializing UTF-16 as an octet stream for transmission over the Internet'. Does this apply since DFDL isn't really targeting 'transmission over the Internet'? You can't transmit a binary file (whether it includes UTF-16 or not) over the internet without converting it first; often to BASE64. So I'm not sure the standard for when to include/exclude BOMs applies. I would suggest that it if critical to allow a schema writer to specify exactly what to expect on parse and what is allowed on unparse. As long as the writer can do that using a supported encoding and possibly DFDL properties, such as byteOrder we're covered. You may very well need to provide a way to explicitly specify whether a BOM 'is/is not/might be' present and whether a BOM 'must be/can be' written on output. I didn't mention it in my original post but the BOM issue is one of the related issues that Addison Phillips ran into writing classes to serialize text into fixed width fields. Namely: for a fixed-width field (width in bytes) how do you determine how many text characters of a specified encoding will fit into the field? He had to take into account BOMs as well as ensuring that complete 'shift in - shift out' sequences could be written without overflow. Then he had a similar issue to figure out the padding. In the consulting I do my assumption is that data, legacy and otherwise. I usually have no problem proving it even if the user Insists it isn't; it doesn't necessarily obey the business rules that it is supposed to. That is the #1 problem I run into as an ETL consultant. Fields are NULL that shouldn't be. A name field in one system is VARCHAR2(30) and on another the same field is VARCHAR2(40). What do you do with data moving from a '40' to a '30'? So for DFDL to support the parsing/reading of old legacy data, possibly because the original tools don't exist anymore, a schema writer has to be able to explicitly control how the data is interpreted whether it meets the standards or not. 3. To BOM or not to BOM - that is the question. Use Ian's proposal for doing the standard thing in the standard way. But ensure that there is at least some way for a schema writer control explicitly what parse/unparse will do. I wouldn't be inclined to add anything to the spec at this point without a specific use case that requires it. Speaking of which - did anyone ever locate the links on your site where I can find some of your use case descriptions or discussions?

1 0

Initial list of Required encodings for DFDL version 1
by RPost 23 Jun '08

23 Jun '08

Q - What is the current thinking for the character set encodings that MUST be implemented by a conforming DFDL processor for version 1? I have been performing tests with length-prefixed strings and strings using terminators to see what issues affect the ability to detect the boundaries between strings and binary data or terminator strings that immediately follow the string. For length-prefixed strings you need to be able to either encode the byte array and iterate the string character by character or perform byte counting using only the byte stream and the bit ranges in the bytes themselves.. Issue #1 - It will not be trivial to create all of the test cases to fully test the corner cases for each encoding. Obviously the fewer encodings that have to be supported initially the better in terms of implementation. Issue #2 - There is no current support for byte counting in Java or ICU. For encodings that are pure single-byte or pure multi-byte the end of the string can be found by examining the byte string itself without performing character encoding. The classes available all perform conversions of entire buffers (or series of buffers) and the classes also consume large amounts of the byte stream. For some encodings (e.g. UTF-8) an algorithmic process can examine byte values and determine if a character consumes 1, 2 or more bytes. Still other encodings will need to have custom processes written to either encode and iterate the string or use a specially designed table to perform byte counting. As with issue #1 the fewer encodings needing special handling that need to be supported initially the fewer problems for implementers. Issue #3 - some encodings have multiple possible byte representations for the same character. If a terminator string specified as 'END' in a DFDL property it must be converted to the proper encoding when searching for it. The easiest way to do this is to encode it, convert the encoded value to a byte array and then search the input stream byte array for a match. The binary file could include bytes that express one encoding of the character and the Java code could be searching for the character using another byte representation. Q - Does the DFDL spec need to allow a terminator to be specified as a hex byte array so that the exact byte sequence to search for can be specified? Issue #4 - If a string can be specified as using one encoding and a terminator can use a different encoding is it possible that the terminator byte sequence is also a valid string byte sequence even though the characters being represented are different? I haven't been able to determine if this can happen. Q - Does the DFDL spec need to disallow different encodings for strings and terminators for version 1? Or are you confident that this corner case is unlikely to be an issue. I have been in contact with Addison Phillips, the current chair of the W3C Internationalization core WG, and he ran into many of the above issues when implementing character set providers for WebMethos (since consumed by SoftwareAG). He also referred me to a contact at ICU and I hope to hear from them in the next week or two. Meanwhile, any thoughts or suggestions you have on the above would be appreciated. While I am waiting for feedback from ICU and Addison I am trying to determine an effective way to set up an automated test harness that can be used to generate different combinations of strings, terminators and encodings and perform volume testing. Mike suggested using the test example he provided but it only showed one data string for input. That might be adequate for simple tests but, because test cases may need to be shared by multiple test XSD files it may not be scalable for volume testing or testing of multiple cases. Rick

2 1

XSL-style SmartLengthString Sample
by RPost 18 Jun '08

18 Jun '08

Hi, Attached is a doc that describes a (syntax-deprived) version of Mike's SmartLengthString example that is done more in a style that combines XML and XSL. It would use XML for the visible infoset and element declarations but an XSL style for the transforms. It attempts to: 1. Separate input/parse considerations from output/unparse. 2. Provide for all physical elements without requiring them to be in the visible infoset 3. Provide for multiple/nested layers 4. Allow for multiple transforms to be performed even within a single layer. I still have scalability concerns with using one outputValueCalc (or input) expression if multiple or complex transformations need to be done. The idea is to use a simple structure that can be nested to get the complexity that is needed. Here is an excerpt from the doc but the formatting will probably get out of whack: <xs:complexType name="smartLengthString" ...other attributes...> <xs:sequence> <hiddenStructure> -- the annotations and transforms between the physical and logical layers <hiddenElements> . . . one or more elements . . . </hiddenElements> <inputLayer> . . . The transforms and lower layer elements needed to produce the higher-layer Elements and ultimately the infoset informational values. </inputLayer> <outputLayer> . . . The transforms and higher layer elements needed to produce the lower-layer elements and ultimately the physical layer values. </outputLayer> </hiddenStructure> <visibleElements> -- the infoset as experienced by the user </xs:sequence> </xs:complexType> As you can see I took a lot of liberties with the syntax to try to make it easier to see the structure I am using. The 'hiddenStructure' element is what contains the equivalent of the <xs:annotation><xs:appinfo> from the example. The hidden elements are equivalent to elements in the example that are not in the public infoset. The inputLayer contains whatever is needed (including nested inputLayers) for parsing. Likewise for the outputLayer and unparsing. Here is a skeleton inputLayer: <inputLayer> <layerElements>  </layerElements> <transforms>  <transforms> <inputLayer>  </inputLayer> </inputLayer> There is a section for elements specific to this layer; somewhat akin to intermediate working variables or intermediate calculations. Another section is for an ordered list of transforms. This allows for as many or as few as needed but provides clean separation between them. The first transform might create the value of the 'lengthFlag' hidden element. The next transform could then use the lengthFlag value to extract the logicalLength value and so on. It sure seems to flow ok but maybe that is just in my head. The nested 'inputLayer' allows for a chain of transforms to be created. The nested inputLayer could create values for this layer's layerElements and get it's input from either another nested layer or from the physical layer itself. If the doc does nothing more than help solidify your thinking about how to get where you want to go it will have done it's job. By the way, thanks for the excellent example Mike. Rick

2 1

Invitation: OGF DFDL Working Group weekly call (18 Jun 14:00 GDT in DE2K09)
by Steve Hanson 16 Jun '08

16 Jun '08

Calendar Entry ---------------------------------------------------------------------------------------------------------------------------- Subject Chair OGF DFDL Working Group weekly call Steve Hanson/UK/IBM Invitees Date Required (to) 18/06/2008 Alan Powell/UK/IBM, mbeckerle(a)oco-inc.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Time Kimber/UK/IBM 14:00 - 15:00 (1 hour) Optional (cc) dfdl-wg(a)ogf.org Proposed start -1 at Proposed end -1 at Original start 18/06/2008 at 14:00 Original end 18/06/2008 at 15:00 Repeats Show detailed repeat information This meeting is scheduled to repeat Where Location DE2K09 Participant Passcode: 8190865 Country Toll Numbers Freephone/Toll Free Number AUSTRIA 43-1-928-9652 BELGIUM 32-2-400-6791 0800-4-9612 DENMARK 45-7014-0237 FRANCE LYON: 33-4-26-69-12-74 080-511-3725 FRANCE MARSEILLE: 33-4-86-06-00-74 080-511-3725 FRANCE PARIS: 33-1-70-70-81-58 080-511-3725 GERMANY 49-69-2222-20363 0800-100-6504 IRELAND 353-1-247-7768 ITALY 39-02-3600-3648 800-985-679 LUXEMBOURG 352-27-000-1312 NETHERLANDS 31-20-713-2741 0800-020-3237 SPAIN 34-91-414-1542 800-099-667 SWEDEN 46-8-566-10-780 SWITZERLAND 41-44-580-3334 0800-000-516 UNITED KINGDOM BIRMINGHAM: 44-121-210-9014 0800-279-9193 UNITED KINGDOM GLASGOW: 44-141-202-3214 0800-279-9193 UNITED KINGDOM LEEDS: 44-113-301-2114 0800-279-9193 UNITED KINGDOM LONDON: 44-20-7019-0808 0800-279-9193 UNITED KINGDOM MANCHESTER: 44-161-601-1414 0800-279-9193 USA 866-505-4412

1 0

DFDL: new meeting time
by Mike Beckerle 16 Jun '08

16 Jun '08

Can we officially change the weekly meeting time to 9am Wednesday ET.US (2pm UK Time) We discussed this before, but I am now running into many conflicts with our current meeting time, so it's no longer viable for me. Please chime in on whether this time will work for you. .mikeb Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451 | <mailto:mbeckerle.dfdl@gmail.com> mbeckerle.dfdl(a)gmail.com

3 4

DFDL: gridforge up today
by Mike Beckerle 16 Jun '08

16 Jun '08

Gridforge appears to be alive again. See: http://forge.gridforum.org/sf/docman/do/listDocuments/projects.dfdl-wg/docma n.root.current_0?_sortby=documentList%28dateLastModified%29&_sorder=document List%28asc%29 .mikeb Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451 | <mailto:mbeckerle.dfdl@gmail.com> mbeckerle.dfdl(a)gmail.com

2 1

Gridforge Problems
by Mike Beckerle 11 Jun '08

11 Jun '08

Can we get status on gridforge? We've been unable to use it for some time now. It does seem to come up occasionally because a new group member has been able to download some docs, but we've found it unreliable enough that we can't depend on it. Is it being fixed? Thanks Mike Beckerle | OGF DFDL WG Co-Chair | CTO | Oco, Inc. Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451 | <mailto:mbeckerle.dfdl@gmail.com> mbeckerle.dfdl(a)gmail.com

1 0

comments and concerns re choice-discriminator-example
by RPost 11 Jun '08

11 Jun '08

I have some questions and possible concerns re the sample xml file Mike posted. The attached document 'SmartlengthStringComments1 discusses them. I attached the other doc that I found from 2003 written by Martin Westhead. The current sample and an XSLT sample in the old doc bear an uncanny resemblance and thought it might be worthwhile to look at the two side-by-side since they are for the exact same use case. It raises the question of whether DFDL syntax could allow actual XSLT snippets to be included in the annotation section to use the power of XSLT to do these things. XSLT also supports extensions that could handle complex or unusual corner cases. A parser should be able to (but I haven't confirmed it) create a mini XSL stylesheet from the snippet and use a transform to generate the result value that is needed. If so this would allow an existing standard language to be used. See my item #5 in my doc since there I ask the question of how I might contribute without being disruptive. It is not my intention to disrupt your efforts but the resemblance between these two approaches is so similar I had to bring it to your attention. Rick

2 1

Re: [DFDL-WG] Comments on draft 32 of DFDL spec
by JimMyers 10 Jun '08

10 Jun '08

Rick, A quick update from NCSA on Defuddle: after we got things started at PNNL and Tara Talbott created the initial version, we did indeed have an 'ice age' with no direct funding for it. Last year we received a small amount of funding at NCSA (where I moved to) from NARA to start Defuddle moving again and to incorporate it into the EU SHAMAN project's digital preservation architecture (lots of things in SHAMAN but the relevant idea here is the iRODS storage broker calling Defuddle to map things to logical models and the Multivalent Browser viewing the logical model). Due in part to the lack of funding, we've dropped out of the DFDL discussion for a while. Probably the most significant issues where Defuddle doesn't match the draft spec are that I believe the idea of layers has been dropped from the version 1 spec plan (which we think is critical)and there has been a lot of work on the spec to deal with nilability, etc. (some of which I've argued can be avoided if you have layers). In the next year+, we're rebuilding Defuddle on the latest libraries, doing a bunch of scalability and stress testing, targetting some common file formats (.e.g PNG) to show it works beyond the relatively simple scientific formats we've done to date, exploring uses in the digital library/preservation communities and looking to extend from XML modeling to RDF/semantic modeling. The last one of these starts to go in the direction of your questions #2 - if you get to RDF, you can start using OWL/rule constraints to assure that the data coming out/going in is semantically what you want (The DFDL group has discussed the backward direction, but at least for Defuddle it is not yet in our actual development plans). Our intial thought for doing this is to just include a GRRDL annotation in the DFDL file which tells you how to map (via XSLT for example) from the XML created by the current Defuddle to the RDF you want - in essence going from binary to XML and XML to RDF logical model in two sequential steps. We haven't thought as deeply about semantic validation as the DFDL group has about XML-level validation (i.e. the discussion of errors that can be caught at the time of format definition, versus XML schema constraint violations on parsing versus parsing errors themselves, etc.), but I think getting to RDF will allow a lot of what you're talking about with standard semantic web tools. As we get going on Defuddle again, I hope to get reconnected with the DFDL effort - I've mostly been lurking on the list the past year+ - and see how we can help without being disruptive (perhaps looking at post 1.0 changes?). In any case, I wanted to respond to your question and give a 'what's new' report back to the group since I've been quiet a while. For Defuddle, while I still think we need to grow, things are looking up with a very interested sponsor, international collaboration, and some momentum after the 'ice age'. (As always, I'd be very happy to talk with anyone who'd like to get involved in Defuddle (software development or creating format descriptions and using it, etc.) - Defuddle is open source and I know everyone involved to date would really like to see it become community (versus project) driven.) Cheers, Jim James D. Myers, Ph.D. Associate Director, Cyberenvironments National Center for Supercomputing Applications University of Illinois at Urbana Champaign 1205 W Clark St. Urbana, IL 61801 217-244-1934 ----- "RPost" <rp0428(a)pacbell.net> wrote: > Hi, > > > > I have been performing ETL since the early '80s when there were over > 20 different floppy disk formats > > and we had to write products like Uniform to copy data from one format > to another. > > > > At MicroPro Int'l (of WordStar fame) I was also involved in rewriting > domestic versions of software > > to support Shift-JIS for display and input for the Japanese market. > > > > More recently I have written and supported ETL software for the > telecommunications sector, > > using ATIS XML formats) and for banking which uses Automated Clearing > House (ACH) XML standards. > > ACH uses a lot of files with a format: FileHeader, (BatchHeader, > Detail+, BatchTrailer)+, FileTrailer. > > > > As you can imagine I have seen a lot of duplication of effort due to > the lack of a standard way to > > define even the simplest of data formats, let alone the complex ones. > > > > Hence my interest in DFDL, which started with Defuddle. > > > > I have started to read the recent archives to get a sense of where the > DFDL project stands > > now compared to where it was in 2003-2005 when the great DFDL ice-age > began and everyone's > > projects (Defuddle, Virtual XML) froze in their tracks. > > > > Also, I have reviewed the recent Core-032.2 document, added comments > to it and have a few > > general questions about the current state of all things DFDL. > > > > 1. Re Parsing (input) only - How complete is the current draft spec in > terms of being able > > to create schemas and a parser for reading binary files? Would it > support the most common delimited > > and header/detail/trailer types of files? > > > > From my limited exposure to the drafts and emails, and my early use of > Defuddle, it certainly seems like > > the parsing part is nearly complete and ready to have implementations > created. > > > > 2. Will a conforming DFDL processor be required to support both > parsing and unparsing? > > > > I have only needed the parsing direction for most of the ETL work I > have been involved with in > > the last several years.Each company I worked for had their own custom > file readers and parsers. > > Thus I am very interested in having a product like Defuddle that can > read/parse the basics. > > > > 3. Can someone clarify the extent, if any, to which DFDL is expected > to be used to validate data > > content as opposed to data structure. This isn't at all clear anywhere > in the spec that I could find. > > I added a comment suggesting a statement in either the 'What DFDL is' > or the 'What DFDL is not' > > section about this. > > > > My assumption is that if a DFDL schema is used to unparse an infoset > the only guarantee is that the > > resulting physical structure will be correct and not the logical > structure. > > > > Consider an example of a filed with records having two date fields: > start_date, end_date. > > > > There are two input (parsing) operations that are potentially useful: > > > > 1. Physical - Read the date values into internal elements (or XML > elements) and validate the > > presence/absence/nullable state of each > > > > 2. Business/Logical - Validate that the end_date is either null, or if > not null is greater than > > or equal to the start_date. > > > > Naturally DFDL will support #1 but does it support #2? I wouldn't > think so. The ETL work I do might > > not even know what the business rule is and even when we do we always > have to deal with 'dirty' data. > > > > There are two corresponding output (unparsing) operations of interest: > > > > 1. Physical - Write the date values in the proper external physical > format. > > > > 2. Business/Logical - validate that the date values in the infoset > that are to be written meet > > the business rule stated above. > > > > Again, I would expect DFDL to support #1 but not #2. > > > > I suggest that some comment or description be added to the spec to > make clear the extent to which > > DFDL supports the business/logical aspect of the data. > > > > My concern is that users will be misled into thinking they can > arbitrarily populate an infoset and > > then, using a DFDL schema, create an external file that can be > properly used by a native application. > > > > It's one thing to 'roundtrip' data that is sourced from a native > application and quite another to > > produce valid native files using a non-native application. > > > > Unless, of course you have ideas about branching out to BRDL - > Business Rule Definition Language? > > > > Rick Post > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/dfdl-wg

2 1