- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Unparsing and outputValueCalc - A Powell et al
by RPost 26 Jun '08

26 Jun '08

Re the combination of output and input value calc's. If I understand the latest discussions it is possible to create a schema will allows: 1. All input fields to be calculated. 2. All output fields to be calculated and have different values than the input fields. #1 means that any file, including an empty one, can be used for input. #2 means that although the user will interact with the input fields none of the input fields will Ever appear in the augmented infoset or output file. It also means that no infoset value that is set by a user will ever appear in the output file. Sounds like this is pretty well hashed out. By the way, I'm not using the above contrived, unrealistic example as criticism. Rather to Illustrate that it looks like you've provided closure to the issue and haven't left any 'gotchas'. Of course, now you've got me thinking about adding a random number function or two so I can write a schema for test data generation! (ok, maybe that last one was a bit sarcastic)

1 0

Re: [DFDL-WG] Required encodings and testing (Mike Beckerle)
by RPost 26 Jun '08

26 Jun '08

[So if you want BOM behavior it's because the beginning of a stream has special treatment, in this case it is reasonable to model the BOM as a separate element to be found at the beginning of a "stream", optionally hidden, perhaps optional, and compute dfdl:byteOrder in terms of its value. ] Works for me. The spec should deal with the (mostly) normal stuff expected. As a user I just want some way to use the DFDL processor to get the job done. I don't care if I have to clone the 'official' schema for a legacy file and tweak a few things; those are the hard cases. Sounds like there will be a way to deal with all the issues. The only thing I can think of, and it should be pretty rare, is when the length-prefix has a character count that doesn't include a BOM and there is a BOM present. Once the parser gets out of sync it may not be able to recover. Even so a log should show what element it was trying to parse and the position in the file for the start of the element. That would be enough to enable me to use a hex editor, figure out what the problem was and find a way around it.

1 0

DFDL: Minutes from OGF WG call, 18 June 2008
by Alan Powell 25 Jun '08

25 Jun '08

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 14:00 GMT, 18 June 2008 Attendees Mike Beckerle (Oco) Steve Hanson (IBM) Alan Powell (IBM) 1. Discussion of unparsing Discussed Mike's description of unparsing email dated 12/06 outputValueCalc has the highest priority. If there is a value in the infoset it is ignored Use case is using parsed infoset for output. The outputValueCalc can be evaluated multiple times but must return the same result It is an error if there is a value in the onfoset for a hidden element dfdl:lenght of an element with an inputvalueCalc returns aero 2. Hidden Elements Discussion about alternative methods of defining hidden elements using minoccurs=maxOccurs=0. SH to discuss with Sandy Rao 3. Gridforge Gridforge has been unavailable for some time so we need to find an alternative. AP to investigate options 4. Spec work items AP to reissue list of remaining spec work items. Meeting closed, 15:00 GMT Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Required encodings and testing
by RPost 25 Jun '08

25 Jun '08

Thanks for the response re encodings and issues. Very helpful. I put my responses in the attachment but here is the first part about encoding. Your response: We haven't picked a basic set that all conforming implementations must support other than that UTF-8 and USASCII must be supported. We might require more than this though. That's a relief! The current spec mentions UTF-8, ebcdic-cp-us (IBM037), and UTF-16BE. Since Java 1.6 supports 160 encodings using 686 aliases I've no doubt you see the reason for my question about which encodings need initial support. ICU supports even more encodings and requiring some of these could implicitly require implementors to support ICU. Not an issue if that is truly needed but that requirement alone could dissuade some from participating in the project. The encodings I have examined/tested so far are: US-ASCII, ISO-8859-1, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE, IBM1047, IBM500, IBM037, x-UTF-16LE-BOM, X-UTF-32BE-BOM, X-UTF-32LE-BOM. I have not run across any issues with any of the above encodings. ICU includes 175 UCM files of which 135 are for SBCS encodings. I have not tested or examined all of these but would not expect them to be an issue either. Also not examined are the 27 UCM files for MBCS encodings. A brief review shows that many of these should not be an issue. BIG5 or GB18030 could definitely be an issue and there are several others like these that might require a custom effort to support. Ok if you really need it but better delayed initially if you don't. Glad to know we don't need to visit these for the short term. I'm sure implementers would much rather concentrate on the DFDL aspect of things rather than become encoding experts. I know I would.

4 5

DFDL: output and choices/discriminators
by Mike Beckerle 25 Jun '08

25 Jun '08

I'd like to discuss this example in email and/or on our call this week. It illustrates that discriminators must be evaluated both on output and on input. We proposed at the F2F that assertions are only about parsing, but the same cannot be said of discriminators. Because email often line-wraps in ways that break things I've also attached the same example as a file. <?xml version="1.0" encoding="iso-8859-1"?>   <xs:schema targetNamespace="http://dataformat.org/tests" elementFormDefault="qualified" xsi:schemaLocation= "http://www.ogf.org/dfdl/dfdl-0.1 ../../xsd/dfdl.xsd http://www.w3.org/2001/XMLSchema ../../xsd/XMLSchema.xsd" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.ogf.org/dfdl/tests" xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-0.1">  <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/dfdl-0.1"> <dfdl:defineFormat name="default"> <dfdl:format representation="binary" lengthKind="implcit" initiator="" separator="" terminator="" alignmentUnits="bits" /> </dfdl:defineFormat> </xs:appinfo></xs:annotation>  <xs:complexType name="smartLengthString" dfdl:ref="default" dfdl:lengthKind="explicit" dfdl:lengthUnits="bits"> <xs:sequence> <xs:annotation><xs:appinfo source="..."> <dfdl:hidden> <xs:element name="lengthFlag" type="xs:byte" dfdl:length="1" dfdl:alignment="8" dfdl:outputValueCalc='{ if ( ../logicalLength > 127 ) then "1" else "0" }' /> <xs:choice dfdl:choiceKind='variable' dfdl:choiceResolvable="true">  <xs:sequence> <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/dfdl-0.1"> <dfdl:discriminator test="{ ../lengthFlag != '1' }" /> </xs:appinfo></xs:annotation> <xs:element name="oneByte" type="xs:byte" dfdl:alignment="1" dfdl:length="7" dfdl:outputValueCalc="{ ../logicalLength }" /> </xs:sequence>  <xs:element name="fourByte" type="xs:int" dfdl:alignment="1" dfdl:length="31" dfdl:outputValueCalc="{ ../logicalLength }" /> </xs:choice>  <xs:element name="logicalLength" type="xs:int" dfdl:inputValueCalc="{ if (../lengthFlag = '1') then ../fourByte else ../oneByte }" dfdl:outputValueCalc= "{ dfdl:length(../str, 'characters') }" /> </dfdl:hidden> </xs:appinfo></xs:annotation> <xs:annotation><xs:appinfo source="..."> <xs:element name="str" type="xs:string" dfdl:length="{ ../logicalLength }" dfdl:lengthUnits="characters" /> </xs:sequence> </xs:complexType> </xs:schema>

2 5

Confirmed: OGF DFDL Working Group weekly call (2 Jul 14:00 GDT in DE2K09)
by Steve Hanson 25 Jun '08

25 Jun '08

Calendar Entry ---------------------------------------------------------------------------------------------------------------------------- Subject Chair OGF DFDL Working Group weekly call Steve Hanson/UK/IBM Invitees Date Required (to) 02/07/2008 Alan Powell/UK/IBM, mbeckerle(a)oco-inc.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Time Kimber/UK/IBM 14:00 - 15:00 (1 hour) Optional (cc) dfdl-wg(a)ogf.org Proposed start -1 at Proposed end -1 at Original start 02/07/2008 at 14:00 Original end 02/07/2008 at 15:00 Repeats This meeting is scheduled to repeat Show detailed repeat information This meeting is scheduled to repeat Applies to This instance and all future instances Where Location DE2K09 Participant Passcode: 8190865 Country Toll Numbers Freephone/Toll Free Number AUSTRIA 43-1-928-9652 BELGIUM 32-2-400-6791 0800-4-9612 DENMARK 45-7014-0237 FRANCE LYON: 33-4-26-69-12-74 080-511-3725 FRANCE MARSEILLE: 33-4-86-06-00-74 080-511-3725 FRANCE PARIS: 33-1-70-70-81-58 080-511-3725 GERMANY 49-69-2222-20363 0800-100-6504 IRELAND 353-1-247-7768 ITALY 39-02-3600-3648 800-985-679 LUXEMBOURG 352-27-000-1312 NETHERLANDS 31-20-713-2741 0800-020-3237 SPAIN 34-91-414-1542 800-099-667 SWEDEN 46-8-566-10-780 SWITZERLAND 41-44-580-3334 0800-000-516 UNITED KINGDOM BIRMINGHAM: 44-121-210-9014 0800-279-9193 UNITED KINGDOM GLASGOW: 44-141-202-3214 0800-279-9193 UNITED KINGDOM LEEDS: 44-113-301-2114 0800-279-9193 UNITED KINGDOM LONDON: 44-20-7019-0808 0800-279-9193 UNITED KINGDOM MANCHESTER: 44-161-601-1414 0800-279-9193 USA 866-505-4412

1 0

Required encodings and testing
by RPost 25 Jun '08

25 Jun '08

Ian raises a couple of interesting issues. 1. Exactly how should encodings be specified? I like the idea of using http://www.iana.org/assignments/character-sets if possible. Even here the ICU *.ucm files don't always use the 'Name' specified in the standard and the standard does not include the 'x-' (experimental - meaning not a standard). Java mostly uses the standard 'Name' value but doesn't always match the ICU name. Even using the standard the 'Name' itself may not be the preferred MIME name. Name: Extended_UNIX_Code_Packed_Format_for_Japanese MIBenum: 18 Alias: csEUCPkdFmtJapanese Alias: EUC-JP (preferred MIME Name) Java supports the name and both aliases. ICU has several in their 'convrtrs.txt' file including X-EUC-JP. The standard says 'The MIBenum value is a unique value for use in MIBs to identify coded character sets.' Perhaps using the standard 'MIBenum' value for uniqueness and the 'Name' and any or all of the aliases would work. I'm guessing that an appendix will ultimately provide the list? 2. For DFDL, when there is a conflict, what is more important: adherance to a standard? Or providing schema writers to express requirements unambigously? This line in the rfc2781 link Ian provides caught my eye: '...addresses the issues of serializing UTF-16 as an octet stream for transmission over the Internet'. Does this apply since DFDL isn't really targeting 'transmission over the Internet'? You can't transmit a binary file (whether it includes UTF-16 or not) over the internet without converting it first; often to BASE64. So I'm not sure the standard for when to include/exclude BOMs applies. I would suggest that it if critical to allow a schema writer to specify exactly what to expect on parse and what is allowed on unparse. As long as the writer can do that using a supported encoding and possibly DFDL properties, such as byteOrder we're covered. You may very well need to provide a way to explicitly specify whether a BOM 'is/is not/might be' present and whether a BOM 'must be/can be' written on output. I didn't mention it in my original post but the BOM issue is one of the related issues that Addison Phillips ran into writing classes to serialize text into fixed width fields. Namely: for a fixed-width field (width in bytes) how do you determine how many text characters of a specified encoding will fit into the field? He had to take into account BOMs as well as ensuring that complete 'shift in - shift out' sequences could be written without overflow. Then he had a similar issue to figure out the padding. In the consulting I do my assumption is that data, legacy and otherwise. I usually have no problem proving it even if the user Insists it isn't; it doesn't necessarily obey the business rules that it is supposed to. That is the #1 problem I run into as an ETL consultant. Fields are NULL that shouldn't be. A name field in one system is VARCHAR2(30) and on another the same field is VARCHAR2(40). What do you do with data moving from a '40' to a '30'? So for DFDL to support the parsing/reading of old legacy data, possibly because the original tools don't exist anymore, a schema writer has to be able to explicitly control how the data is interpreted whether it meets the standards or not. 3. To BOM or not to BOM - that is the question. Use Ian's proposal for doing the standard thing in the standard way. But ensure that there is at least some way for a schema writer control explicitly what parse/unparse will do. I wouldn't be inclined to add anything to the spec at this point without a specific use case that requires it. Speaking of which - did anyone ever locate the links on your site where I can find some of your use case descriptions or discussions?

1 0

Initial list of Required encodings for DFDL version 1
by RPost 23 Jun '08

23 Jun '08

Q - What is the current thinking for the character set encodings that MUST be implemented by a conforming DFDL processor for version 1? I have been performing tests with length-prefixed strings and strings using terminators to see what issues affect the ability to detect the boundaries between strings and binary data or terminator strings that immediately follow the string. For length-prefixed strings you need to be able to either encode the byte array and iterate the string character by character or perform byte counting using only the byte stream and the bit ranges in the bytes themselves.. Issue #1 - It will not be trivial to create all of the test cases to fully test the corner cases for each encoding. Obviously the fewer encodings that have to be supported initially the better in terms of implementation. Issue #2 - There is no current support for byte counting in Java or ICU. For encodings that are pure single-byte or pure multi-byte the end of the string can be found by examining the byte string itself without performing character encoding. The classes available all perform conversions of entire buffers (or series of buffers) and the classes also consume large amounts of the byte stream. For some encodings (e.g. UTF-8) an algorithmic process can examine byte values and determine if a character consumes 1, 2 or more bytes. Still other encodings will need to have custom processes written to either encode and iterate the string or use a specially designed table to perform byte counting. As with issue #1 the fewer encodings needing special handling that need to be supported initially the fewer problems for implementers. Issue #3 - some encodings have multiple possible byte representations for the same character. If a terminator string specified as 'END' in a DFDL property it must be converted to the proper encoding when searching for it. The easiest way to do this is to encode it, convert the encoded value to a byte array and then search the input stream byte array for a match. The binary file could include bytes that express one encoding of the character and the Java code could be searching for the character using another byte representation. Q - Does the DFDL spec need to allow a terminator to be specified as a hex byte array so that the exact byte sequence to search for can be specified? Issue #4 - If a string can be specified as using one encoding and a terminator can use a different encoding is it possible that the terminator byte sequence is also a valid string byte sequence even though the characters being represented are different? I haven't been able to determine if this can happen. Q - Does the DFDL spec need to disallow different encodings for strings and terminators for version 1? Or are you confident that this corner case is unlikely to be an issue. I have been in contact with Addison Phillips, the current chair of the W3C Internationalization core WG, and he ran into many of the above issues when implementing character set providers for WebMethos (since consumed by SoftwareAG). He also referred me to a contact at ICU and I hope to hear from them in the next week or two. Meanwhile, any thoughts or suggestions you have on the above would be appreciated. While I am waiting for feedback from ICU and Addison I am trying to determine an effective way to set up an automated test harness that can be used to generate different combinations of strings, terminators and encodings and perform volume testing. Mike suggested using the test example he provided but it only showed one data string for input. That might be adequate for simple tests but, because test cases may need to be shared by multiple test XSD files it may not be scalable for volume testing or testing of multiple cases. Rick

2 1

XSL-style SmartLengthString Sample
by RPost 18 Jun '08

18 Jun '08

Hi, Attached is a doc that describes a (syntax-deprived) version of Mike's SmartLengthString example that is done more in a style that combines XML and XSL. It would use XML for the visible infoset and element declarations but an XSL style for the transforms. It attempts to: 1. Separate input/parse considerations from output/unparse. 2. Provide for all physical elements without requiring them to be in the visible infoset 3. Provide for multiple/nested layers 4. Allow for multiple transforms to be performed even within a single layer. I still have scalability concerns with using one outputValueCalc (or input) expression if multiple or complex transformations need to be done. The idea is to use a simple structure that can be nested to get the complexity that is needed. Here is an excerpt from the doc but the formatting will probably get out of whack: <xs:complexType name="smartLengthString" ...other attributes...> <xs:sequence> <hiddenStructure> -- the annotations and transforms between the physical and logical layers <hiddenElements> . . . one or more elements . . . </hiddenElements> <inputLayer> . . . The transforms and lower layer elements needed to produce the higher-layer Elements and ultimately the infoset informational values. </inputLayer> <outputLayer> . . . The transforms and higher layer elements needed to produce the lower-layer elements and ultimately the physical layer values. </outputLayer> </hiddenStructure> <visibleElements> -- the infoset as experienced by the user </xs:sequence> </xs:complexType> As you can see I took a lot of liberties with the syntax to try to make it easier to see the structure I am using. The 'hiddenStructure' element is what contains the equivalent of the <xs:annotation><xs:appinfo> from the example. The hidden elements are equivalent to elements in the example that are not in the public infoset. The inputLayer contains whatever is needed (including nested inputLayers) for parsing. Likewise for the outputLayer and unparsing. Here is a skeleton inputLayer: <inputLayer> <layerElements>  </layerElements> <transforms>  <transforms> <inputLayer>  </inputLayer> </inputLayer> There is a section for elements specific to this layer; somewhat akin to intermediate working variables or intermediate calculations. Another section is for an ordered list of transforms. This allows for as many or as few as needed but provides clean separation between them. The first transform might create the value of the 'lengthFlag' hidden element. The next transform could then use the lengthFlag value to extract the logicalLength value and so on. It sure seems to flow ok but maybe that is just in my head. The nested 'inputLayer' allows for a chain of transforms to be created. The nested inputLayer could create values for this layer's layerElements and get it's input from either another nested layer or from the physical layer itself. If the doc does nothing more than help solidify your thinking about how to get where you want to go it will have done it's job. By the way, thanks for the excellent example Mike. Rick

2 1

Invitation: OGF DFDL Working Group weekly call (18 Jun 14:00 GDT in DE2K09)
by Steve Hanson 16 Jun '08

16 Jun '08

Calendar Entry ---------------------------------------------------------------------------------------------------------------------------- Subject Chair OGF DFDL Working Group weekly call Steve Hanson/UK/IBM Invitees Date Required (to) 18/06/2008 Alan Powell/UK/IBM, mbeckerle(a)oco-inc.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Time Kimber/UK/IBM 14:00 - 15:00 (1 hour) Optional (cc) dfdl-wg(a)ogf.org Proposed start -1 at Proposed end -1 at Original start 18/06/2008 at 14:00 Original end 18/06/2008 at 15:00 Repeats Show detailed repeat information This meeting is scheduled to repeat Where Location DE2K09 Participant Passcode: 8190865 Country Toll Numbers Freephone/Toll Free Number AUSTRIA 43-1-928-9652 BELGIUM 32-2-400-6791 0800-4-9612 DENMARK 45-7014-0237 FRANCE LYON: 33-4-26-69-12-74 080-511-3725 FRANCE MARSEILLE: 33-4-86-06-00-74 080-511-3725 FRANCE PARIS: 33-1-70-70-81-58 080-511-3725 GERMANY 49-69-2222-20363 0800-100-6504 IRELAND 353-1-247-7768 ITALY 39-02-3600-3648 800-985-679 LUXEMBOURG 352-27-000-1312 NETHERLANDS 31-20-713-2741 0800-020-3237 SPAIN 34-91-414-1542 800-099-667 SWEDEN 46-8-566-10-780 SWITZERLAND 41-44-580-3334 0800-000-516 UNITED KINGDOM BIRMINGHAM: 44-121-210-9014 0800-279-9193 UNITED KINGDOM GLASGOW: 44-141-202-3214 0800-279-9193 UNITED KINGDOM LEEDS: 44-113-301-2114 0800-279-9193 UNITED KINGDOM LONDON: 44-20-7019-0808 0800-279-9193 UNITED KINGDOM MANCHESTER: 44-161-601-1414 0800-279-9193 USA 866-505-4412

1 0