October 2013 - dfdl-wg - lists.ogf.org

Public comment 116 - Japanese CCSID 943
by Steve Hanson 07 Oct '13

07 Oct '13

Please see below discussion of the issue raised by public comment 116. While there is a simple workaround, the real issue is that ccsid 943 is in daily use in Japan for zoned decimals and DFDL does not support that. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 07/10/2013 14:05 ----- ICU library treats the ccsid '943' as ICU ibm-943_P130-1999 which is not 100% ASCII compatible due to two code points being different - 0x5C and 0x7E. There is another encoding ICU ibm-943_P15A-2003 which is ASCII compatible, commonly called Shift_JIS. The difference is one that you are probably familiar with - the backslash is replaced by the Yen symbol. Here is the extract from ICU converter site: Internal Converter Name IBM IANA ibm-943_P15A-2003 Shift_JIS MS_Kanji csShiftJIS windows-31j csWindows31J ibm-943_P130-1999 ibm-943 This causes DFDL to reject the ccsid '943' when used on a zoned decimal on the grounds that it is not ASCII-compatible. Why does DFDL do this? It's because it is being safe. So far, we have identified 4 different 'overpunching' schemes for ASCII zoned decimals, and there might well be one or two more used by less common machine architectures: asciiStandard: ASCII characters '0123456789' (0x30-0x39) and 'pqrstuvwxy' (0x70-0x79) for negative sign punch. asciiTranslatedEBCDIC: ASCII characters '{ABCDEFGHI' (0x7B, 0x41-0x49) and '}JKLMNOPQR' (0x7D, 0x4A-0x52) for negative sign punch. asciiCARealiaModified: ASCII characters '0123456789' (0x30-0x39) and '<SP>!"#$%&'()' (0x20-0x29) for negative sign punch. asciiTandemModified: ASCII characters '0123456789' (0x30-0x39) and control characters 0x80-0x89 for negative sign punch. In case other schemes are discovered that use different byte range for negative sign punch (eg, 0x50 to 0x59), the DFDL specification has said that ASCII zoned decimals must be in a 100% ASCII compatible encoding. What we can observe is that ibm-943_P130-1999 is actually safe for representing ASCII zoned decimals in all the above schemes, because the 0x5C and 0x7E characters do not match any of the ranges of bytes used by the schemes. And we can also observe that (apart from the special case of asciiTranslatedEBCDIC) the overpunching schemes simply use the bits in the first nibble of the byte, so any new scheme we discover is very unlikely to affect 0x5C or 0x7E. So, the DFDL specification *could* be changed to treat ibm-943_P130-1999 as ASCII compatible for zoned decimals. The alternative is to use a work around whereby in a DFDL schema that models a data stream in ccsid 943, the default is 943 but zoned decimals override this and use Shift_JIS. When this workaround was discussed with a Japanese user of IBM DFDL, the reaction was "Why do I have to go to that trouble? It should just work in 943." Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Call For Abstract: NIST Data Science Symposium
by Steve Lawrence 04 Oct '13

04 Oct '13

I'm just letting the working group know that we are submitting an abstract today to give a presentation on DFDL at the NIST Data Science Symposium. Below is what we plan to submit. - Steve ---------------------------------------------- Title: Stop Writing Custom Data Parsers -- Write DFDL Instead! This talk gives an introduction to the Data Format Description Language (DFDL), how it can be used to parse both textual and binary data in a standardized way, and how this leads to less time spent on custom data parser development and consequently, more time spent on data processing and analysis. The talk will then describe the current DFDL implementations, with focus on the open-source Daffodil project and its design. It will conclude with a brief walkthrough of real DFDL examples, including commercial and scientific formats, and a demonstration of the parsing capabilities of Daffodil. The DFDL specification, which has completed a second round of public comments as part of the Open Grid Forum (OGF), is a modeling language for describing general text and binary data using a subset of XML Schema augmented with data format annotations. DFDL allows data to be read from its native format and presented as an instance of an information set or an XML document. DFDL also allows the reverse, through conversion of an information set back to its native format. By using the information set, this cleanly integrates with common XML utilities (e.g. XProc, XSLT, XQuery) for data processing and analysis regardless of the format of the native data. Two implementations of DFDL exist, as is required by the OGF to become a standard. The first, created by IBM as part of IBM WebSphere V8, is written in both Java and C and includes graphical tools for modeling, running, and debugging DFDL schemas. The second implementation, Daffodil, is an open-source project written in Scala, with a design focused on speed and correctness. With the two implementations making great strides, and the DFDL specification nearing standardization, DFDL is becoming a promising tool that will ease data parsing, processing, and analysis. Biography: Stephen Lawrence has worked as a software engineer at Tresys Technology since 2007, while contributing to the open-source Daffodil project as a core maintainer for almost two years. He works alongside Michael Beckerle, the co-chair of the DFDL Working Group, to develop Daffodil and improve the DFDL specification. Outside of Daffodil, he focuses on computer security applications, including file inspection and sanitization, Security Enhanced Linux (SELinux), and cross domain solutions.

2 2

OGF DFDL WG Call Minutes 2013-10-01
by Steve Hanson 02 Oct '13

02 Oct '13

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13131?download= Public Comment The following set are now in Public Comment until 15th October. Revised DFDL 1.0 Specification: https://redmine.ogf.org/issues/117 DFDL Experience Document 1 (Spec Errata): https://redmine.ogf.org/issues/118 DFDL Experience Document 2 (Action 140): https://redmine.ogf.org/issues/119 Please download, review and create comments as necessary. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

public comment "Non-deterministic outcome of setVariable evaluations"
by Mike Beckerle 01 Oct '13

01 Oct '13

Some email discussion before posting to the public comment forum may be helpful for this issue. Right now the rules say that all dfdl:newVariableInstance annotations for a schema component are always executed before any setVariable annotations. This is ok. One can introduce an extra sequence in order to allow a setVariable to occur before a newVariableInstance. (Personally: I would like to get a warning from my DFDL implementation if I write statement annotations at one annotation point that are not in the canonical order described in Errata 3.25) The problem is that the order of execution of multiple setVariable statements at one annotation point, or similarly, of multiple newVariableInstance at one annotation point, is not specified. This is what will get users into trouble. If there are multiple newVariableInstance annotations, then the order is important to understanding when exactly a new instance comes into scope, and whether the prior-scope's value for that variable can be read as part of the default value expressions. Consider this: <dfdl:newVariableInstance ref="save">{ $delim }</dfdl:newVariableInstance> <dfdl:newVariableInstance ref="delim">{ ../hdr/d }</dfdl:newVariableInstance> We need to specify whether $delim in the first annotation is referring to that variable before the new instance is created, or after. I suggest that before is more intuitive, as that is lexical order. Note however, that this means: <dfdl:newVariableInstance ref="prefix">{ fn:concat($prefix, "_", fn:name()) }</dfdl:newVariableInstance> is allowed and is not recursion. The expression is referring to $prefix from the enclosing scope, and then initializing a new instance of $prefix for the scope contained by this schema component. The same logic applies to dfdl:setVariable annotations. The order needs to be controlled by the user. I suggest the fix for this issue is as follows: Errata 3.20, 3.21, and Errata 3.25 must be modified (and the corresponding sections of the spec) For errata 3.20, and errata 3.21 the sentence: "However, the order of execution among them is not specified. Schema authors can insert sequences to control the timing of evaluation of statements more precisely." is replaced by: "The order of execution is specified in errata 3.25." Errata 3.25 is modified. Numbered item 3 of both the first list and second list is replaced by: 3. dfdl:setVariable(s) - in lexical order Numbered item 2 of the second list is replaced by: 2. dfdl:newVariableInstance(s) - in lexical order These two paragraphs are added below the numbered lists to clarify what 'lexical order' means. The dfdl:setVariable annotations at any one annotation point of the schema are always executed in lexical order. However, dfdl:setVariable annotations can also be found in different annotation points that are combined into the resolved set of annotations for one schema component. In this case, the order of execution of the dfdl:setVariable statements from any one annotation point remains lexical, but it is unspecified what the order of execution is across dfdl:setVariable statements coming from different annotation points. The dfdl:newVariableInstance annotations at any one annotation point of the schema are always executed in lexical order. However, dfdl:newVariableInstance annotations can also be found in different annotation points that are combined into the resolved set of annotations for one schema component. In this case, the order of execution of the dfdl:newVariableInstance statements from any one annotation point remains lexical, but it is unspecified what the order of execution is across dfdl:newVariableInstance statements coming from different annotation points. ---------------------- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 2

OGF DFDL WG Call Agenda 2013-10-01
by Steve Hanson 01 Oct '13

01 Oct '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13131?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0