October 2013 - dfdl-wg - lists.ogf.org

Fw: Public comment 116 - Japanese CCSID 943
by Steve Hanson 22 Oct '13

22 Oct '13

Suggested words added to the public comment. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 08/10/2013 10:08 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 07/10/2013 19:52 Subject: Re: [DFDL-WG] Public comment 116 - Japanese CCSID 943 I suggest that 100% ascii compatible is too strong a statement and we can dial this back. Example: utf-8 will not handle asciiTandemModified style as that style uses the most-significant bit for the 0x80-0x89 negative sign + digit. That is, asciiTandemModified style is making use of extended-ascii codepoints, and there are many flavors of extended ascii with many different assignments of characters to codepoints for this range 80-89. So the question arises of who's in charge, the characters, or the codepoints? I.e., does the zoned sign style even care about the encoding, or does it translate the associated codepoints without concern for their mappings to actual characters. I think the codepoints should be in charge. So long as you have an encoding where all the characters used by your zoned decimals (as digits, or digits with overpunched signs) have legal codepoints in that encoding, you are good to go. If in one charset encoding one of the characters is assigned the euro symbol, and in other charset encodings that same codepoint is assigned some other character, then it should not matter so long as the codepoint doesn't cause a decode error. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Mon, Oct 7, 2013 at 9:30 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Please see below discussion of the issue raised by public comment 116. While there is a simple workaround, the real issue is that ccsid 943 is in daily use in Japan for zoned decimals and DFDL does not support that. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 07/10/2013 14:05 ----- ICU library treats the ccsid '943' as ICU ibm-943_P130-1999 which is not 100% ASCII compatible due to two code points being different - 0x5C and 0x7E. There is another encoding ICU ibm-943_P15A-2003 which is ASCII compatible, commonly called Shift_JIS. The difference is one that you are probably familiar with - the backslash is replaced by the Yen symbol. Here is the extract from ICU converter site: Internal Converter Name IBM IANA ibm-943_P15A-2003 Shift_JIS MS_Kanji csShiftJIS windows-31j csWindows31J ibm-943_P130-1999 ibm-943 This causes DFDL to reject the ccsid '943' when used on a zoned decimal on the grounds that it is not ASCII-compatible. Why does DFDL do this? It's because it is being safe. So far, we have identified 4 different 'overpunching' schemes for ASCII zoned decimals, and there might well be one or two more used by less common machine architectures: asciiStandard: ASCII characters '0123456789' (0x30-0x39) and 'pqrstuvwxy' (0x70-0x79) for negative sign punch. asciiTranslatedEBCDIC: ASCII characters '{ABCDEFGHI' (0x7B, 0x41-0x49) and '}JKLMNOPQR' (0x7D, 0x4A-0x52) for negative sign punch. asciiCARealiaModified: ASCII characters '0123456789' (0x30-0x39) and '<SP>!"#$%&'()' (0x20-0x29) for negative sign punch. asciiTandemModified: ASCII characters '0123456789' (0x30-0x39) and control characters 0x80-0x89 for negative sign punch. In case other schemes are discovered that use different byte range for negative sign punch (eg, 0x50 to 0x59), the DFDL specification has said that ASCII zoned decimals must be in a 100% ASCII compatible encoding. What we can observe is that ibm-943_P130-1999 is actually safe for representing ASCII zoned decimals in all the above schemes, because the 0x5C and 0x7E characters do not match any of the ranges of bytes used by the schemes. And we can also observe that (apart from the special case of asciiTranslatedEBCDIC) the overpunching schemes simply use the bits in the first nibble of the byte, so any new scheme we discover is very unlikely to affect 0x5C or 0x7E. So, the DFDL specification *could* be changed to treat ibm-943_P130-1999 as ASCII compatible for zoned decimals. The alternative is to use a work around whereby in a DFDL schema that models a data stream in ccsid 943, the default is 943 but zoned decimals override this and use Shift_JIS. When this workaround was discussed with a Japanese user of IBM DFDL, the reaction was "Why do I have to go to that trouble? It should just work in 943." Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Rescheduled: OGF DFDL WG Call (23 Oct 16:00 GDT in DM2046)
by Steve Hanson 21 Oct '13

21 Oct '13

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Rescheduling as Mike and Jonathan can't make Tuesday this week

1 0

How do I preserve order in an unordered list?
by Garriss Jr., James P. 16 Oct '13

16 Oct '13

(Sent on behalf of Roger Costello) Hi Folks, I am trying out the sequenceKind="unordered" property. I created this simple test: <xs:element name="Test"> <xs:complexType> <xs:sequence dfdl:sequenceKind="unordered" dfdl:separator="%SP;"> <xs:element name="A" type="xs:string" dfdl:initiator="A:" /> <xs:element name="B" type="xs:string" dfdl:initiator="B:" /> </xs:sequence> </xs:complexType> </xs:element> That schema says the input data must consist of A:___ and B:___, in any order. Here is sample input: B:Cat A:Dog I processed the input using the schema and here is the result that I got from Daffodil: <Test> <A>Dog</A> <B>Cat</B> </Test> Notice that the order of the input data changed in the result XML. This was quite surprising to me. Upon consulting the DFDL specification, it appears that the exhibited behavior is expected: ... a DFDL processor must sort the members of an unordered group into schema order when parsing. QUESTIONS: 1. Is this really the desired behavior? I would not expect parsing to alter the order of any data. From my XML Schema experience, I would be shocked if an XML Schema validator altered the order of markup in XML instances simply because the XML Schema specified <all> (unordered sequence). 2. If we grant that this really is the desired behavior, then how do I create an unordered sequence in which DFDL parsing preserves the order of the data? In the above example, if the input data lists B:Cat first and A:Dog second, then how do I get that order preserved in the result XML? /Roger

4 5

Fw: [wg-all] New documents in public comment - last day!
by Steve Hanson 15 Oct '13

15 Oct '13

Last day for public comments on the 3 DFDL documents below... Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 15/10/2013 16:16 ----- From: Greg Newby <gbnewby(a)alaska.edu> To: All WG <wg-all(a)ogf.org>, Date: 16/09/2013 19:37 Subject: [wg-all] New documents in public comment Sent by: wg-all-bounces(a)ogf.org Dear OGF Community: New documents have begun public comment. Please consider reading and providing feedback. Brief affirmations are very useful to demonstrate community input and support. More detailed analysis is, of course, welcome. All documents in public comment are accessible here: http://redmine.ogf.org/projects/editor-pubcom/boards Note that this is an updated location, using OGF's new Redmine tracker system. Anyone may register for a Redmine username. These documents are newly available for public comment: * DFDL v1.0 Revision, recommendation track, via DFDL-WG * DFDL v1.0 Experience, experimental track, via DFDL-WG * DFDL v1.0 Rev. Experience, experimental track, via DFDL-WG This document was previously made available for public comment, and is still available for comment: * Grid Certificate Profile, recommendation track, via CAOPS-WG -- Greg Newby, OGF Standards Editor Dr. Gregory Newby, Director of the Arctic Region Supercomputing Center Univ of Alaska Fairbanks-909 Koyukuk Dr-PO Box 756020-Fairbanks-AK 99775-6020 e: gbnewby(a)alaska.edu v: 907-450-8663 f: 907-450-8603 w: people.arsc.edu/~newby -- The wg-all mailing list is an umbrella list for all OGF mailing lists. If you are subscribed to any OGF mailing list, you will receive mails to wg-all. wg-all mailing list wg-all(a)ogf.org https://www.ogf.org/mailman/listinfo/wg-all Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Cancelled: OGF DFDL WG Call
by Steve Hanson 15 Oct '13

15 Oct '13

Mike not available

1 0

OGF DFDL WG Call Agenda 2013-10-15
by Steve Hanson 15 Oct '13

15 Oct '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13138?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Minutes 2013-10-08
by Steve Hanson 08 Oct '13

08 Oct '13

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13136?download= Public Comment The following set are now in Public Comment until 15th October. Revised DFDL 1.0 Specification: https://redmine.ogf.org/issues/117 DFDL Experience Document 1 (Spec Errata): https://redmine.ogf.org/issues/118 DFDL Experience Document 2 (Action 140): https://redmine.ogf.org/issues/119 Please download, review and create comments as necessary. Only 1 week left! Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

bit order - proposed new erratum - response to public comment 43, action item 234
by Mike Beckerle 08 Oct '13

08 Oct '13

Attached is a MS-Word document containing my first cut at an errata to describe adding a new bitOrder property to DFDL. It is more complicated than I would like. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call Agenda 2013-10-08
by Steve Hanson 08 Oct '13

08 Oct '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13136?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] public comment "Non-deterministic outcome of setVariable evaluations" (Action 229)
by Mike Beckerle 07 Oct '13

07 Oct '13

Revised proposal that fully specifies evaluation order for setVariable and newVariableInstance. --------------------------------------------- Errata 3.20, 3.21, and Errata 3.25 must be modified (and the corresponding sections of the spec) For errata 3.20, and errata 3.21 the sentence: "However, the order of execution among them is not specified. Schema authors can insert sequences to control the timing of evaluation of statements more precisely." is replaced by: "The order of execution is specified in errata 3.25." Errata 3.25 is modified. Numbered item 3 of both the first list and second list is replaced by: 3. dfdl:setVariable(s) - in lexical order, innermost schema component first Numbered item 2 of the second list is replaced by: 2. dfdl:newVariableInstance(s) - in lexical order, innermost schema component first These two paragraphs are added below the numbered lists to clarify what 'lexical order, innermost schema component first' means. The dfdl:setVariable annotations at any one annotation point of the schema are always executed in lexical order. However, dfdl:setVariable annotations can also be found in different annotation points that are combined into the resolved set of annotations for one schema component. In this case, the order of execution of the dfdl:setVariable statements from any one annotation point remains lexical. The order of execution of the dfdl:setVariable annotations different annotation points follows the principle of innermost first, meaning that a schema component that references another schema component has its dfdl:setVariable statements executed after those of the referenced schema component. For example, if an element reference and an element declaration both have dfdl:setVariable statements, then those on the element declaration will execute before those on the element reference. Similarly dfdl:setVariable statements on a base simple type execute before those of a simple type derived from it. The dfdl:setVariable statements on a simple type execute before those on an element having that simple type (whether by reference, or when the simple type is lexically nested within the element declaration). The dfdl:setVariable statements on the sequence or choice within a global group definition execute before those on a group reference. The dfdl:newVariableInstance annotations at any one annotation point of the schema are always executed in lexical order. However, dfdl:newVariableInstance annotations can also be found in different annotation points that are combined into the resolved set of annotations for one schema component. In this case, the order of execution of the dfdl:newVariableInstance statements from any one annotation point remains lexical. The order of execution of the dfdl:newVariableInstance annotations different annotation points follows the principle of innermost first, meaning that a schema component that contains or references another schema component has its dfdl:newVariableInstance statements executed after those of the contained or referenced schema component. For example, if a group reference and the sequence or choice group of a group definition both have dfdl:newVariableInstance statements, then those on the global group definition will execute before those on the group reference. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

1 0