November 2013 - dfdl-wg - lists.ogf.org

Action 236: decoding UTF-16 sequence with an unpaired surrogate in ICU.
by Steve Hanson 12 Nov '13

12 Nov '13

Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 12/11/2013 13:55 ----- From: Steve Hanson/UK/IBM To: Alex Wood1/UK/IBM@IBMGB, Date: 12/11/2013 12:19 Subject: Re: decoding UTF-16 sequence with an unpaired surrogate in ICU. Thanks Alex. So we can control what ICU does in this scenario using dfdl:encodingErrorPolicy in the expected way, as the DFDL spec says. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Date: 12/11/2013 12:12 Subject: decoding UTF-16 sequence with an unpaired surrogate in ICU. So I coded a java program to test this in ICU4J So when decoding in ICU it seems to class an unpaired UTF-16 surrogate as malformed input. ICU API allows the programmer to specify the behaviour for malformed input. ignore, replace or report the offending code point. default is to report it and therefore the decode would fail with an error. the ICU4C api has similar options available. test program: public class test1 { /** * @param args */ public static void main(String[] args) { // TODO Auto-generated method stub final byte[] byteArray = { (byte) 0xD8, 0x34, (byte) 0xDD, 0x1E, (byte) 0xD8, 0x34}; CharsetProvider cp = new CharsetProviderICU(); CharsetDecoder decoder = cp.charsetForName("UTF-16" ).newDecoder(); decoder.onMalformedInput(CodingErrorAction.IGNORE); decoder.reset(); ByteBuffer bb = ByteBuffer.wrap(byteArray, 0, 6); CharBuffer cb = CharBuffer.allocate(6); CoderResult decodeResult = decoder.decode(bb, cb, true); if (decodeResult.isMalformed() || decodeResult.isUnmappable()) { System.out.println("Error at " + bb.position() ); } System.out.println("Result" + cb.toString() ); } } Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Further progress on action 235: Request clarifications of the Escape Schemes.
by Steve Hanson 12 Nov '13

12 Nov '13

Some extra notes added by Steve to Mike's original answers. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 12/11/2013 12:14 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB, Date: 23/10/2013 17:02 Subject: Partial progress on action 235: Request clarifications of the Escape Schemes. Questions from Taylor Wise: 1. Does any character effectively escape the block start or is the block start inside the data a syntax error (is a valid escape block start only one that appears at the beginning of the data?). 2. Does an end block have to be followed by the delimiter (optionally padding first) or does the absence of a delimiter mean that it is not an end block? 3. Without an escape block start are the escape escape characters still interpreted. 4. Does extra escape characters require escape kind = escapeCharacter. 5. What is the appropriate behavior for the following: Assuming escapeBlockStart="Start", escapeBlockEnd="END", escapeEscapeCharacter="!" ,4StartStart1!!23END4!ENDEND, ( comma is delimiter) ---------------------------------------------------------------- Answers: 1. Yes 2. Yes - no lookahead. To be clear: There may not be a delimiter. When following a block start, the block end, not preceded by an escape escape character, is always interpreted as ending the content region. It may be followed by a delimiter if that is what is expected in the model; however, there is no lookahead for the delimiter or anything else. For an element with dfdl:lengthKind='delimited', it is a processing error if the block end is not followed by optional padding and a delimiter. 3. No - without a block start nothing will be interpreted as an escape escape character nor as a block end. 4. No - For escapeKind="escapeBlock" presence of any of the extra escaped characters in the data implies that the data must be surrounded by the block start and block end when unparsing. This is stated in the spec. See dfdl:generateEscapeBlock. 5. <SMH>Taking the example above: ,4StartStart1!!23END4!ENDEND, ( comma is delimiter). a) If the leading '4' is not trimmed as a padding character, then the escape block start is not treated as such because it is not at the start of the data, so the infoset contains '4StartStart1!!23END4!ENDEND' - no escaping is applied. b) If the leading '4' is trimmed as a padding character, then the first 'Start' is treated as escape block start, and the first unescaped 'END' is treated as escape block end. The '4' after 'END' may also be trimmed as a padding character if justification is 'center'. But the first '!' will cause a processing error, because the next character is expected to be the ',' delimiter. c) If the data was instead ',4StartStart1!!23!END!ENDEND,' and the leading '4' is trimmed' as per b) then the first two occurrences of 'END' are escaped by the '!' and the last 'END' is treated as the escape block end. The infoset contains 'Start1!!23ENDEND' (because spec says the escape escape character is not removed when it does not precede the escape block end **).</SMH> The definition of escapeKind for escapeBlock needs clarification, because it implies one can isolate the data without interpreting the block start and block end. For delimited formats, the block start and block end are integral to identification of the delimiter. <SMH> Agree. And it's just not delimited formats. The text needs to be processed from start to finish to handle the escape escape character. </SMH> Need to clarify that the escape escape character does not apply to the block start ever. Consider expressing this with a small grammar. ** <SMH>Is this really correct, or should the escape escape character always be removed? </SMH> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2013-11-12
by Steve Hanson 11 Nov '13

11 Nov '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13154?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Analysis of bitOrder and byteOrder (public comment: byteOrder not sufficient to implement MIL-STD-2045-47001D)
by Mike Beckerle 11 Nov '13

11 Nov '13

I have done a fairly thorough analysis of the bit-order issue. Bit order only matters when an element constructs a value from part of a byte, or spans bytes, with a partial byte somewhere on one end, the other, or both. The attached spreadsheet walks through all the cases. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2013-11-05 - updated
by Steve Hanson 07 Nov '13

07 Nov '13

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13151?download= Minutes updated to cover extra WG call on 2013-11-07. Public Comment for the revised DFDL 1.0 specification is now finished. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Count is 14 unresolved/undeferred public comments
by Mike Beckerle 07 Nov '13

07 Nov '13

I just marked some of the simple public comments (like broken links, typos) as Resolved - will fix. I also marked the issues that have action items and are not deferred with "Action NNN" in their subject line. What remains to discuss are 14 public comments that are neither resolved, deferred, nor have action items assigned. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2013-11-05
by Steve Hanson 06 Nov '13

06 Nov '13

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13151?download= Public Comment for the revised DFDL 1.0 specification is now finished. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Public comment 25 / action 227
by Steve Hanson 05 Nov '13

05 Nov '13

Resending with corrections after review.... Erratum 2.100 clarified that such an element has variable length when unparsing. Sections 12.3.7 and dfdl:textPadKind have not been fully updated to make this clear. Attached Word document has marked up changes to correct the above, plus some comments. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] possible editorial issue - Schema Definition Warning
by Cranford, Jonathan W. 05 Nov '13

05 Nov '13

On second thought, instead of a Schema Definition Warning, could this be a recoverable error instead? That is, would the following convey the originally intended meaning better? Note: Unrecognized DFDL properties or property values can produce a recoverable error, and an implementation can attempt to process data despite the error. IIRC, this section on specific errors was added *before* recoverable errors were added to the spec. Thoughts? -Jonathan >-----Original Message----- >From: Cranford, Jonathan W. >Sent: Tuesday, November 05, 2013 8:30 AM >To: dfdl-wg(a)ogf.org >Subject: possible editorial issue - Schema Definition Warning > >The phrase "Schema Definition Warning" only occurs once in the published 1.0.4 >spec (without respect to capitalization), in Section 2.6 Specific Errors Classified: > > "Note: Unrecognized DFDL properties or property values can produce a >Schema Definition Warning and an implementation can attempt to process data >despite the warning." > >Is the concept of a Schema Definition Warning a nascent concept already present >in the spec but not well-defined? If so, this might be a minor issue that warrants >an errata. My recommendation would be that Schema Definition Warning be >defined and used consistently throughout the document. > >Otherwise, I recommend that "Schema Definition Warning" be replaced in the >above sentence with simply "warning". In this case, this is probably just an >editorial issue. > >Very respectfully, > >-- >Jonathan W. Cranford >Senior Information Systems Engineer >The MITRE Corporation (http://www.mitre.org)

2 1

possible editorial issue - Schema Definition Warning
by Cranford, Jonathan W. 05 Nov '13

05 Nov '13

The phrase "Schema Definition Warning" only occurs once in the published 1.0.4 spec (without respect to capitalization), in Section 2.6 Specific Errors Classified: "Note: Unrecognized DFDL properties or property values can produce a Schema Definition Warning and an implementation can attempt to process data despite the warning." Is the concept of a Schema Definition Warning a nascent concept already present in the spec but not well-defined? If so, this might be a minor issue that warrants an errata. My recommendation would be that Schema Definition Warning be defined and used consistently throughout the document. Otherwise, I recommend that "Schema Definition Warning" be replaced in the above sentence with simply "warning". In this case, this is probably just an editorial issue. Very respectfully, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

1 0