Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 12/11/2013 13:55 ----- From: Steve Hanson/UK/IBM To: Alex Wood1/UK/IBM@IBMGB, Date: 12/11/2013 12:19 Subject: Re: decoding UTF-16 sequence with an unpaired surrogate in ICU. Thanks Alex. So we can control what ICU does in this scenario using dfdl:encodingErrorPolicy in the expected way, as the DFDL spec says. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Date: 12/11/2013 12:12 Subject: decoding UTF-16 sequence with an unpaired surrogate in ICU. So I coded a java program to test this in ICU4J So when decoding in ICU it seems to class an unpaired UTF-16 surrogate as malformed input. ICU API allows the programmer to specify the behaviour for malformed input. ignore, replace or report the offending code point. default is to report it and therefore the decode would fail with an error. the ICU4C api has similar options available. test program: public class test1 { /** * @param args */ public static void main(String[] args) { // TODO Auto-generated method stub final byte[] byteArray = { (byte) 0xD8, 0x34, (byte) 0xDD, 0x1E, (byte) 0xD8, 0x34}; CharsetProvider cp = new CharsetProviderICU(); CharsetDecoder decoder = cp.charsetForName("UTF-16" ).newDecoder(); decoder.onMalformedInput(CodingErrorAction.IGNORE); decoder.reset(); ByteBuffer bb = ByteBuffer.wrap(byteArray, 0, 6); CharBuffer cb = CharBuffer.allocate(6); CoderResult decodeResult = decoder.decode(bb, cb, true); if (decodeResult.isMalformed() || decodeResult.isUnmappable()) { System.out.println("Error at " + bb.position() ); } System.out.println("Result" + cb.toString() ); } } Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda@uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU