[DFDL-WG] Action 236: decoding UTF-16 sequence with an unpaired surrogate in ICU.

12 Nov 2013

      Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
----- Forwarded by Steve Hanson/UK/IBM on 12/11/2013 13:55 -----

From:   Steve Hanson/UK/IBM
To:     Alex Wood1/UK/IBM@IBMGB, 
Date:   12/11/2013 12:19
Subject:        Re: decoding UTF-16 sequence with an unpaired surrogate in 
ICU.

Thanks Alex. 

So we can control what ICU does in this scenario using 
dfdl:encodingErrorPolicy in the expected way, as the DFDL spec says.

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848

From:   Alex Wood1/UK/IBM
To:     Steve Hanson/UK/IBM@IBMGB, 
Date:   12/11/2013 12:12
Subject:        decoding UTF-16 sequence with an unpaired surrogate in 
ICU.

So I coded a java program to test this in ICU4J

So when decoding in ICU it seems to class an unpaired UTF-16 surrogate as 
malformed input.

ICU API allows the programmer to specify the behaviour for malformed 
input.

ignore, replace or report the offending code point. 

default is to report it and therefore the decode would fail with an error.

the ICU4C api has similar options available.

test program:

public class test1 {

        /**
         * @param args
         */
        public static void main(String[] args) {
                // TODO Auto-generated method stub

                final byte[] byteArray = { (byte) 0xD8, 0x34, (byte) 0xDD, 
0x1E, (byte) 0xD8, 0x34};

                CharsetProvider cp = new CharsetProviderICU();

                CharsetDecoder decoder = cp.charsetForName("UTF-16"
).newDecoder();
                decoder.onMalformedInput(CodingErrorAction.IGNORE);
                decoder.reset();
                ByteBuffer bb = ByteBuffer.wrap(byteArray, 0, 6);
                CharBuffer cb = CharBuffer.allocate(6);
                CoderResult decodeResult = decoder.decode(bb, cb, true);

                if (decodeResult.isMalformed() || 
decodeResult.isUnmappable()) {
                        System.out.println("Error at " + bb.position() );
                } 
                System.out.println("Result" + cb.toString() );

        }
}

Kind Regards,

- Alex

Alex Wood - 
Software Engineer - 
WebSphere Message Broker Development
DFDL Development

MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN.
Tel: Internal 246272, External 01962 816272
Notes: Alex Wood1/UK/IBM@IBMGB
e-mail: wooda@uk.ibm.com

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU