|
072
| TK: Byte Order Mark and Unicode
signature
16/12: Investigate whether the spec's position on UTF-16/32 BOM is implementable |
The implementation team have carried
out tests on the Java and C implementations of ICU. The results are:
Java ICU libraries
| Encoding
| Input
| BOM included in decoded string?
|
| UTF-8
| <BOM>AAA
| yes
|
| UTF-16
| <BOM>AAA
| yes
|
| UTF-16-LE
| <BOM>AAA
| yes
|
| UTF-16-BE
| <BOM>AAA
| yes
|
| UTF-32
| <BOM>AAA
| no
|
| UTF-32-LE
| <BOM>AAA
| no
|
| UTF-32-BE
| <BOM>AAA
| no |
C ICU libraries:
| Encoding
| Input
| BOM included in decoded string?
|
| UTF-8
| <BOM>AAA
| yes
|
| UTF-16
| <BOM>AAA
| yes
|
| UTF-16-LE
| <BOM>AAA
| yes
|
| UTF-16-BE
| <BOM>AAA
| yes
|
| UTF-32
| <BOM>AAA
| yes
|
| UTF-32-LE
| <BOM>AAA
| yes
|
| UTF-32-BE
| <BOM>AAA
| yes |
I suspect that the UTF-32 anomaly is
a defect in ICU. I tried to confirm this using Google, but I didn't find
any reference to it online.
Before we conclude that the spec is
OK as it stands, we should consider whether it is correct to treat a BOM
as a character. The Unicode standard makes a clear distinction between
characters and BOMs:
http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf
section 2.13 ( don't skip the first couple of paragraphs )
http://unicode.org/faq/utf_bom.html#bom1
regards,
Tim Kimber, Common Transformation Team,
Hursley, UK
Internet: kimbert@uk.ibm.com
Tel. 01962-816742
Internal tel. 246742
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU