Many thanks Mike - this is a valuable improvement
in the consistency and accuracy of this important area.
regards,
Tim Kimber, DFDL Team,
Hursley, UK
Internet: kimbert@uk.ibm.com
Tel. 01962-816742
Internal tel. 37246742
From:
Mike Beckerle <mbeckerle.dfdl@gmail.com>
To:
dfdl-wg@ogf.org,
Date:
29/01/2013 19:59
Subject:
Re: [DFDL-WG]
Glossary items needed (for a v12 errata?)
Sent by:
dfdl-wg-bounces@ogf.org
Revised per discussion on call 2013-01-29.
We agreed to change our terminology to align with the Unicode standard
terminology. Alas, it isn't that simple, as Unicode's terminology is a
little incompatible with XML terminology ('encoding'), and with IANA terminology
"character set". The Unicode terminology also draws distinctions
that we don't really need.
I have also retained the term 'Character code' to mean the cannonical unicode
integer for a character, which is the same as the ISO10646 code point for
a character.
All appearances of "codepoint" will change to "code unit"
consistent with the Unicode glossary. We do not need the term "Code
Point" after that, so I've dropped it.
The term 'code page' appears only once in the standard, and can be
changed to 'character set encoding' there. So I suggest we stick with CCSID,
and drop the term 'code page'.
Here is the revised set of definitions in alphabetical order:
CCSID - see Coded Character Set Identifier
Character - A ISO10646 character having a unique character code
as its identifier. This concept is independent of font, typeface, size,
and style, so 'F',
'F', 'F', are all the same character 'F'
Character Code - The canonical integer used to identify a character in
the ISO10646 standards. This number identifies the character, but can be
independent of any specific character set encoding of the character. Example:
The '{' character known in Unicode as LEFT CURLY BRACKET. Has character
code U+007B. However, depending on the character set encoding, the
value 0x7B may or may not appear in the representation of that character.
Character Set - An abstract set of characters that are assigned (or mapped
to) a representation by a particular character set encoding.
For most character set encodings their character set is a subset of the
Unicode character set.
Character Set Encoding - Often abbreviated to just 'encoding'. A specific
representation of a character set as bytes or bits of data. A character
set encoding is usually identified by a standard character set encoding
name or a recognized alias name, or by a coded character set identifier
or CCSID. These identifiers are standardized. The names and aliases
are standardized by the IANA (where unfortunately, they are called character
set names). CCSIDs are an industry standard. Examples of character set
encoding names are UTF-8, USASCII, GB2312, ebcdic-cp-it, ISO-8859-5,
UTF-16BE, Shift_JIS. The DFDL standard allows for implementation-specific
character set encodings to be supported, and standardizes one name that
is DFDL-specific which is USASCII-7bit-packed.
Character Width - The number of code units or alernatively the number of
bytes used to represent a character in a specific character set encoding
is called the character width. Encodings are either fixed width (all characters
encoded using the same width), or variable-width (different characters
are encoded using different widths). For example the UTF-32 character set
encoding has 4-byte character width, whereas USASCII has a 1-byte character
width. UTF-8 is variable width, and any specific character has width 1,
2, 3, or 4 bytes.
Code Unit - When a character set encoding uses differing variable width
representations for characters, the units making up these variable width
representations are called code units. For example the UTF-8 encoding
uses between 1 and 4 code units to represent characters, and for UTF-8,
the individual code units are single bytes. DFDL's interpretation of the
UTF-16 encoding is either fixed or variable width. When format property
dfdl:utf16Width='variable' then UTF-16 is variable width and this encoding
uses either one or two code units per character, but in this case each
individual code unit is a 16-bit value. When a character set is fixed width,
then there is no distinction between a code unit and a code point.
Coded Character Set Identifier (CCSID) - An alternate identifier of a character
set encoding. Originally created by IBM, CCSIDs are a broadly used industry
standard.
Encoding - See Character Set Encoding
Fixed-Width Character Encoding - A character set encoding where all characters
are encoded using a single code unit for their representation. Note that
a code unit is not necessarily a single byte.
Surrogate Pair - A Unicode character whose character code value is greater
than 0xFFFF can be encoded into variable-width UTF-16BE or UTF-16LE (which
are variable-width encodings when the DFDL property utf16Width='variable').
In this case the representation uses two adjacent code units each
of which is called a surrogate, and the pair of which is called a surrogate
pair.
Unicode - A character set defined by the Unicode Consortium, and standardized
at the International Standards Organization (ISO) as ISO10646.
Variable-Width Character Encoding - A character set encoding where characters
are encoded using one or more code units for their representation depending
on which specific character is being encoded. An example is UTF-8 which
uses from 1 to 4 bytes to encode a character.
On Mon, Jan 28, 2013 at 10:57 AM, Mike Beckerle <mbeckerle.dfdl@gmail.com>
wrote:
We need to add some entries associated with character set and encoding
terminology that we use quite a bit.
I would note that our usage of the term 'codepoint' differs somewhat from
the Unicode Glossary: http://unicode.org/glossary.
First, we use codepoint as one word not "code point" (there was
some inconsistency on this that I have now fixed), second, what we call
codepoint is closer to what Unicode Glossary calls 'code unit'. I suspect
we should just provide our definitions rather than switching terms, but
I'm open to it if we want to convert all uses of codepoint to "code
unit".
Encoding - See Character Set Encoding
Codepoint - When a character set encoding uses differing variable
width representations for characters, the units making up these variable
width representations are called codepoints. For example the UTF-8 encoding
uses between 1 and 4 codepoints to represent characters, and for UTF-8,
the codepoints are single bytes. The UTF-16 encoding is either fixed or
variable width. When dfdl:utf16Width='variable' this encoding uses either
one or two codepoints per character and each codepoint is a 16-bit value.
When a character set is fixed width, then there is no distinction between
a codepoint and a character code.
Code page - An alternate identifier for a Character Set Encoding.
Character Code - The numeric value assigned to a character in a character
set that is independent of any specific encoding of that character set.
For any fixed-size encoding (all characters have the same size representation)
Character Set - An abstract set of characters independent of any specific
encoding scheme: Examples are the Unicode character set, or the USASCII
character set.
Character Set Encoding - A specific representation of a character set as
bytes or bits of data. A character set encoding is usually identified by
a standard character set name or a recognized alias name, or by a code
page identifier. These identifiers are standardized by the IANA. Examples
are UTF-8, USASCII, GB2312, ebcdic-cp-it, ISO-8859-5, UTF-16BE, Shift_JIS.
The DFDL standard allows for implementation-specific character set encodings
to be supported, and standardizes one name that is DFDL-specific which
is USASCII-7bit-packed.
Character Width - The number of codepoints or bytes used to represent a
character in a specific character set encoding is called the character
width. Encodings are either fixed width (all characters encoded using the
same width), or variable-width (different characters are encoded using
different widths). For example the UTF-32 character set encoding has 4-byte
character width, whereas USASCII has a 1-byte character width.
Fixed-Width Character Encoding - A character set encoding where all characters
are encoded using a single codepoint for their representation. Note that
a codepoint may take up one or more bytes.
Surrogate Pair - A Unicode character whose character code value is greater
than 0xFFFF can be encoded into variable-width UTF-16BE or UTF-16LE which
are variable-width encodings when the DFDL property utf16Width='variable'.
In this case the representation uses two adjacent codepoints each
of which is called a surrogate, and the pair of which is called a surrogate
pair.
Variable-Width Character Encoding - A character set encoding where characters
are encoded using one or more codepoints for their representation depending
on which specific character is being encoded. An example is UTF-8 which
uses from 1 to 4 bytes to encode a character.
...mike
--
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com
--
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com
--
dfdl-wg mailing list
dfdl-wg@ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU