Yes it is regrettable that we need that
2nd case escape sequence, but I did some real digging into this, and the
w3c groups would I believe not be receptive to reopening this issue. It's
not a syntactic thing, it's about C language API access to the XML Infoset.
No string from XML data can have a nul (zero) char code in its content
so that XML string content is compatible with C's nul terminated string
convention. Basically, this restriction lets fairly ordinary C code
manipulate XML infoset content. If the nul char code restriction were not
there then you'd need a whole non-standard string library in C to deal
with XML infoset contents. I don't think this is feasible. The other character
codes they disallow are about the non-characters of Unicode. I.e., the
XML infoset can't contain surrogate-pair fragments. Rather the infoset
contents must contain the full characters those represent, and finally
the byte-order-mark codepoints are not considered valid contents for the
infoset either.
Note that for DFDL, since data we're
describing can contain NUL in strings, a C-callable API to a DFDL described
'infoset' must allow for the strings to contain NUL bytes. As a prescriptive
format for data, XML could get away with solving the problem by avoidance.
As a descriptive approach DFDL does not have this luxury, so dealing with
DFDL-described string data from C means dealing with string data with C
where the string terminating NUL conventions aren't necessarily respected.
Similarly we must be able to deal with things that contain surrogate pairs,
even unpaired, and byte order marks, etc.
I considered whether we could get away
with just the 'bytes' escape sequence (the \%xHH; format in my proposed
syntax), but I concluded we also need the '\#xHH;' syntax for dealing with
UTF-16 and UTF-32 character sets, which I expect will become more common
over time, particularly UTF-16.
Also, one of the strong arguments for
'\#xHH;' syntax is the 'foolish consistency' argument. You can always use
'\#xHH;', you don't have to think about whether XML allows the codepoint
or not . So it simplifies life for DFDL schema authors. DFDL authors can
just uniformly use '\#xHHHHHH;' (up to 6 hex digits, maximum value 10FFFF)
when they want to specify a unicode character code and forget about XML
character references entirely if they want.
BTW: I consider the "\#" as
similar to '&#' in XML syntax. I.e., you can specify hex via the 'x'
or decimal without the 'x', and up to 6 hex digits are allowed, 7 decimal
digits (max value 1114111 decimal, 0x10FFFF hex). This is different from
the '\%HH; syntax which requires exactly one or two hex digits. This behavior
is needed to deal with endianness issues. I.e., in a single byte charset
if you allowed '\%xFEFF;' then which byte appears at the lower address
in the data, the #xFE or the #xFF ? The above restriction avoids all of
that at the cost of requiring you to explicitly specify '\%xFE;\%xFF;'.
As for syntax '\#0xHH;' or '\%xHH;'.
I used a percent sign because it's used for hex byte specifications in
a URL standard format which is a very very minor precedent. I am
a bit concerned that people will expect '\#xHH;' and '\#0xHH;' to be synonyms.
I.e., that use of the leading zero seems rather subtle. I'd kind of prefer
an entirely different escape sequence indicator than a subtle distinction
like this. What's going on is quite significantly different. One is talking
about unicode codepoints, the other about representation bytes. I think
they shoudl be loudly separated.
Mike Beckerle
Architect, Scalable Computing
IBM Software Group
Information Integration Solutions
Westborough, MA
Steve Hanson <smh@uk.ibm.com> Sent by: owner-dfdl-wg@ggf.org
09/15/2005 04:56 AM
To
dfdl-wg@gridforum.org
cc
Subject
Fw: [dfdl-wg] minutes of
call 2005-09-14
Hi Mike
Re the three escape mechanisms. It is unfortunate that we need a separate
mechanism for the second case (as discussed in the draft spec section 26).
Re the third mechanism for escaping hex literals. An alternative syntax
could be \#0x2C; using 0x to indicate hex. �x2C; would be even
better -
but does it cause XML problems?
Regards, Steve
----- Forwarded by Steve Hanson/UK/IBM on 15/09/2005 09:33 -----
Mike Beckerle
<beckerle@us.ibm.
com>
To
Sent by:
dfdl-wg@gridforum.org
owner-dfdl-wg@ggf
cc
.org
Subject
[dfdl-wg]
minutes of call
14/09/2005 22:18
2005-09-14
Who: Jim Myers, Mike Beckerle, Tara Talbott, Steve Hanson, Geoff Judd,
Bob
McGrath
Agenda:
GGF plans - we will have F2F at GGF15. The schedule of our WG sessions
at
the GGF15 are not yet set, but we'll have F2F meetings during the
non-scheduled times. We're counting on the fact that most of our DFDL WG
members are focused on DFDL and won't mind the conflict with the rest of
GGF15 much. People should make travel plans for all 3 days M, T, W.
GridForge Forums - these seem to be working now. Will try them when one
of
our subcommittees reports back to the broader WG. E.g., scoping or arrays.
Issues list - we made it through items 5 to 15
Issue 5 - resolution. No built-in set of defaults. If you need a property
and your configuration doesn't have one specified then it is an error.
There is a small set of named configurations provided with DFDL each of
which is a self-consistent set of properties. This is probably the set
that
we find useful in the primer and other WG docs.
Issue 6 - resolution. change attribute name from 'base' to 'extends'
Issue 7 - belongs on same tracker as 8, 9, 10
Issue 11 - resolution: add an attribute 'byteOrderMarkPolicy' values are:
required, notAllowed, optionalButGenerateOnOutput, optionalButOmitOnOutput
(the generate on output control aspect was not discussed on the call. I
thought of that just now while typing.)
Issue 12 - fix diagram - omission of these types was unintentional. Better
diagram next draft
Issue 13 and 14 - new tracker item - Steve Hanson, Geoff Judd and someone
from MikeB's team at IBM to address
Issue 15 - resolution: each delimiter will grow an extra rep-property for
its regexp variant. E.g., separator will have corresponding separatorRegexp
property. Only one of the two may be specified.
We discussed that separator (or any delimiter) can be a text string, and
we
had previously decided that the separatorEncoding attribute would go away
to be replaced by a syntax for expressing hex bytes (not hex character
codes, real non-charset transformed hex bytes) as part of delimiter strng
literals. This same way of putting hex bytes would also be supported as
part of the regexp language so that the regexp language remains able to
include any non-regular expression for a delimiter into a regular
expression.
(Editor's Note: this is a bit tricky. We now have in string literals 3
different escape mechanisms that are different.
1) XML character code specifiers e.g., 'abc,def' which is the
normal
XML way of specifying the unicode #x2C character code. (This is an XML
standard escape convention.) Used inside a delimiter string this means
take
the unicode #x2c code point, figure out what character it corresponds to
in
the charset of the data element and use that character. Note that this
is
the same thing that 'a' means. Take the 'a' unicode code point, figure
out
what character corresponds to 'a' in the charset of the data element, and
look for that.
2) nonXML character code specifiers e.g., 'abc\#x00;def' which is the way
we allow you to put XML-disallowed unicode character codes like #x00 into
a
string literal. (This is a DFDL convention). The interpretation here is
exactly like the above. I.e., '\#x2c;' is exactly equivalent to ','
and the charset mapping applies as above. This rule is only needed because
of the XML restriction disallowing certain unicode character codes from
the
XML infoset.
3) non-character byte specifiers. E.g., 'abc\%01;\%02;def' which is a
proposed escape syntax that means put the bytes #x01 and #x02 into the
string literal bypassing any considerations of charset, i.e., without
considering them to be character codes in any character set. That is, these
byte values have nothing to do with unicode codepoint values. (This is
a
DFDL convention.) The other characters of the string would be treated as
per (1) and/or (2) above.