Sounds symmetric and expedient, but the problem is that some character encodings have no reserved replacement character, and we expect that DFDL users will need a variety of different choices for how to deal with characters that cannot be encoded.

SMH: From what Andreas says, the vast majority of encodings supported by ICU have a replacement character.

Suggested Resolution: Summary

• DFDL property dfdl:inputEncodingErrorPolicy with values 'skip', 'error', 'replace'
• DFDL property dfdl:outputEncodingErrorPolicy with values 'skip', 'error', 'replace'
• DFDL annotation element. Example: <dfdl:encodingReplacementCharacter encoding="ASCII" character="%#x7c;"/>    

If we do keep it, then

1) Not clear from the syntax that this is intended for unparse only. The word 'output' should be in there, eg, outputEncodingReplacementCharacter.

2) As proposed there is no way to place encoding modifiers in one xsd and have them picked up by another. We need a scoping mechanism.

My vote for 1.0 is not to add the new annotation, just the new properties.

Can we get away with just one property that applies to both input and output? Most DFDL properties apply to both.

For Parsing/Decoding Errors

There are two errors that can occur when decoding characters into Unicode/ISO 10646.
1.        the data is broken - invalid byte sequences that don't match the definition of the encoding are encountered.
2.        not enough bytes are found to make up the entire encoding of a character. That is, a fragment of a valid encoding is found.
The behavior in these cases is controlled by dfdl:inputEncodingErrorPolicy.

If 'replace', then the Unicode replacement character '�' (U+FFFD) is substituted for the offending bytes, one replacement character for each invalid byte, one replacement character for any fragment of an encoding.

(TBD: Assumptions to validate: I am assuming here that if there are 6 invalid bytes, none of which can validly be unit 1 of the encoding of any character, that ICU will call the error hook either (a) 6 times, or (b) once but notifying about all 6 bad units - but providing a way for the hook-writer to say they want to substitute 6 characters for the 6 units.

I am also assuming in the end-of-data fragment case that the ICU hook gets called once for the fragment, not once per byte of the fragment.)

(TBD: We did not discuss on the call on Dec 6th, the issue of errors in unicode encodings. While there are no encodings where a properly encoded character is unmapped to unicode, the unicode UTF encodings themselves can contains things that are errors. Here's a short list of some things that can happen:

• utf-16 and unpaired surrogate code-point
• utf-16 and out-of-order surrogate code-point pair
• utf-8 parsing and 3-byte encoding of a surrogate code-point is found
• utf-8 unparsing and code-point of an isolated surrogate is to be encoded.
• utf-8 decoding, and if you assemble the bits the usual way, you get a code point out of range (higher than 0x10FFFF)
• utf-8 encoding, and code-point to encode is higher than 0x10FFFF.
• utf-16 encoding utf16Width='fixed' and a surrogate code point is encountered
• utf-16 byte-order-marks found not at the beginning of the data
  We have an option here to be 'tolerant' of unicode-encoding foibles. We can preserve isolated surrogates in a natural way if we wish. I believe many Unicode and UTF implementations tolerate these situations. For example the standard Java utf-8 decoder/encoder InputStreamReader and OutputStreamWriter, is tolerant of incorrectly paired and isolated surrogate code points in the Java string data.
  
  I do not know what ICU does in these cases, i.e., if it provides us enough flexibility to do whatever we want, or if it doesn't even detect some of these things as errors.)

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU