We discussed the correct XML schema type for DFDL String Literal on the last WG call. I read up on xs:NMTOKEN - not appropriate as it is basically a name so does not allow the full range of characters we need. Then I looked at restricting xs:token, but I could not work out from the XML Schema 1.0 spec how whitespace facets were handled when other facets were present. So I asked Sandy, and got the very useful clarification below. Please review for next call.

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
----- Forwarded by Steve Hanson/UK/IBM on 17/07/2013 16:01 -----

From: Sandy Gao/Toronto/IBM@IBMCA
To: Steve Hanson/UK/IBM@IBMGB,
Date: 17/07/2013 13:33
Subject: Re: DFDL String Literal type

Hi Steve,

Yes, that should work. All other facet checking, including pattern, happens *after* whitespace handling.

This was made clearer in Schema 1.1, where "whitespace" is called a pre-lexical facet, and "pattern" etc. are called lexical facets.

Thanks,
Sandy Gao
Source Code Monitoring (SCMon)
IBM Canada
sandygao@ca.ibm.com

From: Steve Hanson/UK/IBM@IBMGB
To: Sandy Gao/Toronto/IBM@IBMCA,
Date: 2013-07-17 06:07 AM
Subject: DFDL String Literal type

Hi Sandy

Please can I ask your advice on use of the whitespace facet in conjunction with the pattern facet? This is in order to model the correct data type for a DFDL String Literals. This is defined as:

DFDL String Literal
DFDL String Literals represent a sequence of literal bytes or characters which appear in the data stream. This presents the following challenges

- the literal characters in the data stream might not be in the same encoding as the DFDL schema

- it may be necessary to specify a literal character which is not valid in an XML document

- it may be necessary to specify one or more raw byte values

A DFDL string literal can describe any of the following types of literal data in any combination:

- a single literal character in any encoding

- a string of literal characters in any encoding

- a bi-directional character string

- one or more characters from a set of related characters ( e.g. end-of-line characters)

- a literal byte value

A DFDL string literal is therefore able to describe any arbitrary sequence of bytes and characters.

Empty Strings: Empty string is not allowed as a DFDL string literal value unless explicitly stated otherwise in the description of a property. In this case the use of empty string provides some property specific behavior different from simply using the empty string as a value. When the empty string is to be used as a value, the entity %ES; must be used in the corresponding DFDL string literal.

Whitespace: When whitespace must be used as part of a property value, the DFDL string literal must use entities (such as %WSP;) to represent the whitespace. (This allows a property to represent lists of DFDL string literals by using literal spaces to separate list elements.)

The nearest match to an XSDL built-in type is xs:token, but we require the additional constraint that no whitespace can appear. My thought is to define a restriction of xs:token that applies a pattern facet to disallow use of #x20, given that the whitespace 'collapse' implied by xs:token would have replaced #x9, #xA, #xD with #x20, collapsed contiguous #x20, and trimmed leading/trailing #x20. Does that sound right?

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU