- dfdl-wg - lists.ogf.org

Fw: DFDL ICU Challenges for Implementation - textNumberExponentRep
by Steve Hanson 28 Aug '13

28 Aug '13

Given that ICU do plan to offer this support, the DFDL spec will remain as it is, with textNumberExponentRep being sensitive to ignoreCase, and DFDL implementations that ICU should include a release note that says all matches today will be case insensitive but are liable to change. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2013 16:15 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 21/08/2013 12:21 Subject: Fw: [DFDL-WG] DFDL ICU Challenges for Implementation Regarding Mike's second point: Here's a link to an ICU ticket on this subject: http://bugs.icu-project.org/trac/ticket/9659 - currently targetted for ICU 52.1 Here's some more details from IBM's calls with ICU on this subject: - Exponent character / ignoreCase : Exponent char is not case sensitive. Is this intentional? * Priority : Medium ICU see two options for this: Option 1: Provide an API call to set a flag on the DecimalFormat object. Option 2: Make it a global policy settable via a config switch. This would allow other 'site policies' to be made settable using the same mechanism. There would be one set of policy flags, including this flag, per address space. There are differences in date/time processing between C and Java that could be dealt with using this mechanism. DFDL needs some of these flags to be configurable at runtime. 2012/10/19 Hit an issue where case handling was inconsistent. Fix needs care to avoid changing default behaviour and thus breaking existing users of the API. Currency and prefix/suffix may need separate switch so global switch for the DecimalFormat not appropriate. Could provide a patch to ICU for setting exponent char for now. 2013/1/17 - ICU external ticket #9659 ICU had an issue with the new API being specific to just case sensitivity of exponent (and not other regions). DFDL clarified the requirement is for an API to change global case sensitivity (not just exponent). This is targetted at ICU51 Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 14/08/2013 16:55 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 14/08/2013 14:14 Subject: [DFDL-WG] DFDL ICU Challenges for Implementation Sent by: dfdl-wg-bounces(a)ogf.org There are a couple of features in DFDL that ICU doesn't support, yet where all or nearly all the related functionality is supported by ICU. Perhaps these aspects of the spec can be revisited? 1) List of Decimal Separators The textStandardDecimalSeparator property is a list of characters. However, ICU only supports a single character. I see lots of potential for error here, confusing diagnostics, etc. It is not consistent with textStandardGrouping separator, which allows only a single character. Is there a use case where we know we need more than one decimal separator? The only thing I can think of is a blend of say classic European-style decimal numbers like "1 234 567,89" and USA style " 1,234,567.89", but ICU won't deal with different grouping separators either. In any case if there are multiple decimal and grouping separators we really don't have these properties right in DFDL. We should require them to be specified not as two separate lists, but as a list of pairs, because grouping separators match up with specific decimal separator values in a format. 2) Case Insensitivity Some properties that we use to configure ICU are affected by ignoreCase="yes", but ICU does not support case insensitivity. The properties are: textStandardExponentRepCharacter textStandardInfinityRep textStandardNaNRep I can certainly imagine a need for case insensitivity here, and even for multiple values for these (though we allow only one for Infinity and NaN). For the infinity and nan reps that isn't so problematic as one can easily do a pre-check before calling ICU, but for the exponent rep, that is needed down in the detailed number format parsing. I can see no certain algorithm other than creating separate number format parsers for each exponent rep character in provided case, and opposite case, and then using them one by one until a successful parse. Is this ok or do we consider this a mistake? 3) We are not very consistent in these properties. We allow multiple textStandardZeroRep values, but only a single textStandardInfinityRep, and only a single textStandardNaNRep. We allow multiple textStandardExponentRepCharacter, and multiple textStandardDecimalSeparator, but only a single textStandardGroupingSeparator. This kind of inconsistency is always problematic for users. Comments? -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: clarification needed: 16.3.1 and totalDigits, fractionDigits and textNumberPattern compatibility checks
by Steve Hanson 28 Aug '13

28 Aug '13

Agreed that the statement should be dropped from the spec. DFDL implementations are free to issue a warning if they detect such a mismatch (so the IBM DFDL check will change from an error to a warning). Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2013 16:31 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 22/08/2013 16:02 Subject: Re: [DFDL-WG] clarification needed: 16.3.1 and totalDigits, fractionDigits and textNumberPattern compatibility checks To comply with the spec as stated, the actual check implemented by IBM DFDL is that the number of fraction/total digits in the DFDL textNumberPattern must not exceed the number of fraction/total digits specified by the facet. That's a more liberal interpretation of 'match' but makes sense. However the spec does not say that the textNumberPattern should not be allowed to describe a number greater than any maxInclusive/maxExclusive facet. And I don't think that there are any other checks to see that DFDL properties clash with schema facets. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Cc: Jessie Chab <jchab(a)tresys.com> Date: 22/08/2013 15:35 Subject: [DFDL-WG] clarification needed: 16.3.1 and totalDigits, fractionDigits and textNumberPattern compatibility checks Sent by: dfdl-wg-bounces(a)ogf.org The spec says that these need to be compatible. "If the pattern uses digits/fractions then these must match any XML schema facets." We've fallen into this trap before. Facet constraints are about the logical value, not the physical rep, and so must make equal sense for binary representations as text representations. I was considering what kinds of static cross checks would make sense here. Example: suppose fractionDigits='2'. Suppose textNumberPattern="##" the type must be xs:decimal since that's the only type that can have fractionDigits constrained in DFDL. Is there a conflict of the facet with the textNumberPattern? I think not. Even though the pattern doesn't allow for any fractional part to be expressed, that means that all values will have less than or equal to 2 fractional digits (because no fractional digits always satisfies that.) Now suppose textNumberPattern="#.########" Now the textNumberPattern will parse and accept things like 1.234 which have more than 2 fraction digits, but suppose the data simply doesn't contain any such. So all number in the data will obey the less than or equal to 2 fractional digits. Lastly, suppose textNumberPattern="9.9999" So, interpreted naively, this requires the data to contain 5 digits, so this seems inconsistent with fractionDigits of 2. However, we know this depends on the strict/lax behavior of ICU libraries, and I believe this will happily parse the data "0.0" without error. Hence the data could still all have 2 or fewer fractional digits. So, I think this idea that the totalDigits and fractionDigits facets can be cross-checked with the textNumberPattern is spurious and we should drop it. I think it dates from a time when we were still confused about the separation of the DFDL Infoset, and facet constraints on its value spaces, which are abstract, versus the XML Infoset, where everything truly is a string as well as whatever its XML Schema type says. Comments? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Information Update - Location has changed: OGF DFDL WG Call
by Steve Hanson 28 Aug '13

28 Aug '13

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Added Hursley room

1 0

dfdl:lengthKind='prefixed' , different encodings for prefix and content where dfdl:prefixIncludesPrefixLength is ‘yes.
by Steve Hanson 27 Aug '13

27 Aug '13

Assuming that the prefix contains the length in characters then I think this works ok when the encoding is different. The parser will first parse the prefix according to prefixLengthType to get the prefix value, which is always of known length. If prefixIncludesPrefixLength is 'yes' then it subtracts this known length from the prefix value, giving the length of the data, which might be in a different encoding. I think we should continue to allow this. In the past we have talked about a DFDL 2.0 feature that allowed the initiator and terminator to be specified using a simple type, precisely to cover the (rare) cases where the characteristics of these delimiters are different to the data itself. Doing it this way prevents a property explosion on the element itself. I view prefixLengthType as the first example of this principle. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/08/2013 16:20 ----- From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Date: 23/08/2013 14:58 Subject: dfdl:lengthKind='prefixed' , different encodings for prefix and content where prefixIncludesPrefixLength is ‘yes. Hi All, Considering a case similar to that excluded by errata 2.76. An element with lengthKind 'prefixed' and prefixIncludesPrefixLength 'true' but where the prefix type and the element both have lengthUnits 'characters' but have different encodings (or specifically encodings with different lengths of characters). I believe the issue that 2.76 is trying to avoid is the issue of determining the length value in say characters when the prefix contains no characters. I am wondering if there is also a slightly subtler issue when we are calculating a length in characters but where a part of the length is in a different encoding from the other. For example the prefix contains 2 UTF16 (2 byte) characters and the content contains 2 UTF32 (4 byte) characters.. Do we just quote a length in characters regardless of encoding. eg. 4 characters. Or is this confusing .... 2.76 . Section 12.3.4 . When property prefixIncludesPrefixLength is ‘yes’there are some restrictions that need to be added to enable reliable lengths to be calculated: o If the prefix type is lengthKind 'implicit' or 'explicit' then the lengthUnits properties of both the prefix type and the element must be the same. Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development IBM DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Agenda 2013-08-28
by Steve Hanson 27 Aug '13

27 Aug '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13107?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL WG Call (28 Aug 15:00 GDT in DE3S09)
by Steve Hanson 27 Aug '13

27 Aug '13

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Mike unavailable on Tuesday

1 0

errors and clarifications needed in section 14.2.1 "enclosing group" should be "enclosing sequence" in a few places, "followed in the data" ambiguous.
by Mike Beckerle 23 Aug '13

23 Aug '13

Definitions of Potentially Trailing things mention "enclosing group", but choices don't qualify. A group in XSD is a sequence or a choice, so if we mean enclosing sequence, we can't slide by with enclosing group. We need to to a complete pass through the spec, searching for uses of the word group. All such should mean "sequence or choice", or should be changed to one of those otherwise. My edits for section 14.2.1 are here in colored background: Potentially trailing element – An array or optional element describes an occurrence that is said to be potentially trailing if the element is capable of having a zero length representation and is followed in its enclosing sequence definition by only additional potentially trailing elements or potentially trailing groups. Intuitively, the array or optional element occurrence could be last. Potentially trailing group – A group is said to be potentially trailing if the group has no framing and contains only potentially trailing element declarations/references, or recursively similar sequence or choice groups, and is followed in its enclosing sequencedefinition by only additional potentially trailing elements or potentially trailing groups. Trailing or Actually Trailing – An element occurrence or group occurrence in the data is said to be actually trailing if it is potentially trailing and has zero-length representation and is not followed in the data by any other non-zero length element occurrence or group occurrence limited by the end of the enclosing sequence group. The phrase "followed in its enclosing group definition..." is meaningless if the enclosing group is a choice, as choice alternatives don't have an ordering relationship that corresponds to physical position in the data. All of these definitions are flawed in the following sense. When we say last, we mean "last represented" in almost all cases, because another element following, which has an inputValueCalc doesn't count, nor a sequence group with no framing, containing only elements with inputValueCalc or similar sequences, nor an element with complex type having only that stuff, etc. So it seems we need the more general concept of "is represented" in the spec, so that when we say "last" we can be clear. Trailing or Actually Trailing says "... is not followed in the data by any other ...". This is unclear because "is not followed in the data" doesn't express any limit at all. What we mean here is limited by only up to the end of the sequence group whose children/separators are under discussion. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

dfdl:lengthKind='prefixed' , different encodings for prefix and content where dfdl:prefixIncludesPrefixLength is ‘yes.
by Steve Hanson 23 Aug '13

23 Aug '13

For discussion at next WG call. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/08/2013 16:20 ----- From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Date: 23/08/2013 14:58 Subject: dfdl:lengthKind='prefixed' , different encodings for prefix and content where prefixIncludesPrefixLength is ‘yes. Hi All, Considering a case similar to that excluded by errata 2.76. An element with lengthKind 'prefixed' and prefixIncludesPrefixLength 'true' but where the prefix type and the element both have lengthUnits 'characters' but have different encodings (or specifically encodings with different lengths of characters). I believe the issue that 2.76 is trying to avoid is the issue of determining the length value in say characters when the prefix contains no characters. I am wondering if there is also a slightly subtler issue when we are calculating a length in characters but where a part of the length is in a different encoding from the other. For example the prefix contains 2 UTF16 (2 byte) characters and the content contains 2 UTF32 (4 byte) characters.. Do we just quote a length in characters regardless of encoding. eg. 4 characters. Or is this confusing .... 2.76 . Section 12.3.4 . When property prefixIncludesPrefixLength is ‘yes’there are some restrictions that need to be added to enable reliable lengths to be calculated: o If the prefix type is lengthKind 'implicit' or 'explicit' then the lengthUnits properties of both the prefix type and the element must be the same. Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development IBM DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

clarification needed: 16.3.1 and totalDigits, fractionDigits and textNumberPattern compatibility checks
by Mike Beckerle 22 Aug '13

22 Aug '13

The spec says that these need to be compatible. "If the pattern uses digits/fractions then these must match any XML schema facets." We've fallen into this trap before. Facet constraints are about the logical value, not the physical rep, and so must make equal sense for binary representations as text representations. I was considering what kinds of static cross checks would make sense here. Example: suppose fractionDigits='2'. Suppose textNumberPattern="##" the type must be xs:decimal since that's the only type that can have fractionDigits constrained in DFDL. Is there a conflict of the facet with the textNumberPattern? I think not. Even though the pattern doesn't allow for any fractional part to be expressed, that means that all values will have less than or equal to 2 fractional digits (because no fractional digits always satisfies that.) Now suppose textNumberPattern="#.########" Now the textNumberPattern will parse and accept things like 1.234 which have more than 2 fraction digits, but suppose the data simply doesn't contain any such. So all number in the data will obey the less than or equal to 2 fractional digits. Lastly, suppose textNumberPattern="9.9999" So, interpreted naively, this requires the data to contain 5 digits, so this seems inconsistent with fractionDigits of 2. However, we know this depends on the strict/lax behavior of ICU libraries, and I believe this will happily parse the data "0.0" without error. Hence the data could still all have 2 or fewer fractional digits. So, I think this idea that the totalDigits and fractionDigits facets can be cross-checked with the textNumberPattern is spurious and we should drop it. I think it dates from a time when we were still confused about the separation of the DFDL Infoset, and facet constraints on its value spaces, which are abstract, versus the XML Infoset, where everything truly is a string as well as whatever its XML Schema type says. Comments? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

OGF DFDL WG Call Minutes 2013-08-20
by Steve Hanson 21 Aug '13

21 Aug '13

Please find minutes from the above two calls at http://redmine.ogf.org/dmsf_files/13105?download=. Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0