August 2013 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

Rescheduled: OGF DFDL WG Call (3 Sep 17:00 GDT in DE3S09)
by Steve Hanson 30 Aug '13

30 Aug '13

Rescheduled: OGF DFDL WG Call 03/09/2013 - (Repeats) Location: DE3S09 Steve Hanson has rescheduled this meeting. You have not added this entry to your calendar. Repeats This entry repeats Required: jcranford(a)mitre.org, mbeckerle(a)tresys.com, mbeckerle.dfdl(a)gmail.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org, Stephanie Fetzer/Charlotte/IBM@IBMUS OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

OGF DFDL WG Call Minutes 2013-08-28
by Steve Hanson 28 Aug '13

28 Aug '13

Please find minutes from the above two calls at http://redmine.ogf.org/dmsf_files/13107?download=. Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Action 217 - scannable - Re: clarification needed for scan and scannable
by Mike Beckerle 28 Aug '13

28 Aug '13

I propose this new erratum, and corresponding edits to erratum 2.9. In the draft r14.3 (to be circulated soon), I have OPEN comment bubbles to review the impact of this change, but I have edited this stuff in, as you really have to see it in action to see if it "works for you". *Errata 2.155* Sections 3, 7.3.1, 7.3.2, 12.3.5. Scan, scannable, scannable-as-text These terms all added to/changed in the glossary. Definitions removed from the prose. Scannable now means able to scan, which is natural. More specific term scannable-as-text used when we want the recursive requirement of uniform encoding. Errata 2.9 updated to use term scannable-as-text. On Wed, Aug 14, 2013 at 5:45 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Action 217 raised to decide new terminology for the regex scanning > encoding requirement. > > Regards > > Steve Hanson > Architect, IBM Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Tim Kimber/UK/IBM@IBMGB, > Cc: Steve Hanson/UK/IBM@IBMGB > Date: 30/07/2013 01:01 > Subject: Re: clarification needed for scan and scannable > ------------------------------ > > > > I reviewed the spec. I have to reverse my prior statement. What you > describe is definitely currently allowed by the spec. The 'scannable' > restriction is reserved for lengthKind='pattern'. > > In my view this is a terrible idea, but it is where it is. I.e., that the > lengthKind 'delimited' doesn't require 'scannable' characteristic > throughout everything that is contained within it. It will not allow us to > really assist users in identifying what are very likely to be mistakes in > their schemas, because what we call 'delimited' is much too permissive. > > I am not going to argue to change this at the current time. This is just > due to the need to converge on the standard. My preference would be that > lengthKind 'delimited' requires scannability (as uniform text), and that > some new keyword be used to mean the current algorithm as specified. > > I do suggest that we rename the characteristic scannability to > scannable-as-text, in order to make it clearer what the requirement is, and > clarify that this is a requirement on the schema component. > > I suspect that lengthKind='delimited' will perhaps someday (DFDL v2.0?) be > deprecated and/or augmented by other keywords that are more specific such > as "delimitedText" which means scanning for delimiters over only text in > one encoding (the 'scannable' restriction), and other keywords like > "delimitedBinary" or "delimitedMixed" meaning formats which admit the more > powerful and complex things. > > My argument is simple: if people have something like delimiters in > distinct encodings appearing in their schema, it is most-likely due to a > human error (they missed one place when they were editing the encoding > property), rather than something as obscure as this > delimiters-in-a-different-encoding being a real characteristic of the data. > An SDE here will help them find this error. > > Furthermore, if you truly want a string to be terminated by either an > ebcdic comma (6B) or an ascii linefeed (0A), then you have a few > alternatives before you have to go to the now optional feature of > 'rawbyte', and which are consistent with scannability. > > First, use encoding ascii, and specify terminators of %LF;, and 'k' (or > %#x6B; since 'k' is 6B ascii), in which case the string is assumed to > contain only ascii code units. In that case, if ebcdic 'a' (x81 - illegal > in ascii) is encountered in the data, you will either get an error or a > unicode replacement character depending on encodingErrorPolicy. If you know > your data will only ascii-legal code units, then this is a good solution. > > I would note that your example (previously in the thread) did not specify > an encoding for the innermost string elements. Did you intend for those to > be ebcdic or ascii? > > Alternatively you can specify encoding ebcdic-cp-us, specify terminator of > comma (codepoint x6B) and %#x0A; which is the RPT control code in ebcdic > but not an illegal character. In that case the string can contain any legal > ebcdic code point. However, if code unit x70 is encountered (corresponds to > an ascii 'p', but unmapped in ebcdic-cp-us), you will either get an error, > or a unicode replacement character depending on encodingErrorPolicy. If you > know your data will contain only legal ebcdic code units, then this is a > good solution. > > Finally, you can specify encoding iso-8859-1, and terminators %#x6B; (or > 'k' which is 6B in iso-8859-1) and %#x0A; (linefeed in iso-8859-1). Then > any code units at all will be legal as all bytes have corresponding > character codepoints in iso-8859-1. If you have no idea what the data is, > but just want some sort of string out of it, and know only that the > terminators are these bytes, then this is a good solution. If your data > contains, for example, packed-decimal numbers, then this is the only way to > safely scan past it as a string, because both ebcdic and ascii have > unmapped code points from the code units that could appear in > packed-decimal number bytes. > > All the above are consistent with implementations that attempt to convert > data from code-units (bytes) to Unicode codepoints using a character set > decoder, and then apply any scanning/searching only in the Unicode realm, > and work in one pass over the data. > > I currently think that to implement both rawbytes and encodingErrorPolicy > it will require two passes over the data. I expect this overhead to be > negligible, so I'm not worried about it really. > > > ...mike > > On Mon, Jul 29, 2013 at 7:18 AM, Tim Kimber <*KIMBERT(a)uk.ibm.com*<KIMBERT(a)uk.ibm.com>> > wrote: > Hi Mike, > > Good - that was almost exactly the reply that I was expecting. I now > understand exactly where you are coming from, and how we arrived at this > position. > > First, a few statements that I think are true. I want to establish some > basic ground rules before we decide how to go forward: > a) it is desirable for DFDL ( more accurately, a valid subset of DFDL ) to > be implementable using well known parsing techniques. I think that pretty > much means regular expressions and BNF-style grammars. That implies that it > might be possible to implement a DFDL parser using one of the well-known > parser-generator technologies like Yacc/Bison/JavaCC/Antlr. I'm not > claiming that it *is* possible, but I think it would be a good thing if it > was. > b) It is technically feasible to implement a DFDL parser that can handle > the mixed-encoding example using regex technology. However, it would not be > a very efficient implementation because the scanner would have to scan the > data once per encoding, and it would have to do that for every character > after the end of field1. > c) It is possible to produce an efficient scanner that handles mixed > encodings. Such a scanner cannot use regex technology for scanning - in > fact, I think the only efficient implementation is to convert all > terminating markup into byte sequences and then perform all scanning in the > byte domain. This is what the IBM implementation does. > > The scenario in my example is not entirely far-fetched - it is conceivable > that the encoding might change mid-way through a document, and I think > Steve came up with a real-world format that required this ( I have a hazy > memory of discussing this a couple of years ago ). The requirement for the > raw byte entity is not for this case - it is for the case where the > delimiters are genuinely not characters ( e.g. a UTF-16 data stream > terminated by a single null byte ). However, it is not easy to come up with > realistic examples where the raw byte entity could not be translated into a > character before the processor uses it. I think that's where some of the > confusion has arisen. > > We have already agreed to make the raw byte entity an optional feature. We > should consider disallowing mixed encodings when lengthKind is delimited. > If we cannot do that then I agree with Mike that the descriptions and > definitions in the spec need to be made a bit clearer. > > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: *kimbert(a)uk.ibm.com* <kimbert(a)uk.ibm.com> > Tel. 01962-816742 > Internal tel. 37246742 > > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com> > > > To: Tim Kimber/UK/IBM@IBMGB, > Cc: Steve Hanson/UK/IBM@IBMGB > Date: 26/07/2013 22:49 > Subject: Re: clarification needed for scan and scannable > ------------------------------ > > > > Clearly we have to rework the description and definitions because my > understanding of the current spec would say your schema is not valid > because it is not scannable. Everything in the scope of the enclosing > element which is delimited must be scannable and that means uniform > encoding. > It is exactly to rule out this mixed encoding ambiguity that we have the > restriction. > > I have no idea how to implement delimiters without this restriction. > > I think the case you have here is what raw bytes are for. Though I am no > longer clear on how to implement rawbytes either. > > On Jul 25, 2013 6:40 PM, "Tim Kimber" <*KIMBERT(a)uk.ibm.com*<KIMBERT(a)uk.ibm.com>> > wrote: > Suppose I have this DFDL XSD: > > <*xs:element* name="documentRoot" dfdl:encoding="US-ASCII" dfdl:lengthKind > ="delimited" dfdl:terminator="]]]"> > <*xs:complexType*> > <*xs:sequence*> > <*xs:element* name="delimitedRecord" maxOccurs="unbounded" > dfdl:encoding="US-ASCII" dfdl:lengthKind="delimited"dfdl:terminator > ="%LF;"> > <*xs:complexType*> > <*xs:sequence* dfdl:separator="," dfdl:separatorSuppressionPolicy > ="suppressedAtEndLax" dfdl:encoding="EBCDIC-US"> > <*xs:element* name="field1" type="xs:string" dfdl:lengthKind= > "delimited"> > <*xs:element* name="field2" type="xs:string" dfdl:lengthKind= > "delimited" minOccurs="0"> > <*xs:element* name="field3" type="xs:string" dfdl:lengthKind= > "delimited" minOccurs="0"> > <*/xs:sequence*> > <*/xs:complexType*> > <*/xs:element*> > <*/xs:sequence*> > <*/xs:complexType*> > <*/xs:element*> > > > ...which will parse some data like this: > field1Value,field2Value,field3Value > field1Value, > field1Value,field2Value]]] > ...except that the commas that delimit the fields are in EBCDIC and the > linefeeds that delimit the records are in ASCII . > > I have purposely constructed this example with separatorSuppressionPolicy > set to 'suppressedAtEndLax'. This means that field1 and field2 could both > be terminated by either of > a) an EBCDIC comma or > b) an ASCII line feed > > This is a very artificial example, but it is valid DFDL and should not > produce schema definition error. If we use the term 'scan' when discussing > lengthKind='delimited' then we need to be careful how to define the term > 'scannable' - otherwise we might appear to prohibit things that are > actually valid. I think this is the same point that Steve was making. > > Most implementations will do the same as Daffodil and will use a regex > engine to implement all types of scanning, including > lengthKind='delimited'. It's the most natural solution, and it can be made > to work as long as > a) the implementation takes extra care if/when the encoding changes within > a component and > b) the implementation either does not support the raw byte entity, or only > supports it when it can be translated to a valid character in the > component's encoding. > > There is huge scope for confusion in this area - most readers of the DFDL > specification will assume that delimited scanning can be implemented using > regex technology + EBNF grammar. It may be possible ( I'm not claiming that > it is btw ), but the grammar would be decidedly non-trivial. so the DFDL > specification should take extra care to be unambiguous when discussing how > to scan for terminating markup. > > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: *kimbert(a)uk.ibm.com* <kimbert(a)uk.ibm.com> > Tel. 01962-816742 > Internal tel. 37246742 > > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com> > > > To: Steve Hanson/UK/IBM@IBMGB, > Cc: Tim Kimber/UK/IBM@IBMGB > Date: 25/07/2013 19:04 > Subject: clarification needed for scan and scannable > ------------------------------ > > > > > If a user has read the Glossary and noted the definition of 'scannable' > then when he sees the terms 'scanning' and 'scanned' in 12.3.2 he may think > that this implies data being read by lengthKind must be 'scannable', and it > is not so. That's how I read the spec, hence my original comment. I find > it confusing to define 'scan' and then not to define 'scannable' as 'able > to be *scan*ned'. > Ok, so i am rethinking how we express what we mean by scan. > > scan - verb. The action of attempting to find something in characters. > Implies decoding the characters from code units to character codes. > > A scan can succeed on real data, or fail (no match, not found), but both > are normal behaviors for a scan. > > However, this is predicated on the data at least being meaningfully > decoded into characters of a single character set encoding. This is because > our regex technology isn't expected to be able to shift encodings mid > match/scan, nor is it expected to be able to jump over/around things that > aren't textual so as not to misinterpret them as code units of characters. > > So saying a schema component is scannable - means the schema expresses the > assumption that the data is expected to be all textual/characters in a > uniform encoding so that it is meaningful to talk about scanning it with > the kind of regex technology we have available today. That is, there's no > need to consider encoding changes mid match, nor what data to jump > over/around. > > This is the sense of "scan" -able that we use the term to mean. Our > expectation as expressed in the schema, is that so long as the data decodes > without error the scan will either succeed or fail normally. > > Actual data is scannable if it meets this expectation. > > Contrast with what happens for an assert with testKind='pattern'. In that > case we scan, but DFDL processors aren't required to provide any guarantee > that the data is scannable, which means the designer of that regex pattern > must really know the data, and know what they are doing, as to avoid the > issues of decode-errors. > > Why do we do this? Because we don't want to have to refactor a complex > type element into two sequences one of which is the stuff at the front that > an assert with test pattern examines, and the other of which is the stuff > after that. In some situations it may be very awkward or impossible to > separate out the part at the front that we need to scrutinize. Instead we > just allow you to put the assert at the beginning, and it is up to the > regex designer to know that the first few fields it looks at the data for, > are scannable (in the sense of the expectation of them being text), or if > not scannable in terms of that expectation from the schema, then at least > that the actual contents of them will not have data that interpreted as > character code points, will cause decode errors. In other words the writer > of a test pattern has to know that either the data is scannable (schema > says so), or that the actual data will actually be scan-able, in that > decode errors won't occur if it is decoded. > > This also provides a backdoor by which one can use regex technology to > match against binary data. But then you really have to understand the data > so as to understand exactly what code units will be found in it, so one can > understand what the regex technology will do. One can even play games with > encoding="iso-8859-1", where no decoding errors are possible, so as to > guarantee no decode errors when scanning binary data that really shouldn't > be thought of as text. > > The part of the data that a test pattern of an assert actually looks at > needs to be scannable by schema expectation, or scannable in actual data > contents. > > If you write a test pattern, expecting to match things in what will turn > out to be the first and second elements of a complex type, but the encoding > is different for the second element, then your test pattern isn't going to > work (you may get false matches, false non-matches, or may get decode > errors) because the assumption that the data is scannable for that first > and second element, is violated. > > Summary: can we just say > > scan - verb > scannable - a schema component is scannable if ... current definition. > Data is scannable with respect to a specific character set encoding if, > when it is interpreted as a sequence of code units, they always decode > without error. > > There are two ways to make "any data" scannable for certain. One - set > encodingErrorPolicy to 'replace'. Now all data will decode without error > because you get substitution characters insteead. > > Two - change encoding to iso-8859-1 or other supported encoding where > every byte is a legal code unit. > > If you are in neither situation One or Two above, then you have to say the > data is scannable, but with respect to a specific encoding. So saying the > data is scannable as ASCII means that it will not contain any bytes that > interpreted as ASCII code units will be illegal. For this example, that > means all the bytes have values from 0 to 127 (high bit not set). If you > know that will be true of your data, then even if it is binary data you can > say it is scannable as ASCII. > > The representation of Packed Decimal numbers are not scannable as ascii, > for example, because any high-nibble digit 8 or greater will cause the byte > containing it to have a value 128 or higher, for which there is no > corresponding ascii code unit. > > Packed Decimal numbers are also not scannable as UTF-8, because many > packed decimal bytes will not be legal utf-8 code unit values, nor will > they be in the ordered arrangements UTF-8 requires. > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > > -- > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * > www.tresys.com* <http://www.tresys.com/> > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 3

Fw: clarification needed: 16.3.1 and totalDigits, fractionDigits and textNumberPattern compatibility checks
by Steve Hanson 28 Aug '13

28 Aug '13

Agreed that the statement should be dropped from the spec. DFDL implementations are free to issue a warning if they detect such a mismatch (so the IBM DFDL check will change from an error to a warning). Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2013 16:31 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 22/08/2013 16:02 Subject: Re: [DFDL-WG] clarification needed: 16.3.1 and totalDigits, fractionDigits and textNumberPattern compatibility checks To comply with the spec as stated, the actual check implemented by IBM DFDL is that the number of fraction/total digits in the DFDL textNumberPattern must not exceed the number of fraction/total digits specified by the facet. That's a more liberal interpretation of 'match' but makes sense. However the spec does not say that the textNumberPattern should not be allowed to describe a number greater than any maxInclusive/maxExclusive facet. And I don't think that there are any other checks to see that DFDL properties clash with schema facets. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Cc: Jessie Chab <jchab(a)tresys.com> Date: 22/08/2013 15:35 Subject: [DFDL-WG] clarification needed: 16.3.1 and totalDigits, fractionDigits and textNumberPattern compatibility checks Sent by: dfdl-wg-bounces(a)ogf.org The spec says that these need to be compatible. "If the pattern uses digits/fractions then these must match any XML schema facets." We've fallen into this trap before. Facet constraints are about the logical value, not the physical rep, and so must make equal sense for binary representations as text representations. I was considering what kinds of static cross checks would make sense here. Example: suppose fractionDigits='2'. Suppose textNumberPattern="##" the type must be xs:decimal since that's the only type that can have fractionDigits constrained in DFDL. Is there a conflict of the facet with the textNumberPattern? I think not. Even though the pattern doesn't allow for any fractional part to be expressed, that means that all values will have less than or equal to 2 fractional digits (because no fractional digits always satisfies that.) Now suppose textNumberPattern="#.########" Now the textNumberPattern will parse and accept things like 1.234 which have more than 2 fraction digits, but suppose the data simply doesn't contain any such. So all number in the data will obey the less than or equal to 2 fractional digits. Lastly, suppose textNumberPattern="9.9999" So, interpreted naively, this requires the data to contain 5 digits, so this seems inconsistent with fractionDigits of 2. However, we know this depends on the strict/lax behavior of ICU libraries, and I believe this will happily parse the data "0.0" without error. Hence the data could still all have 2 or fewer fractional digits. So, I think this idea that the totalDigits and fractionDigits facets can be cross-checked with the textNumberPattern is spurious and we should drop it. I think it dates from a time when we were still confused about the separation of the DFDL Infoset, and facet constraints on its value spaces, which are abstract, versus the XML Infoset, where everything truly is a string as well as whatever its XML Schema type says. Comments? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: DFDL ICU Challenges for Implementation - textNumberExponentRep
by Steve Hanson 28 Aug '13

28 Aug '13

Given that ICU do plan to offer this support, the DFDL spec will remain as it is, with textNumberExponentRep being sensitive to ignoreCase, and DFDL implementations that ICU should include a release note that says all matches today will be case insensitive but are liable to change. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2013 16:15 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 21/08/2013 12:21 Subject: Fw: [DFDL-WG] DFDL ICU Challenges for Implementation Regarding Mike's second point: Here's a link to an ICU ticket on this subject: http://bugs.icu-project.org/trac/ticket/9659 - currently targetted for ICU 52.1 Here's some more details from IBM's calls with ICU on this subject: - Exponent character / ignoreCase : Exponent char is not case sensitive. Is this intentional? * Priority : Medium ICU see two options for this: Option 1: Provide an API call to set a flag on the DecimalFormat object. Option 2: Make it a global policy settable via a config switch. This would allow other 'site policies' to be made settable using the same mechanism. There would be one set of policy flags, including this flag, per address space. There are differences in date/time processing between C and Java that could be dealt with using this mechanism. DFDL needs some of these flags to be configurable at runtime. 2012/10/19 Hit an issue where case handling was inconsistent. Fix needs care to avoid changing default behaviour and thus breaking existing users of the API. Currency and prefix/suffix may need separate switch so global switch for the DecimalFormat not appropriate. Could provide a patch to ICU for setting exponent char for now. 2013/1/17 - ICU external ticket #9659 ICU had an issue with the new API being specific to just case sensitivity of exponent (and not other regions). DFDL clarified the requirement is for an API to change global case sensitivity (not just exponent). This is targetted at ICU51 Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 14/08/2013 16:55 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 14/08/2013 14:14 Subject: [DFDL-WG] DFDL ICU Challenges for Implementation Sent by: dfdl-wg-bounces(a)ogf.org There are a couple of features in DFDL that ICU doesn't support, yet where all or nearly all the related functionality is supported by ICU. Perhaps these aspects of the spec can be revisited? 1) List of Decimal Separators The textStandardDecimalSeparator property is a list of characters. However, ICU only supports a single character. I see lots of potential for error here, confusing diagnostics, etc. It is not consistent with textStandardGrouping separator, which allows only a single character. Is there a use case where we know we need more than one decimal separator? The only thing I can think of is a blend of say classic European-style decimal numbers like "1 234 567,89" and USA style " 1,234,567.89", but ICU won't deal with different grouping separators either. In any case if there are multiple decimal and grouping separators we really don't have these properties right in DFDL. We should require them to be specified not as two separate lists, but as a list of pairs, because grouping separators match up with specific decimal separator values in a format. 2) Case Insensitivity Some properties that we use to configure ICU are affected by ignoreCase="yes", but ICU does not support case insensitivity. The properties are: textStandardExponentRepCharacter textStandardInfinityRep textStandardNaNRep I can certainly imagine a need for case insensitivity here, and even for multiple values for these (though we allow only one for Infinity and NaN). For the infinity and nan reps that isn't so problematic as one can easily do a pre-check before calling ICU, but for the exponent rep, that is needed down in the detailed number format parsing. I can see no certain algorithm other than creating separate number format parsers for each exponent rep character in provided case, and opposite case, and then using them one by one until a successful parse. Is this ok or do we consider this a mistake? 3) We are not very consistent in these properties. We allow multiple textStandardZeroRep values, but only a single textStandardInfinityRep, and only a single textStandardNaNRep. We allow multiple textStandardExponentRepCharacter, and multiple textStandardDecimalSeparator, but only a single textStandardGroupingSeparator. This kind of inconsistency is always problematic for users. Comments? -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Information Update - Location has changed: OGF DFDL WG Call
by Steve Hanson 28 Aug '13

28 Aug '13

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Added Hursley room

1 0

dfdl:lengthKind='prefixed' , different encodings for prefix and content where dfdl:prefixIncludesPrefixLength is ‘yes.
by Steve Hanson 27 Aug '13

27 Aug '13

Assuming that the prefix contains the length in characters then I think this works ok when the encoding is different. The parser will first parse the prefix according to prefixLengthType to get the prefix value, which is always of known length. If prefixIncludesPrefixLength is 'yes' then it subtracts this known length from the prefix value, giving the length of the data, which might be in a different encoding. I think we should continue to allow this. In the past we have talked about a DFDL 2.0 feature that allowed the initiator and terminator to be specified using a simple type, precisely to cover the (rare) cases where the characteristics of these delimiters are different to the data itself. Doing it this way prevents a property explosion on the element itself. I view prefixLengthType as the first example of this principle. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/08/2013 16:20 ----- From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Date: 23/08/2013 14:58 Subject: dfdl:lengthKind='prefixed' , different encodings for prefix and content where prefixIncludesPrefixLength is ‘yes. Hi All, Considering a case similar to that excluded by errata 2.76. An element with lengthKind 'prefixed' and prefixIncludesPrefixLength 'true' but where the prefix type and the element both have lengthUnits 'characters' but have different encodings (or specifically encodings with different lengths of characters). I believe the issue that 2.76 is trying to avoid is the issue of determining the length value in say characters when the prefix contains no characters. I am wondering if there is also a slightly subtler issue when we are calculating a length in characters but where a part of the length is in a different encoding from the other. For example the prefix contains 2 UTF16 (2 byte) characters and the content contains 2 UTF32 (4 byte) characters.. Do we just quote a length in characters regardless of encoding. eg. 4 characters. Or is this confusing .... 2.76 . Section 12.3.4 . When property prefixIncludesPrefixLength is ‘yes’there are some restrictions that need to be added to enable reliable lengths to be calculated: o If the prefix type is lengthKind 'implicit' or 'explicit' then the lengthUnits properties of both the prefix type and the element must be the same. Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development IBM DFDL Development MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Agenda 2013-08-28
by Steve Hanson 27 Aug '13

27 Aug '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13107?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL WG Call (28 Aug 15:00 GDT in DE3S09)
by Steve Hanson 27 Aug '13

27 Aug '13

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Mike unavailable on Tuesday

1 0