- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] DFDL binaryCalendarRep pattern limitations
by Steve Hanson 29 Oct '12

29 Oct '12

DFDL WG agreed that the below behaviour is correct. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Cc: Andrew Edwards/UK/IBM@IBMGB Date: 23/10/2012 10:57 Subject: DFDL binaryCalendarRep pattern limitations For discussion at next DFDL WG call. See summary below but key points are: - DFDL uses a calendar pattern to convert binary calendar 'packed', 'bcd' and 'ibm4690Packed' reps to a schema calendar type - No trimming/padding of binary reps takes place, the parser uses what was extracted from the data - A 'packed' rep will always present an odd number of digits (because of sign nibble) - A 'bcd' rep will always present an even number of digits - ICU gives an error if the number of digits presented to it exceeds the length of the calendar pattern - Therefore the onus is on the user to ensure that the calendar pattern matches the number of digits, eg, by adding leading zeros to the pattern Are we happy with that behaviour? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/10/2012 10:49 ----- From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Date: 22/10/2012 15:36 Subject: DFDL binaryCalendarRep pattern limitations Hi Steve - a summary of the problem is below While adding support for IBM4690 packed representation for binaryNumberRep and binaryCalendarRep, it has become apparent that we may need to place a restriction on the calendarPattern property, depending on the choice of binaryCalendarRep. A problem surfaces in being able to reliably and reversibly distinguish a calendar value when the pattern length is incompatible to the packing type. If we use 'ibm4690packed' and a pattern that is of odd length, we end up matching an even-length string against an odd-length pattern and there isn't necessarily a well-defined defined answer. This is best understood with an example: - Consider a pattern with an odd number of characters, such as calendarPattern=yyyyMMddDDD - A value will require a bytestream of length 6 bytes, which would be serialised as 0x0{y}{y}{y}{y}{M}{M}{d}{d}{D}{D}{D} - For example, 2012-10-22 (day295) would be represented as 0x020121022295. If a parser is represented with this value and pattern, then it will try to match the string "020121022295" against "yyyyMMddDDD". ICU returns an error because the string is longer than the pattern and I can't say I blame it. Should it ignore the zero at the start, or the 5 at the end? Without understanding the pattern, a DFDL parser cannot know. ICU can't resolve the value as it has more than one group in the pattern so it can't resolve to one single solution. This becomes more problematic when we consider behaviour for a pattern of "DDD" and of "SSS" as they expect padding at different ends by default (a value of "0100" resolves as "DDD"=100days and "SSS"=0.010 seconds). For packed representation, the opposite problem occurs: There is always a sign nibble in the packed form, so we will always have a value made up of an odd number of digits. This can't match against all patterns of even length. The solution that we discussed was to require the calendar pattern to have a certain digit count depending on the choice of binaryCalendarRep, and allow the pattern to include number characters as a form of "padding". So for the pattern in the example above, this would have to be changed so that calendarPattern='0'yyyyMMddDDD to force it to have an even digit count. If that fully explains the problem, do you want to take it to the DFDL workgroup and check what the consensus opinion is? Cheers, Andy Andy Edwards - WebSphere Message Broker - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE2 U20 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

simpleType cannot contain assert?
by Mike Beckerle 25 Oct '12

25 Oct '12

Is this just an oversight? I find it very tedious to model if I can't put the asserts on the simpleTypes so that I don't have to repeat them over and over in the model. The composition rule here is simple. If an element has asserts, and its simple type has asserts, both are executed, with the element's asserts run after the simpleType's asserts. ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Clarification needed: pad/trim and delimited length
by Mike Beckerle 25 Oct '12

25 Oct '12

The spec says that pad characters are removed before escape scheme processing. However, in delimited context, I can't even determine the length of the field to trim off the padding unless I can do the escape scheme processing. This is either a Chicken-Egg, or the algorithm for parsing is substantially more complex due to padding. Comments? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

clarification: pad/trim and nilKind='literalValue' for text numeric elements
by Mike Beckerle 25 Oct '12

25 Oct '12

Consider: <element name="e" type="xs:int" nillable="true" dfdl:lengthKind="explicit" dfdl:length="4" dfdl:nilValue="-" dfdl:textPadChar="%SP;" dfdl:textNumberJustification="right" dfdl:representation="text"/> Is a nil justified on the right, or are only number values justified on the right? That is, will [- ] be interpreted as nil, or only [ -] A literal nil value is effectively textual syntax, and has nothing to do with whether the field is number or string, so one might legitimately say that textStringJustification applies to it. But then again.... -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

clarification needed on lengthKind='pattern' inside complexType element with lengthKind='delimited'
by Mike Beckerle 25 Oct '12

25 Oct '12

If I nest an element having lengthKind='pattern' inside a complexType element with lengthKind='delimited' I wanted to clarify the behavior of the pattern matching. I've been assuming the pattern does NOT get stopped by occurrances of the delimiters. That is, the pattern scan is in no way adapted to being inside of a delimited context. The alternative would be that the lengthKind="pattern" element's pattern would be augmented by something allowing it to stop if the delimiter were encountered, possibly taking even escape schemes into account, etc. This does not seem worth it to me. It seems like the behavior for lengthKind='pattern' should be greedy, and blind to enclosing delimiters. Precedent is that lengthKind="explcit" is also blind to enclosing delimiters. Comments? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 2

Broadcast: OGF DFDL Working Group weekly call (continued) (26 Oct 15:00 GDT)
by Steve Hanson 24 Oct '12

24 Oct '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0

Action 188 - Path expressions, empty node sequences, and errors
by Mike Beckerle 23 Oct '12

23 Oct '12

Issue: what is semantics of a path expression returning an empty node sequence. Current spec language says it behaves as if it returned nil. This isn't well formed. nil is not an empty node sequence it's a special reserved value. This definition is neither consistent with XPath (which lets functions decide what the behavior for empty node sequence is depending on the function), nor consistent with use of nil elsewhere in DFDL. *Discussion:* Possible changes 1) Any path expression that evaluates to empty node sequence causes an SDE 2) ditto except PE 3) XPath consistent - let the functions decide. So for string functions, an empty node sequence could be treated as "" as in XPath. An empty node sequence returned as the value of a DFDL Infoset item would depend on the type of the infoset item. For a string it could be "", for a boolean it could be false, etc. 4) ANything else? It is very desirable that they should be schema definition errors because the most likely usage pattern is to create a relative path reaching to a part of the structure that is supposed to exist unconditionally. Since DFDL path expressions are a first order language (meaning you can't construct a path from a string), the DFDL compiler can find the vast majority of Path mistakes (misspelling a path step name for example, or wrong number of "../.." steps in a relative path), all at compile time and issue SDEs for them. The cases where a path might or might not exist will be far more rare. However, there is the issue of deep embedding of a path inside an expression. If we want a DFDL processor to be XPath compatible (roughly), and to be able to be implemented by reusing an XPath implementation, then there is the problem that the DFDL implementation reuses the XPath implementation as a black box, and it does not get to see the path expressions that return empty node sequences unless they are returned to it from the XPath evaluator. An XPath implementation embedded inside a DFDL implementation would happily evaluate concat( path1, path2) and if path1 turned out to be empty node sequence, it would get "" for that, and the DFDL implementation might not have any way to intercept this to implment the more rigorous semantics that issues an SDE (or even a PE). Adopting XPath semantics entirely makes things like concat(../a/complete/nonsense/path, "foobar") into valid code. The path may be meaningless, but that means it will just be treated as "". * Suggested Solution:* We can, however, have our cake and eat it too. Assume we embed an ordinary XPath semantics inside DFDL (choice 3 above). Implementors embed XPath implementations black-box. In this case I believe we badly need the fn:exactly-one(arg) function in the DFDL library so that one can wrap it around almost every path expression to get a processing error if it is not one node, and we need to add a dfdl:nodePath(arg) function (the name 'nodePath' meaning 'is expected to be a path to just one node' - entertain a different name if you prefer) which is the same, but issues an SDE and suggests to the implementation that it should be checked before runtime. This would let a cautious DFDL schema author wrap path expressions with fn:exactly-one or dfdl:nodePath to get the strong checking and behaviour they want. This is tedious, but gives us XPath compatibility and ease of implementation. *Details:* There is the below implication for the spec, among others: In the spec our function signatures use '?' after parameter or return type for expression language functions means they can be either a single value or the empty sequence. If we decide these paths cannot be empty node sequences, then these ? all must be removed. If we decide they can be empty node sequences, then we must specify behavior of each function when empty sequence is the argument. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

OGF DFDL WG Call Agenda 2012-10-23
by Steve Hanson 23 Oct '12

23 Oct '12

Please find agenda for call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Minutes 2012-10-16
by Steve Hanson 23 Oct '12

23 Oct '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is still at v010 here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update
by Steve Hanson 23 Oct '12

23 Oct '12

I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Would like to discuss this when Tim is present, as need to see what IBM DFDL does with separators when backtracking. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix? 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination, so I would prefer not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement). Issue a warning if a discriminator found on a direct child of the choice? b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant, but remember that initiatedContent could have been obtained via scoping rules, so if we made the combination an SDE then we would be forcing users to explictly set initiatedContent to 'no'. I'd be ok with ignoring initiatedContent's PoU resolution behaviour if choiceBranchRef was present on a choice (note that choiceBranchRef can not be scoped). c) choiceBranchRef on choice and a discriminator on a child of the choice. Issue a warning if a discriminator found on a direct child of the choice? 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2