- dfdl-wg - lists.ogf.org

proposal: DFDL needs additional function dfdl:characterCode
by Mike Beckerle 01 Nov '12

01 Nov '12

An important use case for DFDL is converting legacy data to/from XML. XML 1.0 disallows a bunch of string characters. If the data contains those characters, then the question arises of what to turn them into that both preserves information content, but also is legal in XML so that you can convert the DFDL infoset into XML without violating XML's constraints. The natural thing to do is create an element containing the character code of the illegal character, as an integer. E.g., character code U+0001 would become an element. Such as: <ccode>1</ccode>. This could be done using a hidden element that is a string, and the element ccode above would have an inputValueCalc that converts the offending character of that string into an integer. But we need a function dfdl:characterCode(str, pos) : int The arguments would be a string, and a position (base 1) within that string, and the return result would be the character code of the character in the string at that position. If pos is out of the bounds of the string (i.e., is negative, 0, or too large), then a processing error would occur. For unparsing the inverse function would also be needed: dfdl:character(intArg) : string. This would return a string containing one character whose codepoint is the intArg. Example Consider this data: 123<0>456<1>789<2>123l where <0> means just one character with codepoint 0, etc. In hex that would be 313233 00 343536 01 373839 02 313233 The best I can think of for modeling this while preserving all information would end up with XML looking like this: <nonXMLString> <fragment><stringData>123</stringData></fragment> <fragment><nonXMLChar><charCode>0</charCode></nonXMLChar></fragment> <fragment><stringData>456</stringData></fragment> <fragment><nonXMLChar><charCode>1</charCode></nonXMLChar></fragment> <fragment><stringData>789</stringData></fragment> <fragment><nonXMLChar><charCode>2</charCode></nonXMLChar></fragment> <fragment><stringData>123</stringData></fragment> </nonXMLString> So our nonXMLString is of a type which is array of fragment, a fragment is a choice of either (legal XML) stringData, or a nonXMLChar. The nonXMLChar has a child element because it will need to convert to from a string so will use inputValueCalc and outputValueCalc to do so, so it needs to be a sequence so that it can have the other hidden elements needed to pull this off. stringData would have lengthKind="pattern" and a pattern that allows any sequence of XML-allowed characters. nonXMLChar would have a hidden first child element of type string of explicit length 1 with an assertion that the string match a pattern that is any of the illegal characters (but just one of them). The charCode child element would inputValueCalc to get the character code of the character. For 8 bit encodings it would be ok as a table lookup in XPath, but for unicode..... we'd need a function that returns a character code. If you just have one embedded illegal character, like NUL, then you could just model it as a separator, which would simplify things considerably (and is possible in a someday XML 1.1 future since NUL is then the only disallowed character.) But for XML 1.0's illegal characters, we need to be able to convert to/from some non-string representation if we are to preserve information content. Hence we need these additional functions. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

4 10

Re: [DFDL-WG] question/clarification - asserts using test patterns
by Steve Hanson 01 Nov '12

01 Nov '12

From DFDL WG call 2012-10-30: 190 Clarify rules for assert/discriminator testKind 'pattern' (All) 23/10: Need to be clear on data position and whether it is just for text representations. 30/10: Closed. To comply with the timing rules being proposed in action 186, where these things are executed first before a 'format' annotation, the data position must be the beginning of the representation (note warning useful when alignment present). As these things can be used on various objects, the only rule regarding text is that dfdl:encoding must have a value in scope. Errata taken. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 16/10/2012 17:47 Subject: Re: [DFDL-WG] question/clarification - asserts using test patterns Agree that testKind 'pattern' is all about matching the physical data, and not logical value. I am thinking that we should impose a similar restriction on testKind 'pattern' to that imposed on lengthKind 'pattern', specifically (from the latest spec draft): Any element (complex or simple type) may have a dfdl:lengthKind of 'pattern' as long as the data in the content region (which can be either the SimpleContent region or the ComplexContent region defined in Section 9.2) of the element is legal in the stated encoding of that element. Data which satisfies this is referred to as scannable data. Specifically, data is scannable and length kind ‘pattern’ can be used only for: o elements of simple type with representation 'text' o elements of complex type where: 1. all simple child elements must have representation 'text' and have the same encoding as the parent complex element, and 2. all complex child elements must themselves follow 1 and 2 (recursively). This restriction was added because of problems trying to apply patterns to binary data. I don't think it will restrict the utility of testKind 'pattern' in practice. What do you think? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 15/10/2012 20:54 Subject: Re: [DFDL-WG] question/clarification - asserts using test patterns Sent by: dfdl-wg-bounces(a)ogf.org Good questions. The intent of this DFDL feature is as follows: - a point of uncertainty (POI) cannot be resolved using an initiator ( the simplest option ). The data format just doesn't work that way. - it could be resolved using an assert or a discriminator, but that would be too heavyweight. - a simple inspection of the data format reveals that the discrimination can be done by testing the first few characters of each branch of the POI. Example: SWIFT 50K ( multi-line address field): :32A:060929EUR25,36&hex;0D&hex;&hex;0A&hex; :33B:EUR56,78&hex;0D&hex;&hex;0A&hex; :50K:/IT60X0542811101000000123456 ABC Corporation Times Square 7 NY 1 LINE 2 LINE 3 :52A:/<etc> Note that field 50K contains lines of address data, but the actual number of lines is not known. So how will the DFDL parser know when the 50K field has completed? Answer: it encounters a line that starts with a colon. Now, the most natural way to model SWIFT field 50K is as a series of lines. The SWIFT XML format defines it this way. If you work through the possibilities, it turns out that the only way to achieve this using discriminators is: - cause the parser to parse each line and put it into the info set - add a discriminator to the repeating 'addressLine' element. The DFDL expression would be something like this: { if ( fn:exists(./NameAddress_Line) ) then (fn:not(fn:starts-with(./NameAddress_Line, ':'))) else xs:boolean("true") } That's a very expensive way to achieve the intended goal, which is 'treat the data as another addressLine if the next character is a colon'. So that was the motivation for the feature. To answer the questions: - not intended to be limited to xs:string only - not intended to be limited to elements with text representation ( because dfdl:represention only applies to simple elements, and the POI might be a group or an element.) - is intended to be matched against text or binary data, starting at the POI's byte offset. If the element's representation is binary then the 'encoding' property will be required. Sounds as if the spec needs to be clarified in this area. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 15/10/2012 19:26 Subject: [DFDL-WG] question/clarification - asserts using test patterns Sent by: dfdl-wg-bounces(a)ogf.org Question: Is an assertion using a regular expression pattern allowed on (a) xs:string type elements (b) any data with text representation (c) any data with text or binary representation and, does the regular expression apply to the representation or the logical data value? (a) is the only case that is not ambiguous, because the representation and the logical value are the same thing. For everything else, there’s the question of whether the test is on the representation or the logical value. If it's the logical value, then how is a regex made meaningful on a logical value that is, for example, a number, without defining a canonical representation to which the logical value is converted? If it's to apply to the representation, then exactly what data? (Eg., what grammar region) is subject to the regex? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Approaches for handling illegal XML 1.0 characters in DFDL Infoset for XML interchange
by Mike Beckerle 31 Oct '12

31 Oct '12

Thought I'd mention that the preferred approach to this in other software systems seems to be to map the illegal characters to/from the Unicode Private Use Area. So illegal XML character 11 0xB becomes codepoint 0xE00B. Apparently this approach is used by some pieces of commercial software, notably Microsoft Visio. http://msdn.microsoft.com/en-us/library/office/aa218415%28v=office.10%29.as… ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

AUTO: Sunil Dandamudi is out of the office (returning 11/01/2012)
by Sunil Dandamudi 31 Oct '12

31 Oct '12

I am out of the office until 11/01/2012. Note: This is an automated response to your message "dfdl-wg Digest, Vol 76, Issue 45" sent on 10/31/2012 13:36:40. This is the only notification you will receive while this person is away.

1 0

Fw: One email or a flock or... - re: 10.03 draft - open review items - update
by Suman Kalia 31 Oct '12

31 Oct '12

confirmed.. symbols P & E in PIC clause cannot be used together.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… ----- Forwarded by Suman Kalia/Toronto/IBM on 10/31/2012 03:24 PM ----- From: Mahendra S Akkina/Silicon Valley/IBM@IBMUS To: Suman Kalia/Toronto/IBM@IBMCA, Date: 10/31/2012 02:47 PM Subject: Re: Fw: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Hi Suman, You are correct. They can't be used together. Thanks Have a nice day. Mahendra S Akkina Developer, Rational Developer for System z Enterprise Language Importers, Rational Application Developer Ph: (408) - 506 - 5986 Email: akkina(a)us.ibm.com "The more the opposition there is the better. Does a river acquire velocity unless there is resistance? the newer and better a thing is the more opposition it will meet with the onset. It is the opposition which foretells success" - Swami Vivekananda From: Suman Kalia/Toronto/IBM@IBMCA To: Mahendra S Akkina/Silicon Valley/IBM@IBMUS, Date: 10/31/2012 11:40 AM Subject: Fw: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Mahi- Can you please confirm that symbols P & E cannot be used together in a picture clause.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… ----- Forwarded by Suman Kalia/Toronto/IBM on 10/31/2012 02:39 PM ----- From: Suman Kalia/Toronto/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 02:39 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Yes - based on my understanding and reading of spec, symbol P & E cannot be used together in a COBOL PIC clause.. However I will double check with language folks and will send reply.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 12:59 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Thanks Suman. Specifically, please can you confirm that the P symbol and the E symbol may not be used together in a COBOL PIC clause. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 31/10/2012 15:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update
by Suman Kalia 31 Oct '12

31 Oct '12

Yes - based on my understanding and reading of spec, symbol P & E cannot be used together in a COBOL PIC clause.. However I will double check with language folks and will send reply.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 12:59 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Thanks Suman. Specifically, please can you confirm that the P symbol and the E symbol may not be used together in a COBOL PIC clause. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 31/10/2012 15:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

OGF DFDL WG Call Minutes 2012-10-30
by Steve Hanson 31 Oct '12

31 Oct '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/5720?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/3184?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

7-bit ascii packed together
by Mike Beckerle 31 Oct '12

31 Oct '12

I have a data format in front of me that has 64 7-bit ASCII characters, but the format has them bit-packed, i.e., 448 = 7 * 64 bits, so ....the character codes aren't octet/byte aligned. Furthermore, the 'string' either uses up the entire 64 character maximum length OR it has a terminating character which is a 0x7F character code. I believe I was the advocate for a position that character codes should always be 8-bit aligned. That would be because I had never seen anything like this. I am told there are also 6-bit ascii-variations, similarly packed together to save space. BTW: This occurs in a specific US MIL STD message header format, so it's not like it's some obscure unused corner case. Right now, the best I think I can do is to model this data not as a string at all, but as an array of integers, each one having 7-bit length, and not aligned (that is, aligned to 1-bit). Doing that I can use occursCountKind='parsed', and an assertion to deal with the optional termination by 0x7F value. To handle this as a string, we'd need to be able to specify that the character codes are not aligned, and the width of the bit-fields making up each character code. Or I suppose we could just say this is a special kind of character set encoding "ASCII-7-bit-packed" or something. Having that, I could deal with the termination via a choice of either the terminated flavor, or the fixed length flavor (which excludes the terminator) by way of a choice of two strings each having a lengthKind="pattern". Comments? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update
by Steve Hanson 31 Oct '12

31 Oct '12

Thanks Suman. Specifically, please can you confirm that the P symbol and the E symbol may not be used together in a COBOL PIC clause. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 31/10/2012 15:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update
by Suman Kalia 31 Oct '12

31 Oct '12

We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0