- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2012-10-30 - resending
by Steve Hanson 02 Nov '12

02 Nov '12

** Resending as some maintenance on Redmine changed all the doc ids! ** Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/8145?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

clarification needed: assert evaluation order and arrays
by Mike Beckerle 01 Nov '12

01 Nov '12

Question: If I write <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> I have two sources of constraints. One is the pattern, the other the min/max occurs. Does that one assertion calling dfdl:checkConstraints mean both will be checked? That is, one check occurring as each element is parsed, and the other at the end of the array? Assume I am not using any validation option, so the DFDL processor would not otherwise check the max/minOccurs because occursCountKind is parsed. Will the checkConstraints fail as soon as we parse the 11th element (is it checking the min/max occurs for each element occurrence as it is parsed), or do we parse as many as we can, and fail only when we check and find out that the entire array has 36 elements that were successfully parsed? What I would like the above to mean is this: 1) as each element occurrence is parsed we check the pattern and parse error (assertion failed) if there is no match. 2) Also after a successful parse of an occurrence, we check that the index is <=10, and parse-error (assertion failed) if not. 3) at the end of the array, we check that the number of occurrences is >= 5. If not we get a parse error (assertion failed). Comments? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance)
by Mike Beckerle 01 Nov '12

01 Nov '12

I'll write this up like an errata, but this is for discussion of whether we believe this is clear and complete. ------------------------------------- *Glossary*: DFDL Statements are the annotation elements dfdl:assert, dfdl:discriminator, dfdl:setVariable, and dfdl:newVariableInstance. *Errata*: Locations where DFDL Statements are allowed to appear are extended to also include Global Element Declarations, and on Simple Type Definitions. *Errata*: Clarification about discriminators: Discriminators exclude Assertions even when combining across references. Beyond the stipulation that there can be only one dfdl:discriminator at any annotation point of the DFDL schema, there are further constraints. A single dfdl:discriminator annotation may appear on an element reference, or on the global element declaration it refers to, or on the simple type appearing immediately within or referenced from the global element declaration. But only one of those places. In addition, if a discriminator occupies one of those places, then no dfdl:assert annotations may appear in any of those locations. A dfdl:discriminator annotation may appear on a group reference or on the model group within the global group definition it refers to. But only one of those places, and similarly, if a discriminator appears in any of those places, then no dfdl:assert annotations may appear in any of those locations. *Errata*: Clarification about the execution order of DFDL Statements when they appear on an element reference or element declaration. DFDL Statement annotations for a given schema component are executed as follows: 1) all relevant DFDL statement annotations are gathered to form a single list which preserves schema-definition order. - For a simpleType definition, the DFDL statement annotations found immediately on it are kept in schema-definition order, and are appended to the end of a list of those from any base simpleType definition. - For an element declaration having simple type, the DFDL statement annotations found immediately on the declaration are appended to the end of the list of those from its simple type. - For an element reference, DFDL statement annotations found immediately on the element reference are appended to the end of the list of those from the global element declaration it references. 2) given the combined list, the annotations are executed as follows: 1. before any parsing of the element, a dfdl:discriminator with testKind="pattern" is executed. 2. if there is no discriminator, then before any parsing of the element, all dfdl:asserts (there could be several) with testKind="pattern" are executed in the order they appear in the list of DFDL statements. 3. The element itself is parsed, or its inputValueCalc property is evaluated to create its value. 4. all newVariableInstance annotations are executed and new variables are placed into scope for the duration of these remaining steps. The statements are executed in the order they appear in the list of DFDL statements. 5. all setVariable annotations are executed. The statements are executed in the order they appear in the list of DFDL statements. 6. if a discriminator is present it is executed 7. if no discriminator is present, then assert annotations can be present, and they are executed. If there are multiple assert annotations the statements are executed in the order they appear in the list of DFDL statements. If the element reference or local element declaration is an array, then this evaluation is repeated for each occurrence of the array. *Discussion: * The above allows the default expressions associated with any statement to refer to the value of the element itself as "." However, there's this anomaly of syntax where things don't seem right: <dfdl:newVariableInstance ref="foo:bar" default="{....some expression ...}"/> creates a new variable for the scope of the entity it annotates. It's clear what this means if this annotation is placed on a sequence or choice. For the children of that sequence/choice the new variable instance is in effect. On a simpleType or element having simple type, similarly it is clear (in that case it's a very local variable, just for the expressions in other newVariableInstance statemsnts, setVariableStatements, and discriminators and assertions). The rub: on an element declaration (or reference) when there is a complex type, its not clear. <element name="foo"> <annotation><appinfo ....> <dfdl:newVariableInstance......>  </appinfo></annotation> <complexType> <sequence> <annotation><appinfo ....> <dfdl:assert>... according to rules above this cannot refer to the newVariableInstance...</dfdl:assert> </appinfo></annotation> .... </sequence> </complexType> </element> The above rules say the newVariableInstance isn't evaluated until AFTER the element is parsed, so the assert down inside on the sequence will NOT see this new variable even tho textually, the newVariableInstance annotation looks like it would be in scope over the sequence. Possible ways to avoid this oddity without messing up evaluation order: allow newVariableInstance only on simpleType, element declarations and element references having simpleType, group references, and sequence/choice. Disallow them on element declarations of complex type or on element references to those. The schema author can always introduce an extra tier of sequence to provide the exact behavior they need, and this otherwise error-prone issue can be avoided.

4 12

proposal: DFDL needs additional function dfdl:characterCode
by Mike Beckerle 01 Nov '12

01 Nov '12

An important use case for DFDL is converting legacy data to/from XML. XML 1.0 disallows a bunch of string characters. If the data contains those characters, then the question arises of what to turn them into that both preserves information content, but also is legal in XML so that you can convert the DFDL infoset into XML without violating XML's constraints. The natural thing to do is create an element containing the character code of the illegal character, as an integer. E.g., character code U+0001 would become an element. Such as: <ccode>1</ccode>. This could be done using a hidden element that is a string, and the element ccode above would have an inputValueCalc that converts the offending character of that string into an integer. But we need a function dfdl:characterCode(str, pos) : int The arguments would be a string, and a position (base 1) within that string, and the return result would be the character code of the character in the string at that position. If pos is out of the bounds of the string (i.e., is negative, 0, or too large), then a processing error would occur. For unparsing the inverse function would also be needed: dfdl:character(intArg) : string. This would return a string containing one character whose codepoint is the intArg. Example Consider this data: 123<0>456<1>789<2>123l where <0> means just one character with codepoint 0, etc. In hex that would be 313233 00 343536 01 373839 02 313233 The best I can think of for modeling this while preserving all information would end up with XML looking like this: <nonXMLString> <fragment><stringData>123</stringData></fragment> <fragment><nonXMLChar><charCode>0</charCode></nonXMLChar></fragment> <fragment><stringData>456</stringData></fragment> <fragment><nonXMLChar><charCode>1</charCode></nonXMLChar></fragment> <fragment><stringData>789</stringData></fragment> <fragment><nonXMLChar><charCode>2</charCode></nonXMLChar></fragment> <fragment><stringData>123</stringData></fragment> </nonXMLString> So our nonXMLString is of a type which is array of fragment, a fragment is a choice of either (legal XML) stringData, or a nonXMLChar. The nonXMLChar has a child element because it will need to convert to from a string so will use inputValueCalc and outputValueCalc to do so, so it needs to be a sequence so that it can have the other hidden elements needed to pull this off. stringData would have lengthKind="pattern" and a pattern that allows any sequence of XML-allowed characters. nonXMLChar would have a hidden first child element of type string of explicit length 1 with an assertion that the string match a pattern that is any of the illegal characters (but just one of them). The charCode child element would inputValueCalc to get the character code of the character. For 8 bit encodings it would be ok as a table lookup in XPath, but for unicode..... we'd need a function that returns a character code. If you just have one embedded illegal character, like NUL, then you could just model it as a separator, which would simplify things considerably (and is possible in a someday XML 1.1 future since NUL is then the only disallowed character.) But for XML 1.0's illegal characters, we need to be able to convert to/from some non-string representation if we are to preserve information content. Hence we need these additional functions. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

4 10

Re: [DFDL-WG] question/clarification - asserts using test patterns
by Steve Hanson 01 Nov '12

01 Nov '12

From DFDL WG call 2012-10-30: 190 Clarify rules for assert/discriminator testKind 'pattern' (All) 23/10: Need to be clear on data position and whether it is just for text representations. 30/10: Closed. To comply with the timing rules being proposed in action 186, where these things are executed first before a 'format' annotation, the data position must be the beginning of the representation (note warning useful when alignment present). As these things can be used on various objects, the only rule regarding text is that dfdl:encoding must have a value in scope. Errata taken. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 16/10/2012 17:47 Subject: Re: [DFDL-WG] question/clarification - asserts using test patterns Agree that testKind 'pattern' is all about matching the physical data, and not logical value. I am thinking that we should impose a similar restriction on testKind 'pattern' to that imposed on lengthKind 'pattern', specifically (from the latest spec draft): Any element (complex or simple type) may have a dfdl:lengthKind of 'pattern' as long as the data in the content region (which can be either the SimpleContent region or the ComplexContent region defined in Section 9.2) of the element is legal in the stated encoding of that element. Data which satisfies this is referred to as scannable data. Specifically, data is scannable and length kind ‘pattern’ can be used only for: o elements of simple type with representation 'text' o elements of complex type where: 1. all simple child elements must have representation 'text' and have the same encoding as the parent complex element, and 2. all complex child elements must themselves follow 1 and 2 (recursively). This restriction was added because of problems trying to apply patterns to binary data. I don't think it will restrict the utility of testKind 'pattern' in practice. What do you think? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 15/10/2012 20:54 Subject: Re: [DFDL-WG] question/clarification - asserts using test patterns Sent by: dfdl-wg-bounces(a)ogf.org Good questions. The intent of this DFDL feature is as follows: - a point of uncertainty (POI) cannot be resolved using an initiator ( the simplest option ). The data format just doesn't work that way. - it could be resolved using an assert or a discriminator, but that would be too heavyweight. - a simple inspection of the data format reveals that the discrimination can be done by testing the first few characters of each branch of the POI. Example: SWIFT 50K ( multi-line address field): :32A:060929EUR25,36&hex;0D&hex;&hex;0A&hex; :33B:EUR56,78&hex;0D&hex;&hex;0A&hex; :50K:/IT60X0542811101000000123456 ABC Corporation Times Square 7 NY 1 LINE 2 LINE 3 :52A:/<etc> Note that field 50K contains lines of address data, but the actual number of lines is not known. So how will the DFDL parser know when the 50K field has completed? Answer: it encounters a line that starts with a colon. Now, the most natural way to model SWIFT field 50K is as a series of lines. The SWIFT XML format defines it this way. If you work through the possibilities, it turns out that the only way to achieve this using discriminators is: - cause the parser to parse each line and put it into the info set - add a discriminator to the repeating 'addressLine' element. The DFDL expression would be something like this: { if ( fn:exists(./NameAddress_Line) ) then (fn:not(fn:starts-with(./NameAddress_Line, ':'))) else xs:boolean("true") } That's a very expensive way to achieve the intended goal, which is 'treat the data as another addressLine if the next character is a colon'. So that was the motivation for the feature. To answer the questions: - not intended to be limited to xs:string only - not intended to be limited to elements with text representation ( because dfdl:represention only applies to simple elements, and the POI might be a group or an element.) - is intended to be matched against text or binary data, starting at the POI's byte offset. If the element's representation is binary then the 'encoding' property will be required. Sounds as if the spec needs to be clarified in this area. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 15/10/2012 19:26 Subject: [DFDL-WG] question/clarification - asserts using test patterns Sent by: dfdl-wg-bounces(a)ogf.org Question: Is an assertion using a regular expression pattern allowed on (a) xs:string type elements (b) any data with text representation (c) any data with text or binary representation and, does the regular expression apply to the representation or the logical data value? (a) is the only case that is not ambiguous, because the representation and the logical value are the same thing. For everything else, there’s the question of whether the test is on the representation or the logical value. If it's the logical value, then how is a regex made meaningful on a logical value that is, for example, a number, without defining a canonical representation to which the logical value is converted? If it's to apply to the representation, then exactly what data? (Eg., what grammar region) is subject to the regex? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Approaches for handling illegal XML 1.0 characters in DFDL Infoset for XML interchange
by Mike Beckerle 31 Oct '12

31 Oct '12

Thought I'd mention that the preferred approach to this in other software systems seems to be to map the illegal characters to/from the Unicode Private Use Area. So illegal XML character 11 0xB becomes codepoint 0xE00B. Apparently this approach is used by some pieces of commercial software, notably Microsoft Visio. http://msdn.microsoft.com/en-us/library/office/aa218415%28v=office.10%29.as… ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

AUTO: Sunil Dandamudi is out of the office (returning 11/01/2012)
by Sunil Dandamudi 31 Oct '12

31 Oct '12

I am out of the office until 11/01/2012. Note: This is an automated response to your message "dfdl-wg Digest, Vol 76, Issue 45" sent on 10/31/2012 13:36:40. This is the only notification you will receive while this person is away.

1 0

Fw: One email or a flock or... - re: 10.03 draft - open review items - update
by Suman Kalia 31 Oct '12

31 Oct '12

confirmed.. symbols P & E in PIC clause cannot be used together.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… ----- Forwarded by Suman Kalia/Toronto/IBM on 10/31/2012 03:24 PM ----- From: Mahendra S Akkina/Silicon Valley/IBM@IBMUS To: Suman Kalia/Toronto/IBM@IBMCA, Date: 10/31/2012 02:47 PM Subject: Re: Fw: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Hi Suman, You are correct. They can't be used together. Thanks Have a nice day. Mahendra S Akkina Developer, Rational Developer for System z Enterprise Language Importers, Rational Application Developer Ph: (408) - 506 - 5986 Email: akkina(a)us.ibm.com "The more the opposition there is the better. Does a river acquire velocity unless there is resistance? the newer and better a thing is the more opposition it will meet with the onset. It is the opposition which foretells success" - Swami Vivekananda From: Suman Kalia/Toronto/IBM@IBMCA To: Mahendra S Akkina/Silicon Valley/IBM@IBMUS, Date: 10/31/2012 11:40 AM Subject: Fw: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Mahi- Can you please confirm that symbols P & E cannot be used together in a picture clause.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… ----- Forwarded by Suman Kalia/Toronto/IBM on 10/31/2012 02:39 PM ----- From: Suman Kalia/Toronto/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 02:39 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Yes - based on my understanding and reading of spec, symbol P & E cannot be used together in a COBOL PIC clause.. However I will double check with language folks and will send reply.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 12:59 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Thanks Suman. Specifically, please can you confirm that the P symbol and the E symbol may not be used together in a COBOL PIC clause. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 31/10/2012 15:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update
by Suman Kalia 31 Oct '12

31 Oct '12

Yes - based on my understanding and reading of spec, symbol P & E cannot be used together in a COBOL PIC clause.. However I will double check with language folks and will send reply.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 12:59 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Thanks Suman. Specifically, please can you confirm that the P symbol and the E symbol may not be used together in a COBOL PIC clause. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 31/10/2012 15:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

OGF DFDL WG Call Minutes 2012-10-30
by Steve Hanson 31 Oct '12

31 Oct '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/5720?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/3184?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0