October 2012 - dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Issue 140 spinoff - proposed revised grammar
by Mike Beckerle 19 Nov '12

19 Nov '12

Attached revised grammar 005 - proposed as "to become an errata". On Mon, Jul 30, 2012 at 6:29 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > A couple of further revisions. > > > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Steve Hanson/UK/IBM@IBMGB > Cc: Tim Kimber/UK/IBM@IBMGB > Date: 25/07/2012 16:49 > Subject: Re: Issue 140 spinoff - proposed revised grammar > ------------------------------ > > > > Revised per call today. For your consideration. > > ...mikeb > > On Wed, Jul 11, 2012 at 7:39 AM, Steve Hanson <*smh(a)uk.ibm.com*<smh(a)uk.ibm.com>> > wrote: > I've made some updates which I think solves the complex empty issue, ties > the grammar more closely to the action 140 rep definitions, and addresses > Tim's content v value distinction. > > Let me know what you think. > > > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:*+44-1962-815848* <%2B44-1962-815848> > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com> > > > To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB > Date: 10/07/2012 16:25 > Subject: Re: Issue 140 spinoff - proposed revised grammar > ------------------------------ > > > > > Accept-all-changes on that document before you try to read it. Or use the > attached. > > On Tue, Jul 10, 2012 at 11:19 AM, Mike Beckerle <*mbeckerle.dfdl(a)gmail.com > * <mbeckerle.dfdl(a)gmail.com>> wrote: > I took a crack at revising the grammar to something more accurate, more > useful. It's bigger, but not 3 times as big. > > I like it, but then I'm an implementor, not a user of this spec trying to > devise a schema. Our audience is both however. > > I am curious what you think. > > One thing it cannot capture is the complexity of complex empty's, i.e., > the fact that a complex empty requires one to sometimes descend recursively > first, and then check for empty. From the grammar you would think you can > always check for this first. > > Fact is, you have to descend recursively in order to pop back up, and have > the representation be found to match what I call EmptyElement. But the > grammar can't show that. (well, maybe it could... I'll think about that > further.) > > ...mike > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: *781-330-0412* <781-330-0412> > > > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: *781-330-0412* <781-330-0412> > [attachment "grammar-revision.docx" deleted by Steve Hanson/UK/IBM] > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > [attachment "grammar-revision-003.doc" deleted by Steve Hanson/UK/IBM] > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 5

Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger
by Steve Hanson 19 Nov '12

19 Nov '12

Agreed on DFDL WG call 4th Sept to continue to support twos-complement decimal, integer or nonNegativeInteger, but to make the maximum implementation defined. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 15/08/2012 13:10 Subject: Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Mike Having thought about this since our call yesterday, I am uncomfortable with the proposed narrowing. It means DFDL can not be guaranteed to handle the two's complement result of Java BigInteger.toByteArray(). I don't have a concrete use case of two's complement data > 8 bytes but I am sure that someone in the scientific community will need it at some point. Hence why it is allowed in the spec today. If we do allow it, then we have to decide what happens during unparsing and lengthKind is 'prefixed'. The equivalent of Table 16 for lengthKind 'pattern' is needed for lengthKind 'prefixed', as we discussed. That would imply that the unparsing behaviour would be 'Minimum number of bytes to represent significant digits and sign'. In other words, a two's complement binary integer is created that is just long enough to represent the infoset value. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org Date: 14/08/2012 19:33 Subject: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Sent by: dfdl-wg-bounces(a)ogf.org The spec is not clear on whether decimal, integer, and nonNegativeInteger support representation='binary' with binaryNumberRep='binary' (aka twos-complement). Since these types do not have any size limit, such a capability allows arbitrarily large twos-complement integers, or perhaps it would be an implementation defined limit. I believe the right choice is that it should be disallowed. That is, only binaryNumberRep of 'bcd' or 'packed' should be allowed for these types. These types have always been about decimal-style representations to me. This keeps twos-complement only for the types long, int, short, byte and their unsigned counterparts. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

discriminators near PoU or not
by Mike Beckerle 19 Nov '12

19 Nov '12

I thought this situation was worth discussing. I have a bunch of messages. They all work like this: <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> There are hundreds of those. These all are used in this way: <element name="topLevel"> <complexType> <sequence> <element name="message" minOccurs="1" maxOccurs="unbounded"> <complexType> <choice> <element ref="messageName1"/> <element ref="messageName2"/> .... </choice> </complexType> </element> </sequence> </complexType> </element> So, there are two points of uncertainty here. The choice, and the unbounded array surrounding it. For each message, it is not until the sequence carrying the initiator, which is down inside the message structure, that we know for sure we've got this kind of message. So currently each message element is designed to be used ONLY in the above context of two surrounding points of uncertainty, like so (same message format as above but with TWO discriminators added). <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> <annotation><appinfo...>  <dfdl:discriminator>{ true() }</dfdl:discriminator>  <dfdl:discriminator>{ true() }</dfdl:discriminator> </appinfo></annotation> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> I'd love to have a better solution to this issue. But in the absence of one this works and achieves what I want which is that once it hits the initiator, we discriminate both which message we have, and we discriminate that we in fact have a message. Interesting that I'd really prefer to discriminate them in the opposite order of their nesting. I could discriminate the array the the very start of the message. But I have no way to do that because the discriminators apply in inward-out nesting order, and the nearest enclosing PoU is the choice. So I have to discriminate that one first, and I can't discriminate that one until I see the initiator, which is later into the message. Possible fixes/improvements: allow a label/id on each PoU, and allow referencing that label from a discriminator to say exactly which PoU you are discriminating. This makes the outward reference implied by a discriminator explicit. You still have the context issue, but it's not implied anymore, it is explicit. (This makes the problem like expressions with "../../.." that reach up and out of a construct. They are reaching up and out, but at least you can see directly that they are doing so.) -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance)
by Mike Beckerle 01 Nov '12

01 Nov '12

I'll write this up like an errata, but this is for discussion of whether we believe this is clear and complete. ------------------------------------- *Glossary*: DFDL Statements are the annotation elements dfdl:assert, dfdl:discriminator, dfdl:setVariable, and dfdl:newVariableInstance. *Errata*: Locations where DFDL Statements are allowed to appear are extended to also include Global Element Declarations, and on Simple Type Definitions. *Errata*: Clarification about discriminators: Discriminators exclude Assertions even when combining across references. Beyond the stipulation that there can be only one dfdl:discriminator at any annotation point of the DFDL schema, there are further constraints. A single dfdl:discriminator annotation may appear on an element reference, or on the global element declaration it refers to, or on the simple type appearing immediately within or referenced from the global element declaration. But only one of those places. In addition, if a discriminator occupies one of those places, then no dfdl:assert annotations may appear in any of those locations. A dfdl:discriminator annotation may appear on a group reference or on the model group within the global group definition it refers to. But only one of those places, and similarly, if a discriminator appears in any of those places, then no dfdl:assert annotations may appear in any of those locations. *Errata*: Clarification about the execution order of DFDL Statements when they appear on an element reference or element declaration. DFDL Statement annotations for a given schema component are executed as follows: 1) all relevant DFDL statement annotations are gathered to form a single list which preserves schema-definition order. - For a simpleType definition, the DFDL statement annotations found immediately on it are kept in schema-definition order, and are appended to the end of a list of those from any base simpleType definition. - For an element declaration having simple type, the DFDL statement annotations found immediately on the declaration are appended to the end of the list of those from its simple type. - For an element reference, DFDL statement annotations found immediately on the element reference are appended to the end of the list of those from the global element declaration it references. 2) given the combined list, the annotations are executed as follows: 1. before any parsing of the element, a dfdl:discriminator with testKind="pattern" is executed. 2. if there is no discriminator, then before any parsing of the element, all dfdl:asserts (there could be several) with testKind="pattern" are executed in the order they appear in the list of DFDL statements. 3. The element itself is parsed, or its inputValueCalc property is evaluated to create its value. 4. all newVariableInstance annotations are executed and new variables are placed into scope for the duration of these remaining steps. The statements are executed in the order they appear in the list of DFDL statements. 5. all setVariable annotations are executed. The statements are executed in the order they appear in the list of DFDL statements. 6. if a discriminator is present it is executed 7. if no discriminator is present, then assert annotations can be present, and they are executed. If there are multiple assert annotations the statements are executed in the order they appear in the list of DFDL statements. If the element reference or local element declaration is an array, then this evaluation is repeated for each occurrence of the array. *Discussion: * The above allows the default expressions associated with any statement to refer to the value of the element itself as "." However, there's this anomaly of syntax where things don't seem right: <dfdl:newVariableInstance ref="foo:bar" default="{....some expression ...}"/> creates a new variable for the scope of the entity it annotates. It's clear what this means if this annotation is placed on a sequence or choice. For the children of that sequence/choice the new variable instance is in effect. On a simpleType or element having simple type, similarly it is clear (in that case it's a very local variable, just for the expressions in other newVariableInstance statemsnts, setVariableStatements, and discriminators and assertions). The rub: on an element declaration (or reference) when there is a complex type, its not clear. <element name="foo"> <annotation><appinfo ....> <dfdl:newVariableInstance......>  </appinfo></annotation> <complexType> <sequence> <annotation><appinfo ....> <dfdl:assert>... according to rules above this cannot refer to the newVariableInstance...</dfdl:assert> </appinfo></annotation> .... </sequence> </complexType> </element> The above rules say the newVariableInstance isn't evaluated until AFTER the element is parsed, so the assert down inside on the sequence will NOT see this new variable even tho textually, the newVariableInstance annotation looks like it would be in scope over the sequence. Possible ways to avoid this oddity without messing up evaluation order: allow newVariableInstance only on simpleType, element declarations and element references having simpleType, group references, and sequence/choice. Disallow them on element declarations of complex type or on element references to those. The schema author can always introduce an extra tier of sequence to provide the exact behavior they need, and this otherwise error-prone issue can be avoided.

4 12

proposal: DFDL needs additional function dfdl:characterCode
by Mike Beckerle 01 Nov '12

01 Nov '12

An important use case for DFDL is converting legacy data to/from XML. XML 1.0 disallows a bunch of string characters. If the data contains those characters, then the question arises of what to turn them into that both preserves information content, but also is legal in XML so that you can convert the DFDL infoset into XML without violating XML's constraints. The natural thing to do is create an element containing the character code of the illegal character, as an integer. E.g., character code U+0001 would become an element. Such as: <ccode>1</ccode>. This could be done using a hidden element that is a string, and the element ccode above would have an inputValueCalc that converts the offending character of that string into an integer. But we need a function dfdl:characterCode(str, pos) : int The arguments would be a string, and a position (base 1) within that string, and the return result would be the character code of the character in the string at that position. If pos is out of the bounds of the string (i.e., is negative, 0, or too large), then a processing error would occur. For unparsing the inverse function would also be needed: dfdl:character(intArg) : string. This would return a string containing one character whose codepoint is the intArg. Example Consider this data: 123<0>456<1>789<2>123l where <0> means just one character with codepoint 0, etc. In hex that would be 313233 00 343536 01 373839 02 313233 The best I can think of for modeling this while preserving all information would end up with XML looking like this: <nonXMLString> <fragment><stringData>123</stringData></fragment> <fragment><nonXMLChar><charCode>0</charCode></nonXMLChar></fragment> <fragment><stringData>456</stringData></fragment> <fragment><nonXMLChar><charCode>1</charCode></nonXMLChar></fragment> <fragment><stringData>789</stringData></fragment> <fragment><nonXMLChar><charCode>2</charCode></nonXMLChar></fragment> <fragment><stringData>123</stringData></fragment> </nonXMLString> So our nonXMLString is of a type which is array of fragment, a fragment is a choice of either (legal XML) stringData, or a nonXMLChar. The nonXMLChar has a child element because it will need to convert to from a string so will use inputValueCalc and outputValueCalc to do so, so it needs to be a sequence so that it can have the other hidden elements needed to pull this off. stringData would have lengthKind="pattern" and a pattern that allows any sequence of XML-allowed characters. nonXMLChar would have a hidden first child element of type string of explicit length 1 with an assertion that the string match a pattern that is any of the illegal characters (but just one of them). The charCode child element would inputValueCalc to get the character code of the character. For 8 bit encodings it would be ok as a table lookup in XPath, but for unicode..... we'd need a function that returns a character code. If you just have one embedded illegal character, like NUL, then you could just model it as a separator, which would simplify things considerably (and is possible in a someday XML 1.1 future since NUL is then the only disallowed character.) But for XML 1.0's illegal characters, we need to be able to convert to/from some non-string representation if we are to preserve information content. Hence we need these additional functions. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

4 10

Approaches for handling illegal XML 1.0 characters in DFDL Infoset for XML interchange
by Mike Beckerle 31 Oct '12

31 Oct '12

Thought I'd mention that the preferred approach to this in other software systems seems to be to map the illegal characters to/from the Unicode Private Use Area. So illegal XML character 11 0xB becomes codepoint 0xE00B. Apparently this approach is used by some pieces of commercial software, notably Microsoft Visio. http://msdn.microsoft.com/en-us/library/office/aa218415%28v=office.10%29.as… ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

AUTO: Sunil Dandamudi is out of the office (returning 11/01/2012)
by Sunil Dandamudi 31 Oct '12

31 Oct '12

I am out of the office until 11/01/2012. Note: This is an automated response to your message "dfdl-wg Digest, Vol 76, Issue 45" sent on 10/31/2012 13:36:40. This is the only notification you will receive while this person is away.

1 0

Fw: One email or a flock or... - re: 10.03 draft - open review items - update
by Suman Kalia 31 Oct '12

31 Oct '12

confirmed.. symbols P & E in PIC clause cannot be used together.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… ----- Forwarded by Suman Kalia/Toronto/IBM on 10/31/2012 03:24 PM ----- From: Mahendra S Akkina/Silicon Valley/IBM@IBMUS To: Suman Kalia/Toronto/IBM@IBMCA, Date: 10/31/2012 02:47 PM Subject: Re: Fw: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Hi Suman, You are correct. They can't be used together. Thanks Have a nice day. Mahendra S Akkina Developer, Rational Developer for System z Enterprise Language Importers, Rational Application Developer Ph: (408) - 506 - 5986 Email: akkina(a)us.ibm.com "The more the opposition there is the better. Does a river acquire velocity unless there is resistance? the newer and better a thing is the more opposition it will meet with the onset. It is the opposition which foretells success" - Swami Vivekananda From: Suman Kalia/Toronto/IBM@IBMCA To: Mahendra S Akkina/Silicon Valley/IBM@IBMUS, Date: 10/31/2012 11:40 AM Subject: Fw: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Mahi- Can you please confirm that symbols P & E cannot be used together in a picture clause.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… ----- Forwarded by Suman Kalia/Toronto/IBM on 10/31/2012 02:39 PM ----- From: Suman Kalia/Toronto/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 02:39 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Yes - based on my understanding and reading of spec, symbol P & E cannot be used together in a COBOL PIC clause.. However I will double check with language folks and will send reply.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 12:59 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Thanks Suman. Specifically, please can you confirm that the P symbol and the E symbol may not be used together in a COBOL PIC clause. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 31/10/2012 15:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update
by Suman Kalia 31 Oct '12

31 Oct '12

Yes - based on my understanding and reading of spec, symbol P & E cannot be used together in a COBOL PIC clause.. However I will double check with language folks and will send reply.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 10/31/2012 12:59 PM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Thanks Suman. Specifically, please can you confirm that the P symbol and the E symbol may not be used together in a COBOL PIC clause. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 31/10/2012 15:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

OGF DFDL WG Call Minutes 2012-10-30
by Steve Hanson 31 Oct '12

31 Oct '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/5720?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/3184?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0