November 2012 - dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Fw: setVariable with no value set - supported?
by Steve Hanson 22 Nov '12

22 Nov '12

Proposal to allow empty string to be used as a value for DFDL variables of type xs:string and xs:hexBinary (or derived from them) without resorting to using an expression. To set empty string use the attribute syntax, eg: <dfdl:defineVariable name="myVar" type="xs:string" defaultValue=""/> <dfdl:newVariableInstance ref="myVar" defaultValue=""/> <dfdl:setVariable ref="myVar" value=""/> The following are schema definition errors as they do not set the value to anything. <dfdl:setVariable ref="myVar"></dfdl:setVariable> <dfdl:setVariable ref="myVar"/> That allows us to apply a semantic to these syntaxes in a future DFDL release if we need to (eg, unsets the value). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Mark Frost/UK/IBM, Tim Kimber/UK/IBM@IBMGB Date: 21/11/2012 09:48 Subject: Re: [DFDL-WG] Fw: setVariable with no value set - supported? Here's what Sandy had to say on this. He raised a good point about the future possibility of unsetting a variable using the empty syntax. Steve Hanson Hi Sandy - please can you ping back when free ? Ta! Sandy (S.) Gao hi steve Steve Hanson Hi ◄ From DFDL WG call today... ◄ 2. What is correct syntax for setting the value of a variable to empty string? DFDL variables allow both XSDL lexical form and DFDL expressions as values, so it seems plausible that <dfdl:setVariable ref="x" value=""/> or even <dfdl:setVariable ref="x"/> is an allowed syntax for setting the value of a string or hexBinary variable to empty string ◄ What's your opinion here? ◄ Syntax for setting a DFDL variable to empty string. Sandy (S.) Gao the first looks reasonable. if you want to distinguish between an unset value / null value vs. empty value, then you may not want to use the second. in xml, there's a difference between an attribute that's not present and an attribute with empty value. ◄ unless you are using element content as the value for the second case. Steve Hanson That's the issue - DFDL treats the attr form and element form as equivalent Sandy (S.) Gao so for non-empty value, you could <setV ... value="abc"/> or <setV ...>abc</setV>? Steve Hanson correct Sandy (S.) Gao ic... then i suppose you need to be consistent and allow both of those forms. ◄ but can someone specify a null value? or unset a variable? Steve Hanson null not allowed ◄ I don't think we allow unsetting either ◄ Good points though Sandy (S.) Gao if a value is required, then i don't see any issue in this approach. Steve Hanson ok - I will raise the unset issue, as we don't want to box ourselves in for future ◄ thanks Sandy Sandy (S.) Gao np... i guess you'll need to add a new element e.g. <unsetValue> or a special value <setV...>##unset</setV> to support it. Steve Hanson yes - if we don't allow the empty element + missing attr to mean that Sandy (S.) Gao exactly Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Date: 19/11/2012 15:11 Subject: Re: [DFDL-WG] Fw: setVariable with no value set - supported? Here is the way I would set a variable to the empty string. <dfdl:setVariable ref="someQName">{ "" }</dfdl:setVariable> Note that the value is an expression and so must have the {...} surround, the expression returns an empty string as its value. If you don't mind quoting hell, then <dfdl:setVariable ref="someQName" value='{ "" }'/> would also work. The DFDL validator doesn't catch the error about the value attribute being missing because both the above forms are accepted and XML Schema's validation can't enforce an OR of "you have to have an element child value OR the value attribute". This check has to be done by the DFDL implementation itself, not just XSD validation of the DFDL annotations. On Mon, Nov 19, 2012 at 9:34 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: See below... Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/11/2012 14:23 ----- From: Steve Hanson/UK/IBM To: Mark Frost/UK/IBM, Cc: Shane Cartledge/Toronto/IBM@IBMCA, Anita Leung/Toronto/IBM@IBMCA Date: 16/11/2012 11:46 Subject: Re: setVariable with no value set - supported? The specific example that Shane gives is invalid anyway as you can't set the value of one of the pre-defined DFDL variables. Assuming a realistic example like: <dfdl:setVariable ref="myVar"/> I'm wondering how you would set empty string for an element of type xs:string or xs:hexBinary. <dfdl:setVariable ref="myVar" value=""/> But why isn't that the same as <dfdl:setVariable ref="myVar"></dfdl:setVariable> which in XML rules is equivalent to: <dfdl:setVariable ref="myVar"/> I will raise with the DFDL WG. Thanks Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mark Frost/UK/IBM To: Shane Cartledge/Toronto/IBM@IBMCA, Steve Hanson/UK/IBM@IBMGB, Cc: Anita Leung/Toronto/IBM@IBMCA Date: 16/11/2012 11:19 Subject: Re: setVariable with no value set - supported? Hi Shane, While the spec isn't super-specific, I'd interpret it to mean "you must give exactly one of the value attribute or element value" Spec Section 7.9 : "The syntax supports both a dfdl:value attribute and the 'value' being specified by the element value. Only one or the other may be present. (Schema definition error otherwise.)" Steve: do you think altering the spec wording above to something like " Exactly one of the dfdl:value or the element value must be present. (Schema definition error otherwise.)" Regards, Mark Frost _____________________________________ MP 211, IBM Hursley, Winchester, SO21 2JN Phone: (01962) 817009 or x247009 Email: frostmar(a)uk.ibm.com From: Shane Cartledge/Toronto/IBM@IBMCA To: Mark Frost/UK/IBM@IBMGB, Cc: Anita Leung/Toronto/IBM@IBMCA Date: 15/11/2012 22:56 Subject: setVariable with no value set - supported? Hi Mark, Do you know if DFDL should supports a setVariable with no value set? Thought this may have already come up with your runtime testing. For example: <dfdl:setVariable ref="dfdl:binaryFloatRep"/> The DFDL specification is not clear on this point, and the DFDL validator does not flag it as an error. Thanks. Shane Cartledge Business Integration Tooling Phone: 1-905-413-5171, Tie Line: 313-5171 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 192: URN scheme - was: Re: new action item needed: DFDL URN specification
by Mike Beckerle 20 Nov '12

20 Nov '12

I'd like to add a sub-space for implementations to use for their implementation-specific things. Nice thing about a URN is that it can specify a structured scheme for this: I suggest "imp" as a sub-specifier for implementation-specific URNs. E.g., urn:ogf:dfdl:2011:imp:... What goes after imp is an identifier of an implementation. E.g., - urn:ogf:dfdl:2011:imp:opensource.ncsa.illinois.edu:... - urn:ogf:dfdl:2011:imp:ibm.com:... and beneath that is anything that implementation needs/wants. E.g., a namespace for extension properties that are non-standard, a version ID for sub-versions of that implementation, etc. etc. This allows an implementation to define extension properties, and have other implementations give warnings that those are detected, but not understood. ...mikeb On Mon, Oct 29, 2012 at 11:37 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Using the OGF's URN registration document [GFD.191] as a reference, I > suggest: > *Type* *URL* *Proposed URN* common root urn:ogf:dfdl xmlns attribute > http://www.ogf.org/dfdl/dfdl-1.0/ urn:ogf:dfdl:2011 source attribute > http://www.ogf.org/dfdl/ urn:ogf:dfdl > > [GFD.191] http://www.ogf.org/documents/GFD.191.pdf > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Suman Kalia <kalia(a)ca.ibm.com> > To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, > Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org > Date: 25/10/2012 17:15 > Subject: Re: [DFDL-WG] new action item needed: DFDL URN > specification > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > I tend to agree that specifying urn for namespaces is better choice and > URL should be used for schema locations; however convention of specifying > URL for namespaces is long established some implementations use the > namespace URL to return the actual schema. When I try to access the > namespace URL for XML schema , it gives me reference to the document but > not schema.. > > > > Suman Kalia > IBM Canada Lab > WMB Toolkit Architect and Development Lead > Tel: 905-413-3923 T/L 313-3923 > Email: kalia(a)ca.ibm.com > > For info on Message broker * > ** > http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… > *<http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h…> > > > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org, > Date: 10/25/2012 08:18 AM > Subject: Re: [DFDL-WG] new action item needed: DFDL URN > specification > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > I wanted to add to this discussion the w3c blog page that highlights the > problem of the http-based naming scheme. There are many articles about > this, this is just one of them.* > ** > **http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic/*<http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic/> > > On Wed, Oct 24, 2012 at 10:35 AM, Mike Beckerle <*mbeckerle.dfdl(a)gmail.com > * <mbeckerle.dfdl(a)gmail.com>> wrote: > > Introduction: > > Using URLs as identifiers has caused no end of problems. E.g., in DFDL we > have *http://www.ogf.org/dfdl/dfdl-1.0/*<http://www.ogf.org/dfdl/dfdl-1.0/>as an identifier. W3C has > *http://www.w3.org/2001/XMLSchema* <http://www.w3.org/2001/XMLSchema> and > others. > > W3C has badly regretted establishing this convention, as they have farms > of servers that do nothing but quickly return 404 errors to save > network-aware applications the network-timeout delay that would otherwise > occur. > > There are parties interested in exploiting DFDL who want DFDL schemas to > NOT contain network URLs because it simply creates a concern about network > access whenever a DFDL schema is inspected/used. > > Pre-Proposal: > > The new way to do this is with URNs which would look roughly like this: > urn:ogf:dfdl:standard:dfdl-1.0. The whole point is that some other > mechanism is used to establish correspondences between these and any > resources in file systems, networks, or built-in to implementations. One > such mechanism is called XML Catalog. > > The point is that it is a name in a managed namespace which cannot be > confused with a network protocol URL. > > OGF is already establishing urn:ogf, and an ogf subgroup has already > proposed urn:ogf:network for network resources. DFDL schemas aren't network > resources so we don't want to be a substructure underneath network. > > Some other mechanism is used to establish correspondences between these > and any resources in file systems, networks, or built-in to > implementations. One such mechanism is called XML Catalog. > > Summary: > > An action item should be to specify DFDL urn, submit to OGF as a proposed > namespace, and then produce errata/spec changes to specify its use. > > This requires a small design activity to specify a scheme for the > sub-structure of the DFDL URNs (i.e., scheme for the stuff after > urn:ogf:dfdl:...) where we want standard identifiers for versions of the > standard, but we probably also want a few other things (e.g., I would like > a space for implementations to identify themselves, i.e., an > implementation-specific sub-area within our URNs.) > > Our existing URLs can be compatible (deprecated) practice vs the preferred > URNs. > > > > > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: *781-330-0412* <781-330-0412> > > > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > *https://www.ogf.org/mailman/listinfo/dfdl-wg*<https://www.ogf.org/mailman/listinfo/dfdl-wg> > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

Action 186: Re: DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance)
by Mike Beckerle 20 Nov '12

20 Nov '12

(Missed the CC of the whole WG. Here it is.) On Tue, Nov 20, 2012 at 6:49 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Hi Mike > > I have incorporated all this into Errata v011. A couple of comments: > > - You use 'parse error' in a few places where correct term is 'processing > error' > I try to say parse error in a parsing specific context, and processing error when something applies both parsing/unparsing, but obviously I blow it sometimes. Doesn't the spec say a parse error is a parse-specific processing error? > - I think your new section 7.10 should be in section 9, as it is talking > about order of execution and consequences of failure which are introduced > in section 9. > Sounds fine to me. I just picked one possible logical spot. > > - Regarding "*It is also a processing error to read any variables defined > by any dfdl:newVariableInstance statement evaluations that failed, or to > read any variables set by any dfdl:setVariable statements that failed.* ", > can you give a scenario where the processor can get into this situation? > > Sure. I was just having to implement this the other day, which is where this notion came from. It certainly requires scrutiny. Imagine a big complex sequence type. There's a discriminator, but it involves a complex expression, perhaps more than one field is involved in the calculation (e.g., if either this or that, then we're on a B-type record) however, the structure has substantial parse complexity beyond just identifying the two fields. Real work to do to parse it. The discriminator expression could be expressed in terms of variables just because it is a complex calculation. I think this will be common in XPath since it has no native local variables in the expression language people will use DFDL variables just to avoid having to repeat large complex XPath sub-expressions. That's ok, because if the discriminator calculation is to succeed, then those setVariable expressions would also have to succeed just as if the discriminator were expressed as a bigger more complex nested expression without variables. But, there could also be OTHER setVariable statements in the mix, unrelated to evaluating the discriminator. Those might fail if the whole parse failed. The parts of the structure they depend on might not have been successfully parsed at all. We don't want to require implementations to know which setVariable statements are needed for the discriminator and which are not, so we have this simpler strategy where we can attempt them all. If one fails, we save the error. Then when we get to evaluating the discriminator, it can still succeed, i.e., evaluate to true and resolve the PoU, so long as it doesn't need any of the failed setVariables. Then we fail with the saved error from whichever setVariable statement failed. ...mikeb

1 0

First draft of DFDL Errata v011
by Steve Hanson 20 Nov '12

20 Nov '12

Please find attached the first draft of Errata v011. This incorporates all DFDL spec errata raised since v010 (June 22 2012) until the last DFDL WG call (November 13 2012). Change tracking: I accepted all changes from v010 so the changes you see are all due to v011. I would appreciate a review for correctness and omissions. Please note that some of the new errata have been best handled by updating existing related errata, so please read from start. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 186: Re: DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance)
by Mike Beckerle 20 Nov '12

20 Nov '12

I had the action to create the final errata wording for this item (Action 186). Last action issue was to change wording to insure it was all dealing with combined annotations uniformly, and change "annotated construct" to "schema component". I've done that, but also reorganized this as errata on specific sections of the spec consistent with the rest of our errata, and turned the 'ordering' section into numbered lists for clarity. ------------------------------------------------------- * Errata section 6.2. Clarification:* At any single annotation point of the schema, there can be only one format annotation (dfdl:format, dfdl:element, dfdl:sequence, dfdl:choice, dfdl:group, dfdl:simpleType). *Errata section 3. Glossary entries* *Glossary*: Annotation point: A location within a DFDL schema where DFDL annotation elements are allowed to appear. Note that annotation point is defined in section 6.2, but it belongs also in the glossary. * Glossary*: DFDL Statement annotations, or just DFDL Statements, are the annotation elements dfdl:assert, dfdl:discriminator, dfdl:setVariable, and dfdl:newVariableInstance. *Glossary*: DFDL Defining Annotations are the annotation elements dfdl:defineFormat, dfdl:defineVariable, and dfdl:defineEscapeScheme *Glossary*: DFDL Format Annotations are the annotation elements dfdl:format, dfdl:element, dfdl:simpleType, dfdl:group, dfdl:sequence, and dfdl:choice. *Glossary*: Remove existing definition for "Format Annotations" *Glossary*: Physical layer - revised to: "Physical Layer - A DFDL Schema adds DFDL annotations onto an XSDL language schema. The annotations describe the physical representation or physical layer of the data." * Glossary*: *Combined annotations*: When DFDL annotations appear on a group reference and the sequence or choice of the referenced global group, or appear among an element reference, an element declaration, and its type definition, then they are combined together and the resulting set of annotations is referred to as the *combined annotations* for the schema component.* Errata Section 7.3.1: *Remove "DFDL asserts can be placed on components within a DFDL model." as this is made more specifically subsequently. Replace "More than one dfdl:assert may be used at an annotation point. The dfdl:asserts will be evaluated in the order defined in the schema." with "If the combined annotations for a schema component contain multiple dfdl:assert statements, then those with testKind='pattern' are executed before those with testKind='expression' (the default). However, within each group the order of execution among them is not specified. Schema authors can insert sequences to control the timing of evaluation of statements more precisely. See Section REF: *Evaluation Order for Statement Annotations.*" Property testPattern: Add this to the description: Note that the pattern is used to match against the entire representation of the component; hence, the framing (including initiators) is visible to the pattern. It is a schema definition error if dfdl:alignment is not 1 . It is a schema definition error if dfdl:leadingSkip is not 0. It is a schema definition error if dfdl:encoding is not defined. * ** Errata Section 7.3.1, and 7.4.1: Assert and Discriminator Placement* Correct description to be consistent with: dfdl:assert and dfdl:discriminator can be placed as annotations on sequence, choice, group references, local and global element declarations, element references, and simple type definitions. *Errata Section 7.4.1:* Replace: "Any one annotation point can contain only a single dfd:discriminator or one or more dfdl:asserts, but not both. It is a schema definition error otherwise." with: "The combined annotations for a schema component can contain only a single dfd:discriminator or one or more dfdl:asserts, but not both. It is a schema definition error otherwise." Property testPattern: Add this to the description: Note that the pattern is used to match against the entire representation of the component; hence, the framing (including initiators) is visible to the pattern. It is a schema definition error if dfdl:alignment is not 1 . It is a schema definition error if dfdl:leadingSkip is not 0. It is a schema definition error if dfdl:encoding is not defined. *Errata Section 7.8 newVariableInstance* dfdl:newVariableInstance can be placed as an annotation on sequence, choice, and group references. Replace: "It is a schema definition error to have more than one newVariableInstances for the same variable at any given point in the document." with "The combined annotations for any schema component can contain multiple dfdl:newVariableInstance annotations, but they must each refer to a different variable. It is a schema definition error otherwise." *Errata Section 7.9: setVariable * Correct language to be consistent with: "dfdl:setVariable may be placed as an annotation on sequence, choice, group references, local and global element declarations for elements of simple type, element references to elements of simple type, and simple type definitions. " Replace "It is a schema definition error to have more than one dfdl:setVariable for the same variable at any given point in the document." with "The combined annotations for any schema component can contain multiple dfdl:setVariable annotations, but they must each refer to a different variable. It is a schema definition error otherwise." * Errata: New Section 7.10: Evaluation Order for Statement Annotations *Of the combined annotations for a schema component, some are statement annotations and the order of their evaluation relative to the actual processing of the schema component itself (parsing or unparsing per its format annotations) is as given in the ordered lists below. Implementations are free to optimize by recognizing and executing discriminators or assertions with testKind='expression' earlier so long as the resulting behavior is consistent with what results from the this description: For elements: 1. dfdl:discriminator with testKind='pattern' (parsing only) 2. dfdl:assert(s) with testKind='pattern' (parsing only) 3. Processing of the element schema component itself (as per format annotations) 4. dfdl:setVariable(s) - See note below. 5. dfdl:discriminator with testKind='expression' (parsing only) - Evaluated even on failure of steps 3, or 4. See note below. 6. dfdl:assert(s) with testKind='expression' (parsing only) For sequences and choices: 1. dfdl:discriminator with testKind='pattern' (parsing only) 2. dfdl:assert(s) with testKind='pattern' (parsing only) 3. dfdl:newVariableInstance(s) 4. dfdl:setVariable(s) 5. dfdl:discriminator with testKind='expression' (parsing only) - Evaluated even on failure of steps 3 or 4. See note below. 6. dfdl:assert(s) with testKind='expression' (parsing only) 7. Processing of the sequence/choice component itself (as per format annotations) * Note on Discriminators with testKind='expression' *When parsing, an attempt to evaluate a discriminator must be made even if preceding statements or the parse of the schema component ended in a parse error. This is because a discriminator's expression could evaluate to true thereby resolving a point of uncertainty even if the complete parsing of the construct ultimately caused a parse error. Such discriminator evaluation has access to the DFDL Infoset of the attempted parse as it existed immediately before detecting the parse failure. Attempts to reference parts of the DFDL Infoset that do not exist are parse errors. It is also a parse error to read any variables defined by any dfdl:newVariableInstance statement evaluations that failed, or to read any variables set by any dfdl:setVariable statements that failed. * Note on the dfdl:setVariable Statement for Elements* The combined dfdl:setVariable statements for an element schema com ponent (combined from an element reference, element declaration, and base and derived simpleType definitions) are executed after the parsing of the element. This implies that these variables are set after the evaluation of expressions corresponding to any computed DFDL format properties for the same schema component, and so the variables may not be referenced from expressions that compute DFDL properties. That is, if an expression is used to provide the value of a format property (such as dfdl:terminator, or dfdl:byteOrder), the evaluation of that format property expression occurs before any dfdl:setVariable annotation from the combined annotations for that schema component are executed; hence, the expression providing the value of the format property may not reference the variable. Schema authors can insert sequences to provide more precise control over when variables are set.

2 5

Action 189 - revise precedence order section
by Mike Beckerle 20 Nov '12

20 Nov '12

I suggested revising the precedence order section. So, I spent a few hours trying to reinvent the precedence order section along the lines of the grammar version 007. I failed. I kept ending up with a structure not too different from what exists already in the spec. The thought process goes something like this: - Create a tree corresponding to the grammar - Annotate the leaves with every relevant property. - Annotate the interior nodes of the tree with the properties needed just to get to that node of the tree, or that appear beneath that node down multiple paths. - And finally, don't repeat things. The problem: the grammar productions and terminals don't provide enough ways to share to avoid lots of duplication. The grammar leaves out exactly the details needed - representation (the grammar regions often have both a textual and a binary personality that are quite different), and length (the grammar regions are mostly silent about lengthKind). You end up wanting to create a big class diagram and have each node of the tree associated with multiple classes. It's a mixin-style multiple-inheritance scheme. It's not very interesting because so many things inherit so many properties (because they have both textual and binary forms, and work with different length kinds.) If you approximate this by what you can create in a nested hierarchy, you have to repeat some properties, but you end up being not very different from the breakdown of what is called out in the precedence section of the document as written currently..... voila... full circle. I went round and round this enough times to convince myself that whatever I was producing wasn't adding value over what is already there in the precedence section as written today. I also convinced myself that the precedence section as written, does add substantial value exactly because it is NOT organized along the lines of the grammar, but orthogonally to it. I suggest we close out Action 189. I think it will not pan out. There may be other adjustments to be made to the precedence section, but the rewrite along the lines of the grammar is not an idea that will pan out. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Re: [DFDL-WG] Issue 140 spinoff - proposed revised grammar
by Mike Beckerle 19 Nov '12

19 Nov '12

Attached revised grammar 005 - proposed as "to become an errata". On Mon, Jul 30, 2012 at 6:29 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > A couple of further revisions. > > > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Steve Hanson/UK/IBM@IBMGB > Cc: Tim Kimber/UK/IBM@IBMGB > Date: 25/07/2012 16:49 > Subject: Re: Issue 140 spinoff - proposed revised grammar > ------------------------------ > > > > Revised per call today. For your consideration. > > ...mikeb > > On Wed, Jul 11, 2012 at 7:39 AM, Steve Hanson <*smh(a)uk.ibm.com*<smh(a)uk.ibm.com>> > wrote: > I've made some updates which I think solves the complex empty issue, ties > the grammar more closely to the action 140 rep definitions, and addresses > Tim's content v value distinction. > > Let me know what you think. > > > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:*+44-1962-815848* <%2B44-1962-815848> > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com> > > > To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB > Date: 10/07/2012 16:25 > Subject: Re: Issue 140 spinoff - proposed revised grammar > ------------------------------ > > > > > Accept-all-changes on that document before you try to read it. Or use the > attached. > > On Tue, Jul 10, 2012 at 11:19 AM, Mike Beckerle <*mbeckerle.dfdl(a)gmail.com > * <mbeckerle.dfdl(a)gmail.com>> wrote: > I took a crack at revising the grammar to something more accurate, more > useful. It's bigger, but not 3 times as big. > > I like it, but then I'm an implementor, not a user of this spec trying to > devise a schema. Our audience is both however. > > I am curious what you think. > > One thing it cannot capture is the complexity of complex empty's, i.e., > the fact that a complex empty requires one to sometimes descend recursively > first, and then check for empty. From the grammar you would think you can > always check for this first. > > Fact is, you have to descend recursively in order to pop back up, and have > the representation be found to match what I call EmptyElement. But the > grammar can't show that. (well, maybe it could... I'll think about that > further.) > > ...mike > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: *781-330-0412* <781-330-0412> > > > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: *781-330-0412* <781-330-0412> > [attachment "grammar-revision.docx" deleted by Steve Hanson/UK/IBM] > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > [attachment "grammar-revision-003.doc" deleted by Steve Hanson/UK/IBM] > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 5

Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger
by Steve Hanson 19 Nov '12

19 Nov '12

Agreed on DFDL WG call 4th Sept to continue to support twos-complement decimal, integer or nonNegativeInteger, but to make the maximum implementation defined. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 15/08/2012 13:10 Subject: Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Mike Having thought about this since our call yesterday, I am uncomfortable with the proposed narrowing. It means DFDL can not be guaranteed to handle the two's complement result of Java BigInteger.toByteArray(). I don't have a concrete use case of two's complement data > 8 bytes but I am sure that someone in the scientific community will need it at some point. Hence why it is allowed in the spec today. If we do allow it, then we have to decide what happens during unparsing and lengthKind is 'prefixed'. The equivalent of Table 16 for lengthKind 'pattern' is needed for lengthKind 'prefixed', as we discussed. That would imply that the unparsing behaviour would be 'Minimum number of bytes to represent significant digits and sign'. In other words, a two's complement binary integer is created that is just long enough to represent the infoset value. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org Date: 14/08/2012 19:33 Subject: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Sent by: dfdl-wg-bounces(a)ogf.org The spec is not clear on whether decimal, integer, and nonNegativeInteger support representation='binary' with binaryNumberRep='binary' (aka twos-complement). Since these types do not have any size limit, such a capability allows arbitrarily large twos-complement integers, or perhaps it would be an implementation defined limit. I believe the right choice is that it should be disallowed. That is, only binaryNumberRep of 'bcd' or 'packed' should be allowed for these types. These types have always been about decimal-style representations to me. This keeps twos-complement only for the types long, int, short, byte and their unsigned counterparts. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Agenda 2012-11-20
by Steve Hanson 19 Nov '12

19 Nov '12

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12967?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

discriminators near PoU or not
by Mike Beckerle 19 Nov '12

19 Nov '12

I thought this situation was worth discussing. I have a bunch of messages. They all work like this: <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> There are hundreds of those. These all are used in this way: <element name="topLevel"> <complexType> <sequence> <element name="message" minOccurs="1" maxOccurs="unbounded"> <complexType> <choice> <element ref="messageName1"/> <element ref="messageName2"/> .... </choice> </complexType> </element> </sequence> </complexType> </element> So, there are two points of uncertainty here. The choice, and the unbounded array surrounding it. For each message, it is not until the sequence carrying the initiator, which is down inside the message structure, that we know for sure we've got this kind of message. So currently each message element is designed to be used ONLY in the above context of two surrounding points of uncertainty, like so (same message format as above but with TWO discriminators added). <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> <annotation><appinfo...>  <dfdl:discriminator>{ true() }</dfdl:discriminator>  <dfdl:discriminator>{ true() }</dfdl:discriminator> </appinfo></annotation> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> I'd love to have a better solution to this issue. But in the absence of one this works and achieves what I want which is that once it hits the initiator, we discriminate both which message we have, and we discriminate that we in fact have a message. Interesting that I'd really prefer to discriminate them in the opposite order of their nesting. I could discriminate the array the the very start of the message. But I have no way to do that because the discriminators apply in inward-out nesting order, and the nearest enclosing PoU is the choice. So I have to discriminate that one first, and I can't discriminate that one until I see the initiator, which is later into the message. Possible fixes/improvements: allow a label/id on each PoU, and allow referencing that label from a discriminator to say exactly which PoU you are discriminating. This makes the outward reference implied by a discriminator explicit. You still have the context issue, but it's not implied anymore, it is explicit. (This makes the problem like expressions with "../../.." that reach up and out of a construct. They are reaching up and out, but at least you can see directly that they are doing so.) -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1