- dfdl-wg - lists.ogf.org

Re: Fw: [dfdl-wg] Ambiguous XPaths to hidden elements
by Mike Beckerle 19 Jan '06

19 Jan '06

So we have a quandry here: on one hand we don't want to change the XPath syntax to include a device that would let us be clear that we're navigating a hidden layer on the other hand we don't want to constrain what can be included so that we wouldn't need such a device. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA 01581 voice and FAX 508-599-7148 home/mobile office 508-915-4767 Suman Kalia/Toronto/IBM@IBMCA 01/19/2006 11:52 AM To Mike Beckerle/Worcester/IBM@IBMUS cc dfdl-wg(a)ggf.org, owner-dfdl-wg(a)ggf.org Subject Fw: [dfdl-wg] Ambiguous XPaths to hidden elements As a design point , We should strive not to put limitations on the user defined schemas - it just works out better in the long run. Note the xsd:groups can be nested and they could be many levels deep and this problem is not restricted to groups included from noTarget namespace , it could be from any namespace. As per schema rules, all local elements defined in groups or complex types belong to noTarget namespace unless elementFormDefault is explicitly set to "qualified" at schema level or on the specific element. Detecting such conflicts could be quite expensive particularly when you have very large schemas. Industry standard ACORD messaging schema is a good example it is about 1.5 M and it takes awfully long (hours) to validate it. Putting additional constraints like this will further slow down validation. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 01/19/2006 11:39 AM ----- Mike Beckerle <beckerle(a)us.ibm.com> Sent by: owner-dfdl-wg(a)ggf.org 01/19/2006 10:48 AM To "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> cc dfdl-wg(a)ggf.org, owner-dfdl-wg(a)ggf.org Subject Re: Fw: [dfdl-wg] Ambiguous XPaths to hidden elements One idea that hasn't been advanced yet is ruling out the problematic case. Let me illustrate. Here's the example, modified to have a model group reference which can introduce the name conflict: <xs:element name="root"> <xs:complexType> <xs:sequence> <xs:annotation><xs:appinfo source=?http://dataformat.org? /> <hidden> <xs:element name="repeats" type="xs:integer"/> </hidden> </xs:appinfo></xs:annotation > <xs:element name="testElement" type="xs:integer " minOccurs=?0? maxOccurs=?unbounded? dfdl:repeatCount=?../repeats?> <xs:group ref="groupFromOtherSchemaFile"/>  </xs:complexType> </xs:element> So, what hasn't been suggested yet is this: What if we just say DFDL doesn't allow this? It's an error which must be detected. This DFDL schema is broken because the path "../repeats" cannot be analyzed along with the DFDL schema to return only a single node. I beleive name conflicts like this are what namespace management is for. XSD has truly great namespace managment. You can solve the problem that way. Furthermore, when you define a reusable named group like the definer of the "groupFromOtherSchemaFile" above, and you put it in no target namespace, that's the situation where this conflict can arise. Expecting that your names are never going to conflict with anything in that case is just naive. It's equivalent to having global variables in a C program module and expecting you can never link it to something else that uses the same names. Those name conflicts can occur, and someone has to change the conflicting name. In XSD we can do that by including the group in a schema which puts it into a target namespace so that after that the namespaces can be used to disambiguate. The approach above is consistent with the path "../repeats" still being officialy an "XPath", it just adds the semantic restriction that it must be an XPath that identifies a single node unambiguously, independent of what data is being processed. This is one of these "data independent" notions (what I had previously been calling "static"), as we discussed yesterday. ...mikeb "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> Sent by: owner-dfdl-wg(a)ggf.org 01/19/2006 10:00 AM To dfdl-wg(a)ggf.org cc Subject Re: Fw: [dfdl-wg] Ambiguous XPaths to hidden elements I would want to change XPath only as a last resort. (Any of the options is OK by me, assuming we have to mess with the Xpath at all.) Can we deal with this some other way? Can we document the problematic cases, and suggest best practices that will minimize the problem? On Thursday 19 January 2006 08:45, Suman Kalia wrote: > I fully agree with Steve - let's not invent another XPATH like syntax .. > > Suman Kalia > IBM Toronto Lab > WebSphere Business Integration Application Connectivity Tools > Tel : 905-413-3923 T/L 969-3923 > Fax : 905-413-4850 > Internet ID : kalia(a)ca.ibm.com > ----- Forwarded by Suman Kalia/Toronto/IBM on 01/19/2006 09:43 AM ----- > > Steve Hanson <smh(a)uk.ibm.com> > Sent by: owner-dfdl-wg(a)ggf.org > 01/19/2006 04:43 AM > > To > "Westhead, Martin (Martin)" <westhead(a)avaya.com> > cc > dfdl-wg(a)ggf.org, owner-dfdl-wg(a)ggf.org > Subject > Re: [dfdl-wg] Ambiguous XPaths to hidden elements > > > > > > > As a DFDL parser implementor I do not want modifications to the XPath > syntax. I want to be able to reuse existing XPath implementations. It's > also something else for the user to have to learn. So 2a/b/c are not > attractive. > > Regards, Steve > > Steve Hanson > WebSphere Message Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 > > > > "Westhead, Martin > (Martin)" > <westhead(a)avaya.c To > > om> <dfdl-wg(a)ggf.org> > Sent by: cc > > owner-dfdl-wg@ggf > .org Subject > > [dfdl-wg] Ambiguous XPaths to > hidden elements > 18/01/2006 20:24 > > > > > > > > > > Hi folks, > > This is to try to pick up on the issue identified by Suman in today?s > call. > > The Issue > Consider the following example: > > <xs:element name="root"> > <xs:complexType> > <xs:sequence> > <xs:annotation><xs:appinfo > source=?http://dataformat.org? /> > <hidden> > <xs:element name="repeats" type="xs:integer"/> > </hidden> > > </xs:appinfo></xs:annotation > > <xs:element name="testElement" > type="xs:integer " minOccurs=?0? maxOccurs=?unbounded? > dfdl:repeatCount=?../repeats?> > </xs:complexType> > </xs:element> > > The problem is that the path ?../repeats? can be broken by modifications > to > the logical model due to name clashes on ?repeats? and there are cases > that > can be constructed where this would not be obvious to a user. > > Possible Solutions > Possible fixes to this include: > 1. Disallow XPath references to hidden elements the user is forced > to > place the material into the global context to refer to it. > 2. Provide a special XPath operator to indicate we are referencing > a > hidden element, possibilities include: > a. ?../hidden(repeats)? > b. ?hidden(../repeats)? > c. ?../dfdl:hidden/repeats? > 3. Only allow hidden elements to be present in top level global > complex > types. These can then be included where needed. (This is the > solution > that Suman was pushing but I don?t fully understand it ? > in > particular I don?t see how it resolves the ambiguity issue.) > > > I believe my preference here is 2a or 2b followed by 1. > > Comments/suggestions/opinions? > > Thanks, > > Martin -- --- Robert E. McGrath, Ph.D. National Center for Supercomputing Applications University of Illinois, Urbana-Champaign 1205 West Clark Urbana, Illinois 61801 (217)-333-6549 mcgrath(a)ncsa.uiuc.edu

1 0

RE: [dfdl-wg] Ambiguous XPaths to hidden elements
by Westhead, Martin (Martin) 19 Jan '06

19 Jan '06

Hi All, I believe the XPath standard permits the extension of the language by the addition of new functions. If I'm correct 1 and 2 are not a change to the language but are (I believe) a permitted extension. However I would be happy with 1. It is probably tidier and as Steve points out (later) it seems to be suggestive of a simple implementation. Martin >-----Original Message----- >From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of >Steve Hanson >Sent: Thursday, January 19, 2006 1:44 AM >To: Westhead, Martin (Martin) >Cc: dfdl-wg(a)ggf.org; owner-dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] Ambiguous XPaths to hidden elements > >As a DFDL parser implementor I do not want modifications to the XPath >syntax. I want to be able to reuse existing XPath implementations. It's >also something else for the user to have to learn. So 2a/b/c are not >attractive. > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM Hursley, England >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > "Westhead, Martin > (Martin)" > <westhead(a)avaya.c To > om> <dfdl-wg(a)ggf.org> > Sent by: cc > owner-dfdl-wg@ggf > .org Subject > [dfdl-wg] Ambiguous XPaths to > hidden elements > 18/01/2006 20:24 > > > > > > > > > >Hi folks, > >This is to try to pick up on the issue identified by Suman in today's call. > >The Issue >Consider the following example: > ><xs:element name="root"> ><xs:complexType> > <xs:sequence> > <xs:annotation><xs:appinfo >source="http://dataformat.org" /> > <hidden> ><xs:element name="repeats" type="xs:integer"/> > </hidden> > ></xs:appinfo></xs:annotation > > <xs:element name="testElement" >type="xs:integer " minOccurs="0" maxOccurs="unbounded" > dfdl:repeatCount="../repeats"> > </xs:complexType> ></xs:element> > >The problem is that the path "../repeats" can be broken by modifications to >the logical model due to name clashes on "repeats" and there are cases that >can be constructed where this would not be obvious to a user. > >Possible Solutions >Possible fixes to this include: > 1. Disallow XPath references to hidden elements the user is forced to > place the material into the global context to refer to it. > 2. Provide a special XPath operator to indicate we are referencing a > hidden element, possibilities include: > a. "../hidden(repeats)" > b. "hidden(../repeats)" > c. "../dfdl:hidden/repeats" > 3. Only allow hidden elements to be present in top level global complex > types. These can then be included where needed. (This is the solution > that Suman was pushing but I don't fully understand it - in > particular I don't see how it resolves the ambiguity issue.) > > >I believe my preference here is 2a or 2b followed by 1. > >Comments/suggestions/opinions? > >Thanks, > >Martin

1 0

Fw: [dfdl-wg] Ambiguous XPaths to hidden elements
by Steve Hanson 19 Jan '06

19 Jan '06

I think there is a wider issue here that we need to consider before we can answer this question, and that is what do we plan to say in the DFDL standard about the output we create? Let me explain this further. In IBM we have several different products that could exploit a DFDL parser, but these products all have a different way of exposing the output to the user, normally some kind of DOM-like tree model with an accompanying API. This is almost certainly the case elsewhere too because there is no equivalent standard to DOM for non-XML data. A DFDL parser must therefore provide a SAX-like interface and generate events, leaving it up to the caller to create the tree model (even if it also provided the capability to build an XML DOM model). However both the use of XPath during the parse and hidden elements pose problems for a DFDL parser. - All the XPath implementations I have encountered are bound to the type of tree that is being created - DOM or SDO or WMB for example. With SAX there is no standard tree. - Hidden elements will not by definition appear in a DOM tree. So even in DOM mode a DFDL parser needs some internal mechanism for retaining parsed values that could be referenced by XPath, and its XPath implemetation must therefore use this mechanism. Note it should only use this mechanism for retaining the values of those elements that it knows could be referenced by XPath, ie, it is a sparse mechanism. Otherwise the amount of memory used for large messages becomes unacceptable. An alternative is to force the caller to provide a handler that gets invoked when XPath needs to access the caller's tree (implies disallowing XPath for hidden elements). I don't think a handler approach would be acceptable, even if a DOM alternative was provided. Perhaps it is time to give some thought to the API aspect of DFDL ? (The implication above is that an existing XPath implementation can't be used as is. True, but it is only the tree/model interface that needs changing and this is an internal. I'm still not happy with 2a/b/c below which change the XPath externals. I prefer 1 below.) Regards, Steve Steve Hanson WebSphere Message Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/01/2006 09:53 ----- Steve Hanson/UK/IBM To 19/01/2006 09:43 "Westhead, Martin (Martin)" <westhead(a)avaya.com> cc dfdl-wg(a)ggf.org, owner-dfdl-wg(a)ggf.org Subject Re: [dfdl-wg] Ambiguous XPaths to hidden elements(Document link: Steve Hanson) As a DFDL parser implementor I do not want modifications to the XPath syntax. I want to be able to reuse existing XPath implementations. It's also something else for the user to have to learn. So 2a/b/c are not attractive. Regards, Steve Steve Hanson WebSphere Message Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Westhead, Martin (Martin)" <westhead(a)avaya.c To om> <dfdl-wg(a)ggf.org> Sent by: cc owner-dfdl-wg@ggf .org Subject [dfdl-wg] Ambiguous XPaths to hidden elements 18/01/2006 20:24 Hi folks, This is to try to pick up on the issue identified by Suman in today’s call. The Issue Consider the following example: <xs:element name="root"> <xs:complexType> <xs:sequence> <xs:annotation><xs:appinfo source=”http://dataformat.org” /> <hidden> <xs:element name="repeats" type="xs:integer"/> </hidden> </xs:appinfo></xs:annotation > <xs:element name="testElement" type="xs:integer " minOccurs=”0” maxOccurs=”unbounded” dfdl:repeatCount=”../repeats”> </xs:complexType> </xs:element> The problem is that the path “../repeats” can be broken by modifications to the logical model due to name clashes on “repeats” and there are cases that can be constructed where this would not be obvious to a user. Possible Solutions Possible fixes to this include: 1. Disallow XPath references to hidden elements the user is forced to place the material into the global context to refer to it. 2. Provide a special XPath operator to indicate we are referencing a hidden element, possibilities include: a. “../hidden(repeats)” b. “hidden(../repeats)” c. “../dfdl:hidden/repeats” 3. Only allow hidden elements to be present in top level global complex types. These can then be included where needed. (This is the solution that Suman was pushing but I don’t fully understand it – in particular I don’t see how it resolves the ambiguity issue.) I believe my preference here is 2a or 2b followed by 1. Comments/suggestions/opinions? Thanks, Martin

1 0

Ambiguous XPaths to hidden elements
by Westhead, Martin (Martin) 19 Jan '06

19 Jan '06

Hi folks, This is to try to pick up on the issue identified by Suman in today's call. The Issue Consider the following example: <xs:element name="root"> <xs:complexType> <xs:sequence> <xs:annotation><xs:appinfo source="http://dataformat.org" /> <hidden> <xs:element name="repeats" type="xs:integer"/> </hidden> </xs:appinfo></xs:annotation > <xs:element name="testElement" type="xs:integer " minOccurs="0" maxOccurs="unbounded" dfdl:repeatCount="../repeats"> </xs:complexType> </xs:element> The problem is that the path "../repeats" can be broken by modifications to the logical model due to name clashes on "repeats" and there are cases that can be constructed where this would not be obvious to a user. Possible Solutions Possible fixes to this include: 1. Disallow XPath references to hidden elements the user is forced to place the material into the global context to refer to it. 2. Provide a special XPath operator to indicate we are referencing a hidden element, possibilities include: a. "../hidden(repeats)" b. "hidden(../repeats)" c. "../dfdl:hidden/repeats" 3. Only allow hidden elements to be present in top level global complex types. These can then be included where needed. (This is the solution that Suman was pushing but I don't fully understand it - in particular I don't see how it resolves the ambiguity issue.) I believe my preference here is 2a or 2b followed by 1. Comments/suggestions/opinions? Thanks, Martin

2 1

Notes from Telcon 2006-1-18
by Westhead, Martin (Martin) 18 Jan '06

18 Jan '06

Present: -------- Martin Westhead Mike Beckerle Bob McGrath Steve Hanson Geoff Judd Suman F2F --- It was agreed that we would take forward a plan to have a F2F meeting 1-3 March in Avaya at Milpitas. Final confirmation (hopefully) before next week. Reports ------- Scoping document is progressing. Hopefully we will see something soon. Context document ---------------- - Typo: Zipcode vs numberOfFields - Typo: $ in name attribute (several places) - Issues: If property is not declared with a type first time is it an error Hidden Element document ----------------------- - Issue: Ambiguity of XPath reference (changes to non-hidden portion can break the metadata reference) - Presentation: Clarify text about access to hidden elements

1 0

DFDL Telcon tomorrow 9am PST
by Westhead, Martin (Martin) 17 Jan '06

17 Jan '06

Preliminary Agenda ------------------ 1. Arranging a US F2F in February 2. Reports from trackers 3. Review "Context" document updated from last week - see attached. 3. Review "Hidden Elements" document- see attached. Sorry (again) the documents are going out so late. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference : https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Security Code : 785343; Conference Reference: 158672

1 0

Telcon notes 2006-1-11
by Westhead, Martin (Martin) 11 Jan '06

11 Jan '06

Present: Bob McGrath Tara Talbott Steve Hanson Geoff Judd Martin Westhead Mike Beckerle 1. Face-2-face We are looking at arranging a F2F in San Jose, Chicago or Boston on the weeks of 13th Feb or 28th Feb. 2. Context document This was generally well received. Several issues were raised: * "DFDL Document" should be changed in the document to "DFDL Schema" * Term for describing the input is undecided candidates include: feed, source, stream * "Annotations Schema for DFDL" is used for the Schema document for the DFDL annotations * The term for the code to write data out was discussed candidates include: Unparser, Serializer or Writer * There was some confusion about the fact that the concept of "Context" was just about value store. This should be made more explicit. * Should point out that a "stacked discipline is being followed" i.e. local contexts pop on (and off) * We re-opened the discussion as to whether annotations be allowed on types. (MDW: I believe that the semantics that I am developing here make sense for annotations anywhere in the document) * Simplify the distinction between "properties" and "variables". Everything should be refered to as a "Context value" or (equivalently a "property") distinctions between predefined properties and user-defined properties should be based on name-space * There is a technical issue with establishing namespaces in annotations (and indeed the context) which needs to be addressed.

1 0

Re: [dfdl-wg] DFDL Call tomorrow
by Jim Myers 11 Jan '06

11 Jan '06

FYI: Apologies, but I'll be on a plane tomorrow at conference time. Jim At 07:38 PM 1/10/2006, Westhead, Martin (Martin) wrote: >Preliminary Agenda >------------------ > > 1. Arranging a US F2F in January > 2. Reports from trackers > 3. Review Context document see attached. > >Sorry the document is going out so late. I >messed up when sending it out to the design team for final review. > >Also notice that I have added a URL and codes >for my Web conference facility. It has slide, >application sharing, and whiteboard space. I thought it might be useful. > >Conference call instructions: >Number: 1-800-664-0771 >At the automated answering system, press 0 for the operator. >Ask for the "Daffodil working group" call. > >Web Conference: ><https://webconferencing.avaya.com/>https://webconferencing.avaya.com >Web Conference: Part. Code 785343; WebConf Reference 158672 > > > James D. Myers Associate Director, Collaborative Systems, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

DFDL Call tomorrow
by Westhead, Martin (Martin) 11 Jan '06

11 Jan '06

Preliminary Agenda ------------------ 1. Arranging a US F2F in January 2. Reports from trackers 3. Review "Context" document - see attached. Sorry the document is going out so late. I messed up when sending it out to the design team for final review. Also notice that I have added a URL and codes for my Web conference facility. It has slide, application sharing, and whiteboard space. I thought it might be useful. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

1 0

FYI: xmlecho transformation scripting language
by Mike Beckerle 05 Jan '06

05 Jan '06

Kind of sparse, but conceptually related particularly to our extensibility and layering topics: http://www.xmlecho.org/ ...mikeb

1 0