March 2005 - dfdl-wg - lists.ogf.org

RE: [dfdl-wg] How to handle multi-dimensional arrays - version 2
by mike.beckerle＠ascentialsoftware.com 03 Mar '05

03 Mar '05

I'd like to propose that the kind of thing you are talking about below Jim, where you get to decide what the presentation of the data is, is always done using layers. The point is that this lets us pick a base-level DFDL/XSD schema that is the one implied by the physical representation of the data, and use that as the starting point for layer transformations. Martin and I discussed this privately last fall at some point as the "XML all the way down" approach. I.e., at each layer, the layer above gets to think of the next layer down as if it were XML described by some XSD. The trick is the bottom-most-layer. Here there is no distinction between the shape of the actual physical representation and the shapes described in the XSD. If the physical rep has certain nested hierarchy, then the base-level XSD describes exactly that hierarchy and not any sort of mapping of it to something different in shape. This lowest level one-to-one mapping is how we get the induction of layer transformations started, and it allows for the first such transformation to be no different conceptually from any higher level. All transformations appear to be XML to XML conceptually, rather than the bottom one being something-else to XML, followed by all higher-level XML to XML ones. Now, the alternative is to have the lowest-level model be something different from XSD, and I'm sure this could be made workable too, but I thought we were all reconciled that any such model would be so similar to XSD, having parent-child hierarchy, choice, types, names, dimensions, etc. that making it different from XSD would just be confusing. If you're with me so far, then the only nub was that XSD doesn't directly model multi-dimensional arrays. Which brings us to the topic of the day. So with that I've been forging ahead on the path of worrying only about the lowest-layer schema where it is intended to correspond directly to the structure of the data, and there I'd like to suggest that multi-dimensional arrays have a base-level model in DFDL that looks like the one I proposed first over a week ago, with attributes and all. E.g., like this: <a x="5" y="7">4.32</a> <a x="5" y="8">2.212</a> This is a suitable representation for doing layered transformation into whatever other form you'd like, as it lets xpath-like expressions retrieve individual array elements by index positions. Of course if you actually convert your data into XML you'd pay the cost of these attributes in the text size, but if you're using a DFDL API-based system, then the implementation would be free to respond to requests for the "..../a/@x" attribute which return 5 (in the above example) without actually using any storage for that attribute. > -----Original Message----- > From: Myers, James D [mailto:jim.myers@pnl.gov] > Sent: Thursday, March 03, 2005 10:38 AM > To: dfdl-wg(a)gridforum.org > Subject: RE: [dfdl-wg] How to handle multi-dimensional arrays > - version 2 > > > > > > > Here's a slightly different formulation of the > multi-dimension stuff. > > > > 1) no longer dictates the XSD for representing the array. > > This cuts both ways since you no longer really have an XSD > model for > > multi-dimensional arrays. That is. It is up to the author > of the DFDL > > Schema to insure the needed information about the array > (coordinates > > of each element) make it in to the logical model in a useful way. > > I didn't realize we were proposing to extend the XML schema > to have a multidimensional array type, versus providing a way > for DFDL to read and internally represent a multidimensional > array. The latter seems descriptive and the former prescriptive. > We have a different sense of descriptive and prescriptive. To me descriptive does NOT mean you get to choose and populate an arbitrary XML schema from data of any shape/form. To me "descriptive" means you tell me what your data looks like. But the schema you use to tell me that format is structured directly as a function of what the data format is like. That is, the Schema you provide must be written in a way which informs me (I'm the DFDL processor here) about what the data looks like. The schema is highly constrained in its shape/form by what the data's representation is. One of the members of my team describes this as "it's an input schema, not an output schema." It's a little confusing because DFDL will "prescribe" the schema, but does not "prescribe" the format of the data described by that schema. Rather it uses the "prescribed schema" to "describe" the data format. You do have freedom in what your data format actually is, you don't have freedom to choose what the description of it looks like. I think neither prescriptive nor descriptive data format says anything about the data transformation problem, which is is "how to I get data into a specific form". That's just plain different, and I claim it is the same whether the data started out in XML or in a file described by DFDL. Given data with a certain logical structure transform it into a different logical structure. Use Xquery or XSLT or a variety of non-XML oriented technologies to do so. What makes all this confusing for DFDL is that we have some representations that are complex enough to need layered multi-step descriptions, and once you have that, there's no stopping you from using it to do all sorts of transformation from one format to another. So it feels like you can have your cake and eat it too, which is to say you can pick your XML Schema and populate it from quite differently structured data. And that is probably true, but at the bottom level of the stack of layers you have to have a vocabulary and model for directly describing the structure of the data so as to get the whole ball rolling. And at this bottom layer, the needs of describing the data format completely dictate what the schema is like. Bringing this back to multidimensional arrays, it seems we have this choice: 1) add some new construct to XSD to support base-level description of multi-dimensional arrays 2) choose constructs already present in XSD and use them in a cannonical or "base" way to describe multi-dimensional arrays Currently I'm advocating (2), since there is enough expressive power there. XML/XSD basically can provide an arbitrary map from element names and attribute names and values onto values, and a multi-dimensional array is a subset of this kind of mapping functionality. So we define a canonical way of using these constructs to describe MD arrays and we're done. > > > > 2) I added in the complexity of calculating the array size, > actually > > the lower and upper bounds of each dimension, dynamically based on > > data. This makes the example more real. > > > > This still works out pretty well. I'm still pondering > whether I like > > this better or not. I'm thinking about perhaps some sort of pseudo > > attributes which are guaranteed to be put into XML if you actually > > render to XML, but where a DFDL API-based implementation can choose > > not to realize them. > > I think this example removes some of the prescriptive nature > of the first one, but I'd like to be able to format my array > however I want, e.g. as > > <row><elem>3</elem><elem>2</elem></row> > <row><elem>5</elem><elem>6</elem></row> > ... > > Or even > > <states><state>Alabama</state><state>Alaska</state></states> > <population><pop>34.2</pop><pop>10.6</pop></population> > .... > > (an array containing state names, population and other data, > perhaps serialized in the file as all info for each state together). > > If DFDL could separate the reading of such an array from how > it is output in the schema, I could do any of this. Having > multiple layers is a start - DFDL reads the array in to > something that is addressable along the lines Mike proposes > and then the contents of that layer are referenced via xpath > to provide values in some structure I define in XSD. The only > piece missing (I think) is that we haven't yet defined how to > access iterators, i.e. if I have an element <elem minoccurs="1" > maxoccurs = "5"> , how can I say that element n (n = 1...5) > has dfdl:runtimevalue <a x="n" y="1">, which would put just > the first column of a into the element sequence. If, in > Mike's example, I could define the x and y dimensions > independent of an array-reading context, just so I can use > them in value references for dfdl:runtimevalue elements, I > think we'd be all set. > > This type of capability would allow all sorts of useful > things - including the array to set of vectors conversions > outlined here as well as subsampling, expansion/contraction > of sprase arrays (where the array is stored as a sequence of > x,y, value triples for only nonzero elements), etc. > > > One other minor point - if the order of x and y in the DFDL > file is important (as it is in the example), do we need a > <dfdl:array storageOrder="firstDimensionChangesFirst"> > option? OR can we just list y first and then x? > > Jim > >

1 0

FYI: Oracle article on a similar approach
by Myers, James D 03 Mar '05

03 Mar '05

http://www.sys-con.com/story/?storyid=48039 "Enterprise applications such as banking, healthcare, and so on still use flat files to import/export data between applications. Flat files contain machine-readable data that is typically encoded in printable characters. There is a growing need for these applications to interact with XML-aware applications and Web services, and to satisfy this need these applications must convert flat file data to an XML format..." Jim

1 0

Array storage order attribute - was RE: [dfdl-wg] How to handle m ulti-dimensional arrays - version 2
by Mike Beckerle 03 Mar '05

03 Mar '05

First to address this tiny issue: > > One other minor point - if the order of x and y in the DFDL > file is important (as it is in the example), do we need a > <dfdl:array storageOrder="firstDimensionChangesFirst"> > option? OR can we just list y first and then x? > I thought about using listing order. I decided that people want to list the dimensions in the order they think of them. In this case x first, then y, irrespective of what the storage order is. Then express that representational tweak separately. We could make it a default that lastDimensionChangesFirst then you'd only need to express this attribute if you've listed the dimensions backward, or you've got column-major data. I'm opposed to most default settings for rep properties. I'm with the XML philosophy of being explicit about everything every time. But in this case it seems ok. ...mikeb

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays - version 2
by Myers, James D 03 Mar '05

03 Mar '05

> > > Here's a slightly different formulation of the multi-dimension stuff. > > 1) no longer dictates the XSD for representing the array. > This cuts both ways since you no longer really have an XSD > model for multi-dimensional arrays. That is. It is up to the > author of the DFDL Schema to insure the needed information > about the array (coordinates of each element) make it in to > the logical model in a useful way. I didn't realize we were proposing to extend the XML schema to have a multidimensional array type, versus providing a way for DFDL to read and internally represent a multidimensional array. The latter seems descriptive and the former prescriptive. > > 2) I added in the complexity of calculating the array size, > actually the lower and upper bounds of each dimension, > dynamically based on data. This makes the example more real. > > This still works out pretty well. I'm still pondering whether > I like this better or not. I'm thinking about perhaps some > sort of pseudo attributes which are guaranteed to be put into > XML if you actually render to XML, but where a DFDL API-based > implementation can choose not to realize them. I think this example removes some of the prescriptive nature of the first one, but I'd like to be able to format my array however I want, e.g. as <row><elem>3</elem><elem>2</elem></row> <row><elem>5</elem><elem>6</elem></row> ... Or even <states><state>Alabama</state><state>Alaska</state></states> <population><pop>34.2</pop><pop>10.6</pop></population> .... (an array containing state names, population and other data, perhaps serialized in the file as all info for each state together). If DFDL could separate the reading of such an array from how it is output in the schema, I could do any of this. Having multiple layers is a start - DFDL reads the array in to something that is addressable along the lines Mike proposes and then the contents of that layer are referenced via xpath to provide values in some structure I define in XSD. The only piece missing (I think) is that we haven't yet defined how to access iterators, i.e. if I have an element <elem minoccurs="1" maxoccurs = "5"> , how can I say that element n (n = 1...5) has dfdl:runtimevalue <a x="n" y="1">, which would put just the first column of a into the element sequence. If, in Mike's example, I could define the x and y dimensions independent of an array-reading context, just so I can use them in value references for dfdl:runtimevalue elements, I think we'd be all set. This type of capability would allow all sorts of useful things - including the array to set of vectors conversions outlined here as well as subsampling, expansion/contraction of sprase arrays (where the array is stored as a sequence of x,y, value triples for only nonzero elements), etc. One other minor point - if the order of x and y in the DFDL file is important (as it is in the example), do we need a <dfdl:array storageOrder="firstDimensionChangesFirst"> option? OR can we just list y first and then x? Jim

1 0

How to handle multi-dimensional arrays - version 2
by mike.beckerle＠ascentialsoftware.com 03 Mar '05

03 Mar '05

Here's a slightly different formulation of the multi-dimension stuff. 1) no longer dictates the XSD for representing the array. This cuts both ways since you no longer really have an XSD model for multi-dimensional arrays. That is. It is up to the author of the DFDL Schema to insure the needed information about the array (coordinates of each element) make it in to the logical model in a useful way. 2) I added in the complexity of calculating the array size, actually the lower and upper bounds of each dimension, dynamically based on data. This makes the example more real. This still works out pretty well. I'm still pondering whether I like this better or not. I'm thinking about perhaps some sort of pseudo attributes which are guaranteed to be put into XML if you actually render to XML, but where a DFDL API-based implementation can choose not to realize them. Your comments appreciated. ...mikeb

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays
by mike.beckerle＠ascentialsoftware.com 02 Mar '05

02 Mar '05

I think I get your points Jim. Responding stream of consciousness interlineated below. Later points build on earlier ones, so read it all before typing up any reactions to earlier ones. ...mike > > This approach seems to dictate how the user will represent > the array in XML (dictating their schema) rather than just > describing how to pick up the right content. Which I think we > agreed is a bad thing. Your concern seems to be with the requirment that the element have attributes which make the position of the element explicit. I suppose this is only needed if you want to access the array with some sort of multi-dimensional X-path like expression. We chose this attribute mechanism because it seemed the most compact thing we could come up with that was consistent with xpath-like expressions that could select a specific element of the array. We could make it flexible so that you could use sub-elements to hold the positions, not just attributes, or you could even leave them out. If you leave them out then I don't see a way to both tolerate storage-order variations and provide index-based access within the XSD/XML framework. However, if you are not translating into XML, but using an API to access the data where that API is supplied by a DFDL-aware library and has direct support for multi-dimensional arrays, then you certainly do not want lots of storage for a numeric array being taken up representing redundant attributes that just hold the coordinate positions. Hence, these dimension attributes really should be hidden, and not parts (elements or attributes) of the logical data model. > That's not to say this isn't a > reasonable way to represent a multidim array in XML, just > that having DFDL go look at the attributes outside the > annotation and requiring this formatting in the output XML > doesn't seem right. > Point taken. However, can you suggest what might be alternatives for concrete output XML for a multidim array? I'm kind of stuck in the rut of this one currently. Everything else I look at feels awkward in comparison, but this may just be failure of imagination. > In > <dfdl:dataFormat arrayStorageOrder="@y @x"> > > it seems that DFDL only needs the dimension sizes for its own > purposes and we could probably use our referencing mechanism > to get them (i.e. if I read the two dimension ints earlier > and want to reference them for the array sizes) - maybe some > kind of <dfdl:runtimeoccurs> elements for the n dimensions? (BTW: since the earlier mailing, in our prototype this rep property has been renamed to arrayDimensions, since it serves to identify the attributes which provide the dimensions of the element. The order of the elements in the list continues to be used to specify the storage order.) > > To allow the kind of output XML you propose, we probably need > something new to allow you to loop. I don't see why we need a loop. Well not yet anyway. I can clearly see needing to write expressions that need to know the coordinates of the current element's position in the array. I was thinking of that as using xpath-like expressions that would refer to the "dimension attributes". I.e., if I have a 2d array with axes x and y, then ./@x is this element's x-axis position, and ./@y is this element's y-axis position. In a single-dimensional case with unnamed axis, then ./position() is the X-path way of doing this. So far I think the DFDL system can do the iterations, you just have to write the "loop body" which is called over and over again. E.g., this would let you do stuff like combine two conforming arrays of scalars to create an array of tuples. Another important example is that it lets you do stuff like have the null-indicators for an array of nullable values be stored in a separate array of individual flag bits. I'm rather engaged in trying to see how far we can get without putting in any sort of looping or recursion constructs currently since with layering and the ability to write expressions that refer to the "current element's position" or its multidimensional analogue, and to dimension things using the sizes of other things as parameters I think we can do very powerful things quite easily but still declaratively. I'm not philosophically opposed to having more power, I just don't want to put it in until the need is clear. I haven't seen any examples yet where an explicit loop, not an implied loop, is truly needed. > If the only place we need > looping is for multidimensional arrays (and the special case > of one dim), perhaps we can do something similar to what you > propose and essentially have the array mechanism define some > loop variables that can be referenced (a dfdl layer?). I suppose one could use a layer to hide the dimension attributes from the logical model. If the layer was named "rep", then the arrayDimensions property would be "./rep/@y ./rep/@x", and they wouldn't appear in the actual XML logical model of the data. You just wouldn't then have any X-path like way of indexing the resulting logical structure without digging into the rep. It would be up to you whether you want them in the logical structure or not. A layer is perhaps not abstract enough. You don't really want these things to be realized at all, not in a layer nor in the logical model. I'll give this some more thought and there will be some subsequent proposal. Along the lines of your #currentcursorvalue# thing below. The idea is you get an expression that accesses the current position on any dimension of a multi-dimensional array, and it's up to you if you want to populate an attribute subelement or otherwise with it or not. > don't have a full proposal thought out, but imagine defining > an array as a stream that can be referenced using multiple > dimensions rather than a single cursor, and having a > mechanism so that the current value of the cursor(s) are > available to the user. > > So, from the Reference.xsd example, we might want to have an > attribute that shows the x value of the xdata elements > analogous to the multidim > example: > > <xs:element name="xdata" type="xs:float" > maxOccurs="unbounded"> > <xs:annotation> > <xs:appinfo> > > <dfdl:runtimeoccurs>../x</dfdl:runtimeoccurs> > </xs:appinfo> > </xs:annotation> > <xs:attribute name="x"> > <dfdl:runtimevalue = #currentcursorvalue#/> > </xs:attribute> > </xs:element> > > where #currentcursorvalue# is something we have not yet made > available for output (or is this available via xpath - the > position of the current element in a sequence?). Yes in xpath position() for a 1d array fills this role. > This would > change the Reference.xml example output to have elements like > > <xdata x="1">2.78</xdata> > <xdata x="2">3.14</xdata> > > Right. So the user chooses whether to tuck the position into an attribute or not. > So, if I can summarize/rephrase, I think we should keep the > mechanism for single or multidim arrays separate from how the > output is displayed, but I like the idea of making the > current cursor(s) available for use, which I don't think > we've done yet. And having a real multidimension construct > rather than calculating them from a flat cursor is probably a > requirement for scientific use, so some multidim analog of > dfdl:runtimeoccurs is needed. > Ah, runtimeoccurs. My example was fixed length so this didn't come up. I have a dfdl:lengthCalc expression, and also dfdl:storedLength path property in the prototype today. These need to generalize for multidimensional stuff. I'll put that in the next version. ...mikeb

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays
by Myers, James D 02 Mar '05

02 Mar '05

Mike, Some random quick comments - hopefully not to terse or stream-of-conciousness to be incomprehensible: This approach seems to dictate how the user will represent the array in XML (dictating their schema) rather than just describing how to pick up the right content. Which I think we agreed is a bad thing. That's not to say this isn't a reasonable way to represent a multidim array in XML, just that having DFDL go look at the attributes outside the annotation and requiring this formatting in the output XML doesn't seem right. In <dfdl:dataFormat arrayStorageOrder="@y @x"> it seems that DFDL only needs the dimension sizes for its own purposes and we could probably use our referencing mechanism to get them (i.e. if I read the two dimension ints earlier and want to reference them for the array sizes) - maybe some kind of <dfdl:runtimeoccurs> elements for the n dimensions? To allow the kind of output XML you propose, we probably need something new to allow you to loop. If the only place we need looping is for multidimensional arrays (and the special case of one dim), perhaps we can do something similar to what you propose and essentially have the array mechanism define some loop variables that can be referenced (a dfdl layer?). I don't have a full proposal thought out, but imagine defining an array as a stream that can be referenced using multiple dimensions rather than a single cursor, and having a mechanism so that the current value of the cursor(s) are available to the user. So, from the Reference.xsd example, we might want to have an attribute that shows the x value of the xdata elements analogous to the multidim example: <xs:element name="xdata" type="xs:float" maxOccurs="unbounded"> <xs:annotation> <xs:appinfo> <dfdl:runtimeoccurs>../x</dfdl:runtimeoccurs> </xs:appinfo> </xs:annotation> <xs:attribute name="x"> <dfdl:runtimevalue = #currentcursorvalue#/> </xs:attribute> </xs:element> where #currentcursorvalue# is something we have not yet made available for output (or is this available via xpath - the position of the current element in a sequence?). This would change the Reference.xml example output to have elements like <xdata x="1">2.78</xdata> <xdata x="2">3.14</xdata> So, if I can summarize/rephrase, I think we should keep the mechanism for single or multidim arrays separate from how the output is displayed, but I like the idea of making the current cursor(s) available for use, which I don't think we've done yet. And having a real multidimension construct rather than calculating them from a flat cursor is probably a requirement for scientific use, so some multidim analog of dfdl:runtimeoccurs is needed. Jim -----Original Message----- From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of mike.beckerle(a)ascentialsoftware.com Sent: Friday, February 18, 2005 4:22 PM To: dfdl-wg(a)gridforum.org Subject: [dfdl-wg] How to handle multi-dimensional arrays We have come up with an approach to how to represent multi-dimensional arrays within XSD-described XML. The attached test file (.xml) and DFDL Schema (.dfdl.xsd) illustrate the proposed solution. The proposal does not require any changes to XSD, XML or any other special constructs outside of a single dfdl annotation to specify the storage order of the representation. I'm pretty happy with how this works out. We can handle arrays with different storage orders, like fortran style column-major vs. more common row-major, and it dovetails nicely with XPath expressions and the XSD data model. Schema validation can really do something for you, like tell you if you have all the elements of the array (if it's fixed size), and that you don't have multiple elements occupying the same array location. Those interested in multi-dimensional array support please give this some consideration. That said, I'm departing on vacation for a week, so I'll toss this out there for people to look at, but I won't be able to interact with you all on it until I get back. ...mikeb

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays
by Mike Beckerle 02 Mar '05

02 Mar '05

Thanks for the supportive comments Steve, Anyone else? Particularly those of you closer to the scientific-computing community. Would you care to comment on the multi-dimensional arrays proposal that I sent out a bit more than a week ago? ...mikeb > -----Original Message----- > From: Steve Hanson [mailto:smh@uk.ibm.com] > Sent: Wednesday, March 02, 2005 9:23 AM > To: mike.beckerle(a)ascentialsoftware.com > Cc: dfdl-wg(a)gridforum.org; owner-dfdl-wg(a)ggf.org > Subject: Re: [dfdl-wg] How to handle multi-dimensional arrays > > Hi Mike > > I will confess to having not given a great deal of thought to > multi-dim arrays in the past, but what you propose looks to be sound. > > Regards, Steve > > Steve Hanson > WebSphere Business Integration Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 --------------- > > We have come up with an approach to how to represent > multi-dimensional arrays within XSD-described XML. The > attached test file (.xml) and DFDL Schema (.dfdl.xsd) > illustrate the proposed solution. > > The proposal does not require any changes to XSD, XML or any > other special constructs outside of a single dfdl annotation > to specify the storage order of the representation. > > I'm pretty happy with how this works out. We can handle > arrays with different storage orders, like fortran style > column-major vs. more common row-major, and it dovetails > nicely with XPath expressions and the XSD data model. Schema > validation can really do something for you, like tell you if > you have all the elements of the array (if it's fixed size), > and that you don't have multiple elements occupying the same > array location. > > Those interested in multi-dimensional array support please > give this some consideration. > > That said, I'm departing on vacation for a week, so I'll toss > this out there for people to look at, but I won't be able to > interact with you all on it until I get back. > > ...mikeb > > [attachment "test2dArray.xml" deleted by Steve > Hanson/UK/IBM] [attachment "test2dArray.dfdl.xsd" deleted by > Steve Hanson/UK/IBM] >

1 0

RE: [dfdl-wg] alternate syntax for DFDL annotations
by Mike Beckerle 02 Mar '05

02 Mar '05

Sorry for missing our call today, and thanks to everyone for the rapid feedback on the non-native attributes approach. I think there's a consensus here that non-native attributes are going too far, and that if we can just make the appinfo annotations a bit more dense we'll be happy. So , for now, we're not going to do non-native attributes in our DFDL prototype, I'll make a note about the possiblility in an appendix of the doc though. None of the tools I have will validate them, and that's a big drawback. Right now, our prototype handles small examples with annotations that look like this: <xs:element name="foo" type="xs:int"> <xs:annotation> <xs:appinfo source="http://dataformat.org/"> <dfdl:dataFormat byteOrder="bigEndian" charset="UTF-8">  <dfdl:separator>,</dfdl:separator>  </dfdl:dataFormat> </xs:appinfo> </xs:annotation> </xs:element> Note that if you use appinfo annotations then you don't need namespace qualification on the attribute names. We also allow for "extension" declarations in that the dfdl:dataFormat element is the head of a XSD substitution group, So you can also define an extension like ascl:binaryIntegerFormat below: <xs:element name="foo" type="xs:int"> <xs:annotation> <xs:appinfo source="http://dataformat.org/"> <ascl:binaryIntegerFormat byteOrder="bigEndian"/> </xs:appinfo> </xs:annotation> </xs:element> The ascl:binaryIntegerFormat is a narrowing. It doesn't allow any properties to be specified that don't make sense for a binary integer. It's in a different namespace to illustrate that these could be defined by a DFDL user in their DFDL schema, not just by the dfdl implementation itself. The basic dfdl:dataFormat element allows any and all representation properties. Whether we supply standard library elements which are narrowings of it, or let those be defined by end-users is something we can discuss. This all comes for free by just using XSD properly. It adds almost nothing to the implementation. ...mikeb > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of Steve Hanson > Sent: Wednesday, March 02, 2005 1:08 PM > To: Myers, James D > Cc: dfdl-wg(a)gridforum.org; owner-dfdl-wg(a)ggf.org > Subject: RE: [dfdl-wg] alternate syntax for DFDL annotations > > > > > > I can't say I am over keen on this, because > - you will end up with odd rules where some properties can be > attributes and some can't, which will confuse users. > - you have to invent errors and/or overide behaviour when you > get a clash between attribute and annotation > - (minor) it makes the job of any software that instantiates > a dfdl memory model harder > > If we do adopt this then #2 is preferable. > > Regards, Steve > > Steve Hanson > WebSphere Business Integration Brokers, > IBM Hursley, England > Internet: smh(a)uk.ibm.com > Phone (+44)/(0) 1962-815848 > > > > > "Myers, James D" > > <jim.myers(a)pnl.go > > v> > To > Sent by: dfdl-wg(a)gridforum.org > > owner-dfdl-wg@ggf > cc > .org > > > Subject > RE: [dfdl-wg] > alternate syntax for > 02/03/2005 17:35 DFDL annotations > > > > > > > > > > > > > > > > > > > I agree - I think I like #2. I know at one point in the past > we had complaints when we tried to use attributes with > namespaces - it may just have been that parsers weren't > dealing with them correctly early on. > Whatever it was, option 2 would provide a work around as well > as helping us keep a single 'canonical' form for DFDL. > > Jim > > > -----Original Message----- > > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] > On Behalf > > Of Martin Westhead > > Sent: Wednesday, March 02, 2005 12:16 PM > > To: Mike Beckerle > > Cc: dfdl-wg(a)gridforum.org > > Subject: Re: [dfdl-wg] alternate syntax for DFDL annotations > > > > > > Hi Mike, > > > > This is interesting I wasn't aware you could do this. My > concern would > > be if it turned out the schema tools couldn't handle it but I tried > > XML Spy which seems to just ignore the extra annotations. > > > > So I'd be happy to use them. As you point out we will need > both these > > and annotations so we are left with a choice. Either we: > > > > 1. Specify some (most) dfdl additions in this non-native > attribute > > form and use annotations when required or > > > > 2. We specify everything as annotations but allow this form as a > > systematic short hand. > > > > I think I prefer (2). The sort of thing I have in mind is > making the > > annotation syntax look like: > > > > <xs:element name="foo" type="xs:string"> > > <annotation> > > <appinfo> > > <dfdl:attributes dfdl:repType="text" > > dfdl:charset="UTF-8" > > dfdl:repLength="10"/> > > </appinfo> > > <annotation > > </xs:element> > > > > And then this be directly equivalent (and trivially transformed > > into/from) what you had before: > > > > <xs:element name="foo" > > type="xs:string" > > dfdl:repType="text" > > dfdl:charset="UTF-8" > > dfdl:repLength="10"/> > > > > I'm not sure if it matters that much but it seems like it might be > > neater if everything could be represented in the same form. > > > > Also there may be times when you would like to use the > annotation form > > because you have a bock of stuff that could be reused > multiple times. > > Once your annotations have become attributes in the main > schema they > > are more intimately tied to it. > > > > Cheers, > > > > Martin > > > > > > > > Mike Beckerle wrote: > > > Up til now we've considered DFDL annotations only as inside the > > > appinfo context. However, we should consider whether we should use > > non-native > > > attributes as well or > > > > > > as an alternative: E.g., simple DFDL rep properties could also be > > > expressed like this: > > > > > > <xs:element name="foo" > > > > > > type="xs:string" > > > > > > dfdl:repType="text" > > > > > > dfdl:charset="UTF-8" > > > > > > dfdl:repLength="10"/> > > > > > > > > > > > > This has the advantages of compactness, and is a fully > > supported way > > > of > > > extending XML Schema. That is, using non-native attributes is a > > > supported extension idiom. This won't handle things that > > really need the > > > syntactic support of element structure to express their > complexity, > > > e.g., things like specifying text delimiters with quoting and > > > escape-sequence specifications. For those we'll still need > > to open an > > > appinfo annotation up. However, for basic things like > > byteOrder and such > > > it is far more attractive syntactically to use non-native > > attributes > > > than appinfo annotations. > > > > > > > > > Comments? > > > > > > ...mikeb > > > > > > > > > >

1 0

RE: [dfdl-wg] Conference Call Today - notes
by Myers, James D 02 Mar '05

02 Mar '05

Sorry for the late FYI on this: If we have a meeting in Toronto (anywhere outside the US), PNNL staff will need 45-60 days to get clearance for the foreign travel. That would mean making decisions about dates and specific location as soon as possible for us to be involved. (I would hope to have at least one of us and probably two of us there.) Jim > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of mike.beckerle(a)ascentialsoftware.com > Sent: Wednesday, February 16, 2005 12:45 PM > To: dfdl-wg(a)gridforum.org > Subject: [dfdl-wg] Conference Call Today - notes > > > > DFDL-WG Conference call 2005-02-16 > > Attendees: Suman Kalia, Steve Hanson, Susan Malaika - IBM, > Tara Talbot - PNNL, Mike Beckerle - Ascential Software > > IBM offered to host a F2F meeting in mid to late April, or > early May at their Toronto facility. Seems like a good time > frame. Gives Ascential folks time time to get prototype "in > play", and leaves time before the June GGF > (Chicago) to get some things written up. > > MikeB and Suman to schedule time to work on combined model > and list of representation properties: week of Feb 28th. > > Ascential prototype: hoping to have simplest examples working > by end of this week. (No promises). Things like 2 binary > integers. MikeB to post current XSD for dfdl properties and a > few of our test files for the prototype. > > Susan asked about whether implementations of things like BFD > are available. Tara to investigate whether BFD is publicly > available now. > > We discussed when to start recruiting Oracle, Microsoft, and > other participants, since we will need them. We agreed that > our hope would be the June GGF though it really depends on > having an initial document representing some sort of > converged consensus of the current team on current directions. > > > Mike Beckerle > Ascential > >

2 1