- dfdl-wg - lists.ogf.org

Telcon tomorrow
by Westhead, Martin (Martin) 10 May '06

10 May '06

Agenda * Array's document circulated by Bob Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

1 0

Meeting notes 2006-4-19
by Westhead, Martin (Martin) 09 May '06

09 May '06

It was a very short meeting this week... Present Bob McGrath Geoff Judd Steve Hanson Martin Westhead Notes - Document expected from Bob and Mike on arrays. o Bob gave an overview of this material - Martin reported that Tom Sugden now has some time and will be working on implementing a prototype of the conversion material

2 1

Telcon now
by Westhead, Martin (Martin) 03 May '06

03 May '06

Sorry for the late warning Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

1 0

Where's MikeB - unable to make the call on 5/3/06
by Mike Beckerle 03 May '06

03 May '06

Sorry folks. I'm traveling, booked in meetings and hence unable to make the call this week. However, status: I have started re-drafting the scoping document to be consistent with the context stuff. One more good plane flight and I'll have the respin done. ...mikeb

1 0

Telcon today
by Westhead, Martin (Martin) 26 Apr '06

26 Apr '06

I'm afraid I will be unable to attend this week...sorry. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call.

2 1

Telcon today
by Westhead, Martin (Martin) 19 Apr '06

19 Apr '06

Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

1 0

Telcon today
by Westhead, Martin (Martin) 17 Apr '06

17 Apr '06

Sorry for the late reminder. I'm afraid I will be unable to attend this week. Agenda items: - issue of defaults - arrays discussion - outstanding property document discussion Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

3 5

RE: [dfdl-wg] string vs. hexBinary vs. opaque
by Westhead, Martin (Martin) 12 Apr '06

12 Apr '06

I like this Mike, I think its very useful. _____ From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of Mike Beckerle Sent: Wednesday, April 12, 2006 1:01 PM To: dfdl-wg(a)ggf.org Subject: [dfdl-wg] string vs. hexBinary vs. opaque I did some thinking on this topic. At least some IBMers thought this was coherent.

1 0

RE: [dfdl-wg] How to deal with variable length elements?
by Westhead, Martin (Martin) 12 Apr '06

12 Apr '06

Hi Mike, I still don't quite understand could you put it in the context of a more complete example? Thanks, Martin _____ From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of Mike Beckerle Sent: Wednesday, April 12, 2006 12:16 PM To: Westhead, Martin (Martin) Cc: dfdl-wg(a)ggf.org; Robert E. McGrath; owner-dfdl-wg(a)ggf.org Subject: RE: [dfdl-wg] How to deal with variable length elements? The reason for the special @dfdl:index is because of the Xpath rules that position() is always inside the current context expression containing the call to position(). We need to index another structure with our index position. Can't do this in straight Xpath. If we add a "Let x = position() in ..." construct (Xquery has this), then we wouldn't need the @dfdl:index. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA 01581 voice and FAX 508-599-7148 home/mobile office 508-915-4795 "Westhead, Martin (Martin)" <westhead(a)avaya.com> Sent by: owner-dfdl-wg(a)ggf.org 04/05/2006 10:26 AM To Mike Beckerle/Worcester/IBM@IBMUS, "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> cc <dfdl-wg(a)ggf.org> Subject RE: [dfdl-wg] How to deal with variable length elements? Hi, This looks fine to me (modulo old syntax) except I don't understand the need to introduce this construct: "@dfdl:index". I guess in today's terms we might thing of this as a value in the context. However, I claim that we don't need it. I think we can achieve the same effect with the XPath function position(). position() should tell you where you are in the current sequence. If you want to know where you are in the parent sequence you can use ../position(). If your index starts from 0 use position()-1. If you store two elements in your sequence for every index (e.g. flat array of x-y coordinates) use position()/2. What does @dfdl:index do that you don't get out of position()? Cheers, Martin _____ From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of Mike Beckerle Sent: Friday, March 17, 2006 2:14 PM To: Robert E. McGrath Cc: dfdl-wg(a)ggf.org; owner-dfdl-wg(a)ggf.org Subject: Re: [dfdl-wg] How to deal with variable length elements? Hmmm. I think layered value calculation formulas which allow for a magic "myIndex" variable are perhaps an important device to make this class of layering possible.This makes the iteration over the elements implicit. I have one example which is the one where there is a vector of strings where the lengths of all the strings are stored first, separately from all the character data. A formula involving "myIndex" is used to glue the two pieces together. Here's the example as per our prototype from last summer: ...mikeb "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> Sent by: owner-dfdl-wg(a)ggf.org 03/17/2006 10:54 AM To dfdl-wg(a)ggf.org cc Subject Re: [dfdl-wg] How to deal with variable length elements? Following up on my email on ealier this week: I think there was a major flaw in what I wrote, and it is quite an "interesting" challenge. Let me review: I am thinking about how to describe reading data into a 1D array. Steve provided a markup for the XML element. The challenge I'm looking at is that the data need not be a image of the memory layout. To give one example, a very sparse array might be stored as a series of (index, value) pairs for the non-empty places, all others implied to be zero or fill or whatever. The goal is to have the XML array be fully populated from this sparse form--or whatever layout--on disk. (Please assume for now that this is a reasonable goal!) The XML and DFDL will tell us the data type, and presumably we know the extent of the data on disk. But we need to decode the storage to generate all the elements values and fills. In my earlier email, I offered a description that included an 'Iterator' conversion. I now think this is inadequate. In fact you need two cooperating 'Iterators'! Ick! Here is my revised pipeline. Data is read from bottom to top. I sketch what each conversion is tasked to do. I think the 'Decoder' needs to know info from both the 'Iterator' (it asks for each element in the order it wants them) and 'Float' (it tells the size of the 'value' to get). == <<XML element with multiple occurs: 1D array >> ^ | Iterator conversion: relevant props: minOccurs="0" maxOccurs="<<setting>>, et al Get 'maxOccurs' elements of type datatype. ^ | Float conversion: relevant props: data description of element Decode bytes ^ | Decoder conversion: produces the bytes 'nth' _value_ in the array. Input: what position is needed. may need separators and other props: depends on encoding Output: sizeof datatype bytes, the _value_ Side effect: after whole array is read, consumes all the storage. Difficult to characterize the intermediate state. ^ | Data: read as bytes

2 2

RE: [dfdl-wg] How to deal with variable length elements?
by Westhead, Martin (Martin) 12 Apr '06

12 Apr '06

Hi, This looks fine to me (modulo old syntax) except I don't understand the need to introduce this construct: "@dfdl:index". I guess in today's terms we might thing of this as a value in the context. However, I claim that we don't need it. I think we can achieve the same effect with the XPath function position(). position() should tell you where you are in the current sequence. If you want to know where you are in the parent sequence you can use ../position(). If your index starts from 0 use position()-1. If you store two elements in your sequence for every index (e.g. flat array of x-y coordinates) use position()/2. What does @dfdl:index do that you don't get out of position()? Cheers, Martin _____ From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of Mike Beckerle Sent: Friday, March 17, 2006 2:14 PM To: Robert E. McGrath Cc: dfdl-wg(a)ggf.org; owner-dfdl-wg(a)ggf.org Subject: Re: [dfdl-wg] How to deal with variable length elements? Hmmm. I think layered value calculation formulas which allow for a magic "myIndex" variable are perhaps an important device to make this class of layering possible.This makes the iteration over the elements implicit. I have one example which is the one where there is a vector of strings where the lengths of all the strings are stored first, separately from all the character data. A formula involving "myIndex" is used to glue the two pieces together. Here's the example as per our prototype from last summer: ...mikeb "Robert E. McGrath" <mcgrath(a)ncsa.uiuc.edu> Sent by: owner-dfdl-wg(a)ggf.org 03/17/2006 10:54 AM To dfdl-wg(a)ggf.org cc Subject Re: [dfdl-wg] How to deal with variable length elements? Following up on my email on ealier this week: I think there was a major flaw in what I wrote, and it is quite an "interesting" challenge. Let me review: I am thinking about how to describe reading data into a 1D array. Steve provided a markup for the XML element. The challenge I'm looking at is that the data need not be a image of the memory layout. To give one example, a very sparse array might be stored as a series of (index, value) pairs for the non-empty places, all others implied to be zero or fill or whatever. The goal is to have the XML array be fully populated from this sparse form--or whatever layout--on disk. (Please assume for now that this is a reasonable goal!) The XML and DFDL will tell us the data type, and presumably we know the extent of the data on disk. But we need to decode the storage to generate all the elements values and fills. In my earlier email, I offered a description that included an 'Iterator' conversion. I now think this is inadequate. In fact you need two cooperating 'Iterators'! Ick! Here is my revised pipeline. Data is read from bottom to top. I sketch what each conversion is tasked to do. I think the 'Decoder' needs to know info from both the 'Iterator' (it asks for each element in the order it wants them) and 'Float' (it tells the size of the 'value' to get). == <<XML element with multiple occurs: 1D array >> ^ | Iterator conversion: relevant props: minOccurs="0" maxOccurs="<<setting>>, et al Get 'maxOccurs' elements of type datatype. ^ | Float conversion: relevant props: data description of element Decode bytes ^ | Decoder conversion: produces the bytes 'nth' _value_ in the array. Input: what position is needed. may need separators and other props: depends on encoding Output: sizeof datatype bytes, the _value_ Side effect: after whole array is read, consumes all the storage. Difficult to characterize the intermediate state. ^ | Data: read as bytes

2 1