- dfdl-wg - lists.ogf.org

RE: [dfdl-wg] CSV string worked example
by Jim Myers 01 Mar '06

01 Mar '06

> >EOS is up for grabs I was thinking of it as a returned value (e.g. >-1) but an exception might (or might not) be easier to make sense of. Assuming -1 is a valid value and not used as a real value, etc. - I think we need a separate mechanism to indicate the end > >Regarding the new model. I don't think this is a problem at the >level of your example. We could simply use a single sequence and a >more complex "split" conversion. I imagine that the "split" >conversion we would want to settle on should accept a regular >expression (or at least a list of separators). In your example you >just have to allow the separator to be a new line OR a comma and you are done. Yes but - I want to reuse the simple conversions and therefore keep the ability to deal with all the variations you note below without special constructs - if I consider this a sequence of two steps I can deal with missing separators and terminators differently, if I put the two together, I have to account for all the possible variations. And I want to handle a data cube the same way without having to wait for someone to build a new converter...(we talked this through generally before - if you have a way to create new converters from existing ones, you could support both ways). > >A note here this is intended as a rough sketch not a finished >design. I am expecting the details to need to be worked out here. In >particular I think Mike/IBM have some fairly complex ideas for >separator/terminator/initiator/escape that we will have to try to >seat in this framework. > >Thanks, > >Martin > > >---------- >From: Jim Myers [mailto:jimmyers@ncsa.uiuc.edu] >Sent: Wednesday, March 01, 2006 3:49 AM >To: Westhead, Martin (Martin); dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] CSV string worked example > >Martin - two types of comments - things I think are >typos/inconsistencies and an alternate logic: > >Clarifications: >are the initial definitions on the top element defining an order to >use subsequently or are they just there for us to see what you've defined? >Of the four there, you only explicitly (in a comment?) invoke one - >are the others implicit because of the order? >You use dfdl:tokenizer as a conversion later - is that supposed to >be split as well? >bytetochar is used implicitly before the first split? >chartostring is used implicitly before stringtoint which is >implicitly used to get the int element? >is EOS a returned value (and therefore of the type being returned) >or is it an exception? > >Logical - what happens if the rows are not in the logical model - >physically there are 10 rows with 5 elements, but the logical model >is 50 ints in a single sequence. To support this, you'd need to have >both tokenization steps in one sequence annotation with two separate >split separators - does the use of setLocal for split separator work >in this case? (Is this how byteorder is now used?) >Thinking about missing values - is it clear how a missing row versus >a missing element is now handled (I think so) - the split conversion >using comma can define a default input to use if the stream it >recieves is empty (from a \n\n pair) and the stringtoint conversion >can do likewise to cover a ,, pair. > > Jim > > >At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: > >Hi Folks, > >I have tried to work through the CSV example that Mike suggested a >couple of weeks ago. It has turned up some interesting issues which >I have tried to address. These are less about making the underlying >semantics work and more about providing a seamless default set up >that makes the easy things work just as you would like. > >I was pushed for time on this so I apologies if this is unclear in >places, but I wanted to put it out before tomorrow's meeting. > >Thanks, > >Martin > >James D. Myers >Associate Director, Cyberenvironments and Technologies, NCSA >1205 W. Clark St, MC-257 >Urbana, IL 61801 >217-244-1934 >jimmyers(a)ncsa.uiuc.edu James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

RE: [dfdl-wg] CSV string worked example
by Westhead, Martin (Martin) 01 Mar '06

01 Mar '06

Hi Jim, I think the others should be implicitly used because of order and type. Sorry tokenizer should be split - unpropagated change. Chartostring (which I called concatenate) is to be used first (because it is the more specific match). EOS is up for grabs I was thinking of it as a returned value (e.g. -1) but an exception might (or might not) be easier to make sense of. Regarding the new model. I don't think this is a problem at the level of your example. We could simply use a single sequence and a more complex "split" conversion. I imagine that the "split" conversion we would want to settle on should accept a regular expression (or at least a list of separators). In your example you just have to allow the separator to be a new line OR a comma and you are done. A note here this is intended as a rough sketch not a finished design. I am expecting the details to need to be worked out here. In particular I think Mike/IBM have some fairly complex ideas for separator/terminator/initiator/escape that we will have to try to seat in this framework. Thanks, Martin _____ From: Jim Myers [mailto:jimmyers@ncsa.uiuc.edu] Sent: Wednesday, March 01, 2006 3:49 AM To: Westhead, Martin (Martin); dfdl-wg(a)ggf.org Subject: Re: [dfdl-wg] CSV string worked example Martin - two types of comments - things I think are typos/inconsistencies and an alternate logic: Clarifications: are the initial definitions on the top element defining an order to use subsequently or are they just there for us to see what you've defined? Of the four there, you only explicitly (in a comment?) invoke one - are the others implicit because of the order? You use dfdl:tokenizer as a conversion later - is that supposed to be split as well? bytetochar is used implicitly before the first split? chartostring is used implicitly before stringtoint which is implicitly used to get the int element? is EOS a returned value (and therefore of the type being returned) or is it an exception? Logical - what happens if the rows are not in the logical model - physically there are 10 rows with 5 elements, but the logical model is 50 ints in a single sequence. To support this, you'd need to have both tokenization steps in one sequence annotation with two separate split separators - does the use of setLocal for split separator work in this case? (Is this how byteorder is now used?) Thinking about missing values - is it clear how a missing row versus a missing element is now handled (I think so) - the split conversion using comma can define a default input to use if the stream it recieves is empty (from a \n\n pair) and the stringtoint conversion can do likewise to cover a ,, pair. Jim At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: Hi Folks, I have tried to work through the CSV example that Mike suggested a couple of weeks ago. It has turned up some interesting issues which I have tried to address. These are less about making the underlying semantics work and more about providing a seamless default set up that makes the easy things work just as you would like. I was pushed for time on this so I apologies if this is unclear in places, but I wanted to put it out before tomorrow's meeting. Thanks, Martin James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

Re: [dfdl-wg] CSV string worked example
by Jim Myers 01 Mar '06

01 Mar '06

Martin - two types of comments - things I think are typos/inconsistencies and an alternate logic: Clarifications: are the initial definitions on the top element defining an order to use subsequently or are they just there for us to see what you've defined? Of the four there, you only explicitly (in a comment?) invoke one - are the others implicit because of the order? You use dfdl:tokenizer as a conversion later - is that supposed to be split as well? bytetochar is used implicitly before the first split? chartostring is used implicitly before stringtoint which is implicitly used to get the int element? is EOS a returned value (and therefore of the type being returned) or is it an exception? Logical - what happens if the rows are not in the logical model - physically there are 10 rows with 5 elements, but the logical model is 50 ints in a single sequence. To support this, you'd need to have both tokenization steps in one sequence annotation with two separate split separators - does the use of setLocal for split separator work in this case? (Is this how byteorder is now used?) Thinking about missing values - is it clear how a missing row versus a missing element is now handled (I think so) - the split conversion using comma can define a default input to use if the stream it recieves is empty (from a \n\n pair) and the stringtoint conversion can do likewise to cover a ,, pair. Jim At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: >Hi Folks, > >I have tried to work through the CSV example that Mike suggested a >couple of weeks ago. It has turned up some interesting issues which >I have tried to address. These are less about making the underlying >semantics work and more about providing a seamless default set up >that makes the easy things work just as you would like. > >I was pushed for time on this so I apologies if this is unclear in >places, but I wanted to put it out before tomorrow's meeting. > >Thanks, > >Martin James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

CSV string worked example
by Westhead, Martin (Martin) 01 Mar '06

01 Mar '06

Hi Folks, I have tried to work through the CSV example that Mike suggested a couple of weeks ago. It has turned up some interesting issues which I have tried to address. These are less about making the underlying semantics work and more about providing a seamless default set up that makes the easy things work just as you would like. I was pushed for time on this so I apologies if this is unclear in places, but I wanted to put it out before tomorrow's meeting. Thanks, Martin

1 0

DFDL properties - points to discuss at F2F
by Steve Hanson 26 Feb '06

26 Feb '06

I've been progressing the DFDL Properties tracker and several issues have come up that need resolving before I can suggest a complete list of DFDL properties. Please have a think about these and we can discuss at the F2F. 1) When parsing, we want to be flexible and tolerate data appearing different in forms. For example, allow initiators and terminators to be in upper or lower case, allow UTC timezone to be +00:00 or Z, allow thousand separators for decimal numbers, etc. But on output we must make a choice as to which form we use. We need some principles that we apply to all properties that exhibit this behaviour. 2) It looks like there are real-life examples of initiators and terminators being in a different character encoding to the accompanying data. Implies use of initiatorEncoding etc properties. Need to agree that this is the correct way to proceed. 3) Property precedence and the use of 'notApplicable' enums to say that a property is not to be interpreted on this object. Without this it is not possible to override a dfdl:default setting from higher up the scope. For example, how do I switch off the use of separator for a group when a dfdl:default has set it higher up. A value of notApplicable would solve that. Another example, I have a sequence where variable length fields have a terminator but fixed length fields do not. My terminator and most of my lengths are common, so I set both length and terminator at the group level using dfdl:default. When property scoping rules are applied, I will have a length and a terminator for each field, which do I use? I can't just say length takes precedence because every field has been given a length. I need a way of explicitly saying which to use. One approach is to use a special value for length - notApplicable' - which means ignore the property, and I would use that explicitly on all the variable length fields. Alternatively I can be more explicit and have a separate property to control which is used. 4) Use of 'Native' enum meaning use the locally defined value. Used for character encodings, time zones, endian-ness. The DFDL spec draft says: "There should be no ‘platform varying defaults’. For example, byteOrder should default to bigEndian, or littleEndian, or have no default at all (in which case leaving it unspecified will often cause an error except in all-text situations with non-endian character sets). What’s not acceptable is for byteOrder to default to some value based on the current platform or other environmental constraint. Similarly for locale-sensitive things". Does this imply that 'Native' can't be the default? I can see why we would not want something to default silently based on platform/locale, but by being clear that the default is 'Native' avoids that. 5) Agree on what offset facilities are to be offered for establishing the position of an item. Absolute and/or relative. Problem with absolute offsets is their fragility, a single variable length field breaks the scheme unless the offset is given by an expression. Are relative offsets all we really need? And what are they relative to? Last field? 6) Binary versus text. At the moment the Schema for DFDL has the text model inheriting from the binary model. So text isn't really text, it's text or binary. I think this is wrong and that repType=text and repType=binary really imply separate semantics. It still means properties can be shared, but it clarifies things like the behavior of built-in prefixed lengths - a length prefix for a binary string would be a physical integer in the data, a length prefix for a text string would be a physical numeric string in the data. 7) Padding character - does setting this property imply trim on parse as well as pad on output? Do we need a control for this (pad/trim/both) 8) Justification, date/time format, and some other properties have a default that varies depending on the logical type of the element. How does this default interact with setting a default via scoping rules? We have said that there are no hard-coded defaults, implying a dfdl:defineFormat block must exist and must define values for all properties. This contradicts deriving the default from the logical type. For such properties, we could have an accompanying dfdl:defineFormat-only property that says how it defaults. Or we could exempt these from having to have a default set in dfdl:defineFormat. 9) There are many properties that are inherited from the CAM binary model. Suman is establishing with IBM compiler people exactly what ones are needed and what their semantics are. I find some of the names unhelpful and suggest that renaming such properties is a good idea. 10) Decimal properties. There are a host of properties for controlling decimal formats, it's not clear to me what some of them mean. I'd like us to agree on a finished set. 11) Group level properties. We don't have many properties that define how the members of a group behave. We have separator for instance that says all child fields will be separated. Is there benefit in having other properties like this, for example 'initiated' meaning all child fields must have a unique initiator, or 'fixed' meaning all child fields must have a pre-computable length. Such properties have a secondary benefits - they communicate the nature of the group without recourse to examination of all children, and they enable DFDL editors to validate that all intended properties are present. Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

F2F attendance
by Westhead, Martin (Martin) 22 Feb '06

22 Feb '06

(Dates: 1-3rd March 8.30am start on Wed morning.) Hi Folks, I need to give our security a list of people coming. If you are not on this list and planning to attend the f2f next week (in person) please contact me. Mike Beckerle Geoff Judd Steve Hanson Suman Kalia Robert McGrath Vegetarians - I am arranging some lunch to be provided. If you want me to get vegetarian (or other dietary need) food please let me know. FYI - there is a cafeteria very close to our meeting room it is open in the morning for coffee and pastries, fruit salad etc. I am also planning to host a meal for Wed night at a local Tai restaurant please let me know if you do not like Tai food.

1 0

Telcon notes 2006-2-22
by Westhead, Martin (Martin) 22 Feb '06

22 Feb '06

Present Martin Westhead Tara Talbott Suman Kalia Bob McGrath Geoff Judd Steve Hanson Jim Myers 1. F2F Logistics and Agenda Logistics: aim for an 8.30 am start on March 1st. Conference phone dialin: 1- 866-AVAYA-22 Participant Code: 785343 WebConf Reference: 158672 Web conf url: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> (Note that the participant code for the phone and web conference is the same.) 158672 The discussion about variables and the context seems to be largely resolved, otherwise the Agenda 2. Reports from trackers Some progress is being made on several trackers (scoping, defaults, choices...etc.) for presentation at the F2F. 3. Variables and Context We reviewed a discussion between Martin and Geoff over the semantics of variables and the context. Geoff had an important requirement that could not be handled by contexts illustrated by the following example: <xsd:element name="outerElem"> <xsd:complexType> <xsd:sequence> <xsd:element name="innerElem" maxOccurs="10"> <xsd:complexType> <xsd:sequence> <xsd:element name="numberOfEntries" type="xsd:int" dfdl:setGlobal="COUNT"/> <xsd:element name="repString" type="xsd:string" minOccurs="0" maxOccurs="unbounded" dfdl:repeatCount="$COUNT"/> </xsd:sequence> <xsd:complexType> </xsd:sequence> </xsd:complexType> </xsd:element> With the current semantics this would fail because a global context value can only be set once. Setting in the local context would not work because COUNT would not be in the context of the repString element. A resolution to this issue was agreed involving introducing a declaration statement which could declare but not set a variable. When a local context set was used, if the identifier referred to such a declared but unassigned variable it would have the effect of assigning a value to that instead of setting the local context. In this way we could control the depth at which the context value was defined and ensure that it occurred at the right level. 4. Conversions etc. No new material here. We are still waiting for a more fully worked example using some sort of string separator. Hopefully this will be ready for the F2F.

1 0

Telcon tomorrow
by Westhead, Martin (Martin) 22 Feb '06

22 Feb '06

Please note: I have to get into the office for the call this week so I may be late (stuck in 101 traffic). If so please start without me. - Martin Preliminary Agenda ------------------ 1. F2F Logistics and Agenda (see attached from last week's notes) 2. Reports from trackers 3. Variables and Context 4. Conversions etc. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672 F2F Agenda DFDL underlying mechanisms - Martin Conversions/context/hidden elements - Martin - Relationship to variables - Geoff/Martin Scoping document - Mike Properties/conversions/grouping - Steve/Suman Choices and wildcards - Geoff Null and default values - Geoff Validation Actions to lead to a final document set. - Martin - Which XML Schema constructs are supported - Steve - Review of trackers - All Validation * Validation of the DFDL schema (consistent properties) * Validation of the DFDL document * Choices based on failure

1 0

Notes from telcon 2006-2-15
by Westhead, Martin (Martin) 15 Feb '06

15 Feb '06

Present Martin Westhead Suman Kalia Mike Beckerle Tara Talbott Bob McGrath Geoff Judd F2F Agenda DFDL underlying mechanisms - Martin Conversions/context/hidden elements - Martin - Relationship to variables - Geoff/Martin Scoping document - Mike Properties/conversions/grouping - Steve/Suman Choices and wildcards - Geoff Null and default values - Geoff Validation Actions to lead to a final document set. - Martin - Which XML Schema constructs are supported - Steve - Review of trackers - All Validation * Validation of the DFDL schema (consistent properties) * Validation of the DFDL document * Choices based on failure Discussion We spent the last 15 minutes of the call discussing issues around the "conversions" document. Mike requested an example which showed how separator conversions could be written in this way.

1 0

Telcon agenda for tomorrow
by Westhead, Martin (Martin) 15 Feb '06

15 Feb '06

Preliminary Agenda ------------------ 1. Agenda for F2F 2. Reports from trackers 3. Ongoing review "Conversions" document 4. Review "Bundles" document Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference Reference: 158672 Security Code : 785343

2 1