- dfdl-wg - lists.ogf.org

Notes from Face to Face day 2 and 3
by Westhead, Martin (Martin) 03 Mar '06

03 Mar '06

Present Martin Westhead Steve Hanson Suman Kalia Geoff Judd Actions The following actions were assigned: * Scoping and context [guards and tests] (Mike) * Properties and conversions (Steve) * Revisit conversions [Steve's questions, type matching, position, speculative parsing, where are we?](Martin) * Uncertainty [wildcards, substitution groups, choices, optionality] (Geoff) * Defaults and nulls (Geoff) * DFDL Schema (Suman) Open issues: * syntax (sequences, markup) * identifying core properties and conversions (if any) * round tripping * embedded XML Day 2 Notes Properties/conversions/grouping - Steve/Suman (Suman took notes on this topic) Steve will be re-writing the current document to take on board proposals from this meeting and to start the process (with Martin) of trying to converge the properties with the conversions material. Choices and wildcards - Geoff We explored several ideas around the framework that Geoff has. Geoff will be writing these up. Comments included: * Possible use of binhex to represent "deferred" choices * Modeled uncertainty with "any" should be treated as a choice * Substitution groups should be treated as a choice * Wildcards will be handled here and "open" wildcards (unmodeled uncertainty) could be handled by dropping in a string/binhex * Schematron-like assertions could be used as choice guards such assertions could also make general statements about data validity * Use of XML Schema validation to resolve speculative parsing raises issues. (Either all DFDL parsers must validate or would get different results) Which XML Schema constructs are supported - Steve We reviewed Steves slides made various modifications key decisions: * Attributes - wait and see Final documents We had an inconclusive discussion on the structure of the final documents. Day 3 Notes Instance Validation - Efficiency of XML Schema validation - Implementation difficult of XML Schema validation - Therefore cannot force the use of full DFDL parsing - so we cannot use XML Schema validation to resolve speculative parsing - property to state validation to be used could allow parser to produce a warning error if not available - possibility: could use require just facet/value checks (not structure) - XPath evaluation required - Therefore XPath assertions for choice discrimination possible - And XPath assertions for describing valid data possible - EDI community has strong requirement for such assertions to be available (such as coexistance rules) - Is there a requirement for content validation (since construction is driven by model) - Might be useful to have standard names for classes of error DFDL Schema Validation - XML Schema valid - Syntax and semantics of annotations (annotations must be consistent with schema) - XPath references are valid (standard should describe ways validation can be done and features that can be validated, validation should not be mandated).

1 0

Phone number for F2f
by Westhead, Martin (Martin) 02 Mar '06

02 Mar '06

Sorry I forgot to send out a reminder of the F2F call in numbers it is: 1-866-AVAYA-22 Part. Security Code 785343 Web coference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> WebConf Reference 158672 Part. Security Code 785343 Please let us know if you dial in I'll keep the bridge open all day, but email me if there any issues. Thanks, Martin

1 0

notes from F2F 2006-03-01
by Mike Beckerle 02 Mar '06

02 Mar '06

Attendees: Bob McGrath - NCSA Martin Westhead - Avaya Suman Kalia - IBM Geoff Judd - IBM Steve Hanson - IBM Mike Beckerle - IBM Agenda Review Context doc discussion - Martin Issues list: - write once context variables - expressiveness limit? - list valued context variables - define them then 'commit' to make them read-only? - default values for context variables? - breaks write-once behavior? - hidden: don't want paths to hidden peers to be any different from paths to non-hidden peers - mikeb - hidden: need to specify the order of traversal for parsing - hidden: on output, traversal order is very different - need inputCalc and outputCalc - hidden: currently says hidden can only appear where elements could appear - exception needed - simple types need hidden complex sub-structure - 'repDef' - maybe use hidden attributes and avoid element substructure for simple types. - or have an explicit repDef construct with a full indirection from a simple type to a complexType that is its representation - Roundtrip problem is not general data transformation - Separate tracker for round-trip to data issue - Copyright slugline example is sufficient Bundles: - is this just really for the top-level statements like conversion declarations? - revisit this once we've overviewed the scoping stuff - probably some features of one or the other are redundant. - bundles are like xs:include except can be used not-only at top-level. Conversions: - are an expository tool - organize properties into libraries - plug in conversions are allowed, but built-in property/conversions don't have to be implemented using the plug-in mechanism - ? DFDL systems might be allowed to not even implement plug-ins - need to explain XML all the way down idea more clearly. Stream of xs:byte, then streams of higher types. - test guard should select the conversion - once selected then requirements - if you can't find any conversion for something then the schema is invalid. - if the guard of a conversion is satisfied, but required properties are not present in the context, then the schema is invalid. - issue: can more than one conversion have the same name? - issue: what are the update rules for the conversion list in the context as you extend the context with new conversions - issue: type matching - match exactly? or ignore restrictions? E.g., string of length 1 matches general string, but general string doesn't match string of length 1. - issue: type A has child derived type B. Conversions for type B cannot be selected based on existance of conversions that have source/target type A. (isA relationship not involved in conversion selection). The type matching has only to do with restrictions actually. - issue: can different dfdl implementations use different search algorithms? Probably not. They must choose the same set of conversions so the standard can specify the selection algorithm exactly. Implementations are free to use different more efficient algorithms so long as they choose equivalent conversion chains. (need notion of equivalent conversion defined) - issue: context needs to contain the input-stream type (in the "C" context, or just in the informal context of the parser? E.g., you can ask what is the type of this element without that having to be in the "C" context. The type of the source stream could also be in this informal context. - property names should be namespace qualified when used in guard and required formulas of conversions. - conversions stuff doesn't cover all properties. E.g., choice discriminators aren't described how they use as part of conversions. - issue: alignment - can there be an alignment conversion Extensibility: - defineConversion - should express required properties test as well as input type and output type. - whitebox conversions - doubleInt example doesn't seem intuitive to some. suggestion is to name the complex type, separate the named conversion definition and just reference the type name - change keyword for guards to 'guard' not test, since test is used for statement conditional execution. - name conflicts: guard vs. selector vs. test - define terms specifically. - need tracker item re: output - steve hanson suggests using EDI security segments for the useStream example. - guard optional - if not specified means guard="true" Scoping: - simplify: don't allow base format on construct-specific annoation elements if inside dfdl:default or dfdl:defineFormat. variable defaults in the context: - if a variable has a default value, and you keep track of whether anyone has received the default value if they have received the default value then an attempt to set it is an error. This is consistent with single assignment behavior. variable declaration scope - a dfdl:declare on an xsd:element with maxOccurs > 1 is about the type, not the particle (not about the entire repeating element vector, but about each element within). external control of format: - should do overrides much like element references, i.e., external controls have override behavior. leaf contexts: - ? do these distinguish properties from variables Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA 01581 voice and FAX 508-599-7148 home/mobile office 508-915-4795

1 0

RE: [dfdl-wg] CSV string worked example
by Jim Myers 01 Mar '06

01 Mar '06

> >EOS is up for grabs I was thinking of it as a returned value (e.g. >-1) but an exception might (or might not) be easier to make sense of. Assuming -1 is a valid value and not used as a real value, etc. - I think we need a separate mechanism to indicate the end > >Regarding the new model. I don't think this is a problem at the >level of your example. We could simply use a single sequence and a >more complex "split" conversion. I imagine that the "split" >conversion we would want to settle on should accept a regular >expression (or at least a list of separators). In your example you >just have to allow the separator to be a new line OR a comma and you are done. Yes but - I want to reuse the simple conversions and therefore keep the ability to deal with all the variations you note below without special constructs - if I consider this a sequence of two steps I can deal with missing separators and terminators differently, if I put the two together, I have to account for all the possible variations. And I want to handle a data cube the same way without having to wait for someone to build a new converter...(we talked this through generally before - if you have a way to create new converters from existing ones, you could support both ways). > >A note here this is intended as a rough sketch not a finished >design. I am expecting the details to need to be worked out here. In >particular I think Mike/IBM have some fairly complex ideas for >separator/terminator/initiator/escape that we will have to try to >seat in this framework. > >Thanks, > >Martin > > >---------- >From: Jim Myers [mailto:jimmyers@ncsa.uiuc.edu] >Sent: Wednesday, March 01, 2006 3:49 AM >To: Westhead, Martin (Martin); dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] CSV string worked example > >Martin - two types of comments - things I think are >typos/inconsistencies and an alternate logic: > >Clarifications: >are the initial definitions on the top element defining an order to >use subsequently or are they just there for us to see what you've defined? >Of the four there, you only explicitly (in a comment?) invoke one - >are the others implicit because of the order? >You use dfdl:tokenizer as a conversion later - is that supposed to >be split as well? >bytetochar is used implicitly before the first split? >chartostring is used implicitly before stringtoint which is >implicitly used to get the int element? >is EOS a returned value (and therefore of the type being returned) >or is it an exception? > >Logical - what happens if the rows are not in the logical model - >physically there are 10 rows with 5 elements, but the logical model >is 50 ints in a single sequence. To support this, you'd need to have >both tokenization steps in one sequence annotation with two separate >split separators - does the use of setLocal for split separator work >in this case? (Is this how byteorder is now used?) >Thinking about missing values - is it clear how a missing row versus >a missing element is now handled (I think so) - the split conversion >using comma can define a default input to use if the stream it >recieves is empty (from a \n\n pair) and the stringtoint conversion >can do likewise to cover a ,, pair. > > Jim > > >At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: > >Hi Folks, > >I have tried to work through the CSV example that Mike suggested a >couple of weeks ago. It has turned up some interesting issues which >I have tried to address. These are less about making the underlying >semantics work and more about providing a seamless default set up >that makes the easy things work just as you would like. > >I was pushed for time on this so I apologies if this is unclear in >places, but I wanted to put it out before tomorrow's meeting. > >Thanks, > >Martin > >James D. Myers >Associate Director, Cyberenvironments and Technologies, NCSA >1205 W. Clark St, MC-257 >Urbana, IL 61801 >217-244-1934 >jimmyers(a)ncsa.uiuc.edu James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

RE: [dfdl-wg] CSV string worked example
by Westhead, Martin (Martin) 01 Mar '06

01 Mar '06

Hi Jim, I think the others should be implicitly used because of order and type. Sorry tokenizer should be split - unpropagated change. Chartostring (which I called concatenate) is to be used first (because it is the more specific match). EOS is up for grabs I was thinking of it as a returned value (e.g. -1) but an exception might (or might not) be easier to make sense of. Regarding the new model. I don't think this is a problem at the level of your example. We could simply use a single sequence and a more complex "split" conversion. I imagine that the "split" conversion we would want to settle on should accept a regular expression (or at least a list of separators). In your example you just have to allow the separator to be a new line OR a comma and you are done. A note here this is intended as a rough sketch not a finished design. I am expecting the details to need to be worked out here. In particular I think Mike/IBM have some fairly complex ideas for separator/terminator/initiator/escape that we will have to try to seat in this framework. Thanks, Martin _____ From: Jim Myers [mailto:jimmyers@ncsa.uiuc.edu] Sent: Wednesday, March 01, 2006 3:49 AM To: Westhead, Martin (Martin); dfdl-wg(a)ggf.org Subject: Re: [dfdl-wg] CSV string worked example Martin - two types of comments - things I think are typos/inconsistencies and an alternate logic: Clarifications: are the initial definitions on the top element defining an order to use subsequently or are they just there for us to see what you've defined? Of the four there, you only explicitly (in a comment?) invoke one - are the others implicit because of the order? You use dfdl:tokenizer as a conversion later - is that supposed to be split as well? bytetochar is used implicitly before the first split? chartostring is used implicitly before stringtoint which is implicitly used to get the int element? is EOS a returned value (and therefore of the type being returned) or is it an exception? Logical - what happens if the rows are not in the logical model - physically there are 10 rows with 5 elements, but the logical model is 50 ints in a single sequence. To support this, you'd need to have both tokenization steps in one sequence annotation with two separate split separators - does the use of setLocal for split separator work in this case? (Is this how byteorder is now used?) Thinking about missing values - is it clear how a missing row versus a missing element is now handled (I think so) - the split conversion using comma can define a default input to use if the stream it recieves is empty (from a \n\n pair) and the stringtoint conversion can do likewise to cover a ,, pair. Jim At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: Hi Folks, I have tried to work through the CSV example that Mike suggested a couple of weeks ago. It has turned up some interesting issues which I have tried to address. These are less about making the underlying semantics work and more about providing a seamless default set up that makes the easy things work just as you would like. I was pushed for time on this so I apologies if this is unclear in places, but I wanted to put it out before tomorrow's meeting. Thanks, Martin James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

Re: [dfdl-wg] CSV string worked example
by Jim Myers 01 Mar '06

01 Mar '06

Martin - two types of comments - things I think are typos/inconsistencies and an alternate logic: Clarifications: are the initial definitions on the top element defining an order to use subsequently or are they just there for us to see what you've defined? Of the four there, you only explicitly (in a comment?) invoke one - are the others implicit because of the order? You use dfdl:tokenizer as a conversion later - is that supposed to be split as well? bytetochar is used implicitly before the first split? chartostring is used implicitly before stringtoint which is implicitly used to get the int element? is EOS a returned value (and therefore of the type being returned) or is it an exception? Logical - what happens if the rows are not in the logical model - physically there are 10 rows with 5 elements, but the logical model is 50 ints in a single sequence. To support this, you'd need to have both tokenization steps in one sequence annotation with two separate split separators - does the use of setLocal for split separator work in this case? (Is this how byteorder is now used?) Thinking about missing values - is it clear how a missing row versus a missing element is now handled (I think so) - the split conversion using comma can define a default input to use if the stream it recieves is empty (from a \n\n pair) and the stringtoint conversion can do likewise to cover a ,, pair. Jim At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: >Hi Folks, > >I have tried to work through the CSV example that Mike suggested a >couple of weeks ago. It has turned up some interesting issues which >I have tried to address. These are less about making the underlying >semantics work and more about providing a seamless default set up >that makes the easy things work just as you would like. > >I was pushed for time on this so I apologies if this is unclear in >places, but I wanted to put it out before tomorrow's meeting. > >Thanks, > >Martin James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

CSV string worked example
by Westhead, Martin (Martin) 01 Mar '06

01 Mar '06

Hi Folks, I have tried to work through the CSV example that Mike suggested a couple of weeks ago. It has turned up some interesting issues which I have tried to address. These are less about making the underlying semantics work and more about providing a seamless default set up that makes the easy things work just as you would like. I was pushed for time on this so I apologies if this is unclear in places, but I wanted to put it out before tomorrow's meeting. Thanks, Martin

1 0

DFDL properties - points to discuss at F2F
by Steve Hanson 26 Feb '06

26 Feb '06

I've been progressing the DFDL Properties tracker and several issues have come up that need resolving before I can suggest a complete list of DFDL properties. Please have a think about these and we can discuss at the F2F. 1) When parsing, we want to be flexible and tolerate data appearing different in forms. For example, allow initiators and terminators to be in upper or lower case, allow UTC timezone to be +00:00 or Z, allow thousand separators for decimal numbers, etc. But on output we must make a choice as to which form we use. We need some principles that we apply to all properties that exhibit this behaviour. 2) It looks like there are real-life examples of initiators and terminators being in a different character encoding to the accompanying data. Implies use of initiatorEncoding etc properties. Need to agree that this is the correct way to proceed. 3) Property precedence and the use of 'notApplicable' enums to say that a property is not to be interpreted on this object. Without this it is not possible to override a dfdl:default setting from higher up the scope. For example, how do I switch off the use of separator for a group when a dfdl:default has set it higher up. A value of notApplicable would solve that. Another example, I have a sequence where variable length fields have a terminator but fixed length fields do not. My terminator and most of my lengths are common, so I set both length and terminator at the group level using dfdl:default. When property scoping rules are applied, I will have a length and a terminator for each field, which do I use? I can't just say length takes precedence because every field has been given a length. I need a way of explicitly saying which to use. One approach is to use a special value for length - notApplicable' - which means ignore the property, and I would use that explicitly on all the variable length fields. Alternatively I can be more explicit and have a separate property to control which is used. 4) Use of 'Native' enum meaning use the locally defined value. Used for character encodings, time zones, endian-ness. The DFDL spec draft says: "There should be no ‘platform varying defaults’. For example, byteOrder should default to bigEndian, or littleEndian, or have no default at all (in which case leaving it unspecified will often cause an error except in all-text situations with non-endian character sets). What’s not acceptable is for byteOrder to default to some value based on the current platform or other environmental constraint. Similarly for locale-sensitive things". Does this imply that 'Native' can't be the default? I can see why we would not want something to default silently based on platform/locale, but by being clear that the default is 'Native' avoids that. 5) Agree on what offset facilities are to be offered for establishing the position of an item. Absolute and/or relative. Problem with absolute offsets is their fragility, a single variable length field breaks the scheme unless the offset is given by an expression. Are relative offsets all we really need? And what are they relative to? Last field? 6) Binary versus text. At the moment the Schema for DFDL has the text model inheriting from the binary model. So text isn't really text, it's text or binary. I think this is wrong and that repType=text and repType=binary really imply separate semantics. It still means properties can be shared, but it clarifies things like the behavior of built-in prefixed lengths - a length prefix for a binary string would be a physical integer in the data, a length prefix for a text string would be a physical numeric string in the data. 7) Padding character - does setting this property imply trim on parse as well as pad on output? Do we need a control for this (pad/trim/both) 8) Justification, date/time format, and some other properties have a default that varies depending on the logical type of the element. How does this default interact with setting a default via scoping rules? We have said that there are no hard-coded defaults, implying a dfdl:defineFormat block must exist and must define values for all properties. This contradicts deriving the default from the logical type. For such properties, we could have an accompanying dfdl:defineFormat-only property that says how it defaults. Or we could exempt these from having to have a default set in dfdl:defineFormat. 9) There are many properties that are inherited from the CAM binary model. Suman is establishing with IBM compiler people exactly what ones are needed and what their semantics are. I find some of the names unhelpful and suggest that renaming such properties is a good idea. 10) Decimal properties. There are a host of properties for controlling decimal formats, it's not clear to me what some of them mean. I'd like us to agree on a finished set. 11) Group level properties. We don't have many properties that define how the members of a group behave. We have separator for instance that says all child fields will be separated. Is there benefit in having other properties like this, for example 'initiated' meaning all child fields must have a unique initiator, or 'fixed' meaning all child fields must have a pre-computable length. Such properties have a secondary benefits - they communicate the nature of the group without recourse to examination of all children, and they enable DFDL editors to validate that all intended properties are present. Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

F2F attendance
by Westhead, Martin (Martin) 22 Feb '06

22 Feb '06

(Dates: 1-3rd March 8.30am start on Wed morning.) Hi Folks, I need to give our security a list of people coming. If you are not on this list and planning to attend the f2f next week (in person) please contact me. Mike Beckerle Geoff Judd Steve Hanson Suman Kalia Robert McGrath Vegetarians - I am arranging some lunch to be provided. If you want me to get vegetarian (or other dietary need) food please let me know. FYI - there is a cafeteria very close to our meeting room it is open in the morning for coffee and pastries, fruit salad etc. I am also planning to host a meal for Wed night at a local Tai restaurant please let me know if you do not like Tai food.

1 0

Telcon notes 2006-2-22
by Westhead, Martin (Martin) 22 Feb '06

22 Feb '06

Present Martin Westhead Tara Talbott Suman Kalia Bob McGrath Geoff Judd Steve Hanson Jim Myers 1. F2F Logistics and Agenda Logistics: aim for an 8.30 am start on March 1st. Conference phone dialin: 1- 866-AVAYA-22 Participant Code: 785343 WebConf Reference: 158672 Web conf url: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> (Note that the participant code for the phone and web conference is the same.) 158672 The discussion about variables and the context seems to be largely resolved, otherwise the Agenda 2. Reports from trackers Some progress is being made on several trackers (scoping, defaults, choices...etc.) for presentation at the F2F. 3. Variables and Context We reviewed a discussion between Martin and Geoff over the semantics of variables and the context. Geoff had an important requirement that could not be handled by contexts illustrated by the following example: <xsd:element name="outerElem"> <xsd:complexType> <xsd:sequence> <xsd:element name="innerElem" maxOccurs="10"> <xsd:complexType> <xsd:sequence> <xsd:element name="numberOfEntries" type="xsd:int" dfdl:setGlobal="COUNT"/> <xsd:element name="repString" type="xsd:string" minOccurs="0" maxOccurs="unbounded" dfdl:repeatCount="$COUNT"/> </xsd:sequence> <xsd:complexType> </xsd:sequence> </xsd:complexType> </xsd:element> With the current semantics this would fail because a global context value can only be set once. Setting in the local context would not work because COUNT would not be in the context of the repString element. A resolution to this issue was agreed involving introducing a declaration statement which could declare but not set a variable. When a local context set was used, if the identifier referred to such a declared but unassigned variable it would have the effect of assigning a value to that instead of setting the local context. In this way we could control the depth at which the context value was defined and ensure that it occurred at the right level. 4. Conversions etc. No new material here. We are still waiting for a more fully worked example using some sort of string separator. Hopefully this will be ready for the F2F.

1 0