- dfdl-wg - lists.ogf.org

Document on modelling uncertainty
by Geoff Judd 15 Mar '06

15 Mar '06

Hi, I have attached a document that discusses the support of uncertainty within DFDL. It is an extension of the choices document that was discussed at the last F2F but now includes choices, element wildcards, optionality, substitution groups and late binding of DFDL Schemas. There is an Alternatives Considered section at the end which contains ideas that were considered but rejected. Regards, Geoff Judd Websphere MQ Integrator Development IBM United Kingdom Ltd. Hursley Telephone: +44-1962-818461 E-mail: JUDDG(a)uk.ibm.com

1 0

RE: [dfdl-wg] Telcon tomorrow
by Steve Hanson 09 Mar '06

09 Mar '06

As it currently stands, the following properties apply to separators and there may well be more (given there is Encoding there should probably be byteOrder & BOM properties for Unicode). separator separatorEncoding finalSeparatorCanBeMissing separatorPosition separatorPolicyForAbsentElements Maybe it makes more sense to treat separators like quotes and have separator schemes that the user can define and re-use? Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Westhead, Martin \(Martin\)" <westhead(a)avaya.c To om> Steve Hanson/UK/IBM@IBMGB cc 08/03/2006 16:38 <dfdl-wg(a)ggf.org>, <owner-dfdl-wg(a)ggf.org> Subject RE: [dfdl-wg] Telcon tomorrow That seems sensible. Question: Can you/should you be able to extend sub properties. For example I introduce a new conversion based on the existing separator function that adds a tag prefix to each field...I would like to be able to add (when I introduce this conversion) a new sub-property to separator called "tagPrefix". Martin >-----Original Message----- >From: Steve Hanson [mailto:smh@uk.ibm.com] >Sent: Wednesday, March 08, 2006 2:07 AM >To: Westhead, Martin (Martin) >Cc: dfdl-wg(a)ggf.org; owner-dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] Telcon tomorrow > >Hi Martin > >Additional agenda item. This is something us IBMers discussed prior to the >F2F but I omitted to include it in my foils about properties. > >Several properties have a bunch of other sub-properties associated with >them. For example, separator has separatorEncoding and separatorIgnoreCase; >initiator and terminator likewise have equivalent sub-properties. It was >suggested that if sub-properties needed to be specified they should only be >specifiable on a sub-element, as below. This was to reduce the number of >properties on DFDL elements. > ><dfdl:sequence separator="@" /> > > or > ><dfdl:sequence> > <dfdl:separator encoding="850" ignoreCase="true">@</dfdl:separator> ></dfdl:sequence> > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM United Kingdom Ltd, Hursley, UK >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > "Westhead, Martin > (Martin)" > <westhead(a)avaya.c To > om> <dfdl-wg(a)ggf.org> > Sent by: cc > owner-dfdl-wg@ggf > .org Subject > [dfdl-wg] Telcon tomorrow > > 07/03/2006 22:03 > > > > > > > > >Preliminary Agenda >------------------ > > 1. Review outcome of F2F meeting > 2. Planning for next GGF > > >Conference call instructions: >Number: 1-800-664-0771 >At the automated answering system, press 0 for the operator. >Ask for the "Daffodil working group" call. > >Web Conference: https://webconferencing.avaya.com >Web Conference: Part. Code 785343; WebConf Reference 158672 >

1 0

Notes from Telcon 2006-3-8
by Westhead, Martin (Martin) 08 Mar '06

08 Mar '06

Present Martin Westhead Robert McGrath Steve Hanson Geoff Judd Tara Talbott Choices Geoff is working on the choices document. He has suggestions about how Schematron might be incorporated. We may see something before next week's session Separator We discussed some of the issues with the separator logic. Fundamentally we have formats that cannot be generally handled using the simplistic pipeline model that we have. In order to maintain modularity we are considering if there is a way to specify the separation semantics in a way orthogonal to the pipeline (i.e. bolted on the side). GGF It appears that no one from the main group of participants will be attending the next GGF in Tokyo. Since we were not present at the last GGF we may need to do something. A possibility would be to have someone we know (e.g. OGSA-DAI team) host the meeting and call in to give a remote presentation. (BTW - I may not be able to make next weeks call)

1 0

Telcon tomorrow
by Westhead, Martin (Martin) 08 Mar '06

08 Mar '06

Preliminary Agenda ------------------ 1. Review outcome of F2F meeting 2. Planning for next GGF Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

2 1

Notes from Face to Face day 2 and 3
by Westhead, Martin (Martin) 03 Mar '06

03 Mar '06

Present Martin Westhead Steve Hanson Suman Kalia Geoff Judd Actions The following actions were assigned: * Scoping and context [guards and tests] (Mike) * Properties and conversions (Steve) * Revisit conversions [Steve's questions, type matching, position, speculative parsing, where are we?](Martin) * Uncertainty [wildcards, substitution groups, choices, optionality] (Geoff) * Defaults and nulls (Geoff) * DFDL Schema (Suman) Open issues: * syntax (sequences, markup) * identifying core properties and conversions (if any) * round tripping * embedded XML Day 2 Notes Properties/conversions/grouping - Steve/Suman (Suman took notes on this topic) Steve will be re-writing the current document to take on board proposals from this meeting and to start the process (with Martin) of trying to converge the properties with the conversions material. Choices and wildcards - Geoff We explored several ideas around the framework that Geoff has. Geoff will be writing these up. Comments included: * Possible use of binhex to represent "deferred" choices * Modeled uncertainty with "any" should be treated as a choice * Substitution groups should be treated as a choice * Wildcards will be handled here and "open" wildcards (unmodeled uncertainty) could be handled by dropping in a string/binhex * Schematron-like assertions could be used as choice guards such assertions could also make general statements about data validity * Use of XML Schema validation to resolve speculative parsing raises issues. (Either all DFDL parsers must validate or would get different results) Which XML Schema constructs are supported - Steve We reviewed Steves slides made various modifications key decisions: * Attributes - wait and see Final documents We had an inconclusive discussion on the structure of the final documents. Day 3 Notes Instance Validation - Efficiency of XML Schema validation - Implementation difficult of XML Schema validation - Therefore cannot force the use of full DFDL parsing - so we cannot use XML Schema validation to resolve speculative parsing - property to state validation to be used could allow parser to produce a warning error if not available - possibility: could use require just facet/value checks (not structure) - XPath evaluation required - Therefore XPath assertions for choice discrimination possible - And XPath assertions for describing valid data possible - EDI community has strong requirement for such assertions to be available (such as coexistance rules) - Is there a requirement for content validation (since construction is driven by model) - Might be useful to have standard names for classes of error DFDL Schema Validation - XML Schema valid - Syntax and semantics of annotations (annotations must be consistent with schema) - XPath references are valid (standard should describe ways validation can be done and features that can be validated, validation should not be mandated).

1 0

Phone number for F2f
by Westhead, Martin (Martin) 02 Mar '06

02 Mar '06

Sorry I forgot to send out a reminder of the F2F call in numbers it is: 1-866-AVAYA-22 Part. Security Code 785343 Web coference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> WebConf Reference 158672 Part. Security Code 785343 Please let us know if you dial in I'll keep the bridge open all day, but email me if there any issues. Thanks, Martin

1 0

notes from F2F 2006-03-01
by Mike Beckerle 02 Mar '06

02 Mar '06

Attendees: Bob McGrath - NCSA Martin Westhead - Avaya Suman Kalia - IBM Geoff Judd - IBM Steve Hanson - IBM Mike Beckerle - IBM Agenda Review Context doc discussion - Martin Issues list: - write once context variables - expressiveness limit? - list valued context variables - define them then 'commit' to make them read-only? - default values for context variables? - breaks write-once behavior? - hidden: don't want paths to hidden peers to be any different from paths to non-hidden peers - mikeb - hidden: need to specify the order of traversal for parsing - hidden: on output, traversal order is very different - need inputCalc and outputCalc - hidden: currently says hidden can only appear where elements could appear - exception needed - simple types need hidden complex sub-structure - 'repDef' - maybe use hidden attributes and avoid element substructure for simple types. - or have an explicit repDef construct with a full indirection from a simple type to a complexType that is its representation - Roundtrip problem is not general data transformation - Separate tracker for round-trip to data issue - Copyright slugline example is sufficient Bundles: - is this just really for the top-level statements like conversion declarations? - revisit this once we've overviewed the scoping stuff - probably some features of one or the other are redundant. - bundles are like xs:include except can be used not-only at top-level. Conversions: - are an expository tool - organize properties into libraries - plug in conversions are allowed, but built-in property/conversions don't have to be implemented using the plug-in mechanism - ? DFDL systems might be allowed to not even implement plug-ins - need to explain XML all the way down idea more clearly. Stream of xs:byte, then streams of higher types. - test guard should select the conversion - once selected then requirements - if you can't find any conversion for something then the schema is invalid. - if the guard of a conversion is satisfied, but required properties are not present in the context, then the schema is invalid. - issue: can more than one conversion have the same name? - issue: what are the update rules for the conversion list in the context as you extend the context with new conversions - issue: type matching - match exactly? or ignore restrictions? E.g., string of length 1 matches general string, but general string doesn't match string of length 1. - issue: type A has child derived type B. Conversions for type B cannot be selected based on existance of conversions that have source/target type A. (isA relationship not involved in conversion selection). The type matching has only to do with restrictions actually. - issue: can different dfdl implementations use different search algorithms? Probably not. They must choose the same set of conversions so the standard can specify the selection algorithm exactly. Implementations are free to use different more efficient algorithms so long as they choose equivalent conversion chains. (need notion of equivalent conversion defined) - issue: context needs to contain the input-stream type (in the "C" context, or just in the informal context of the parser? E.g., you can ask what is the type of this element without that having to be in the "C" context. The type of the source stream could also be in this informal context. - property names should be namespace qualified when used in guard and required formulas of conversions. - conversions stuff doesn't cover all properties. E.g., choice discriminators aren't described how they use as part of conversions. - issue: alignment - can there be an alignment conversion Extensibility: - defineConversion - should express required properties test as well as input type and output type. - whitebox conversions - doubleInt example doesn't seem intuitive to some. suggestion is to name the complex type, separate the named conversion definition and just reference the type name - change keyword for guards to 'guard' not test, since test is used for statement conditional execution. - name conflicts: guard vs. selector vs. test - define terms specifically. - need tracker item re: output - steve hanson suggests using EDI security segments for the useStream example. - guard optional - if not specified means guard="true" Scoping: - simplify: don't allow base format on construct-specific annoation elements if inside dfdl:default or dfdl:defineFormat. variable defaults in the context: - if a variable has a default value, and you keep track of whether anyone has received the default value if they have received the default value then an attempt to set it is an error. This is consistent with single assignment behavior. variable declaration scope - a dfdl:declare on an xsd:element with maxOccurs > 1 is about the type, not the particle (not about the entire repeating element vector, but about each element within). external control of format: - should do overrides much like element references, i.e., external controls have override behavior. leaf contexts: - ? do these distinguish properties from variables Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA 01581 voice and FAX 508-599-7148 home/mobile office 508-915-4795

1 0

RE: [dfdl-wg] CSV string worked example
by Jim Myers 01 Mar '06

01 Mar '06

> >EOS is up for grabs I was thinking of it as a returned value (e.g. >-1) but an exception might (or might not) be easier to make sense of. Assuming -1 is a valid value and not used as a real value, etc. - I think we need a separate mechanism to indicate the end > >Regarding the new model. I don't think this is a problem at the >level of your example. We could simply use a single sequence and a >more complex "split" conversion. I imagine that the "split" >conversion we would want to settle on should accept a regular >expression (or at least a list of separators). In your example you >just have to allow the separator to be a new line OR a comma and you are done. Yes but - I want to reuse the simple conversions and therefore keep the ability to deal with all the variations you note below without special constructs - if I consider this a sequence of two steps I can deal with missing separators and terminators differently, if I put the two together, I have to account for all the possible variations. And I want to handle a data cube the same way without having to wait for someone to build a new converter...(we talked this through generally before - if you have a way to create new converters from existing ones, you could support both ways). > >A note here this is intended as a rough sketch not a finished >design. I am expecting the details to need to be worked out here. In >particular I think Mike/IBM have some fairly complex ideas for >separator/terminator/initiator/escape that we will have to try to >seat in this framework. > >Thanks, > >Martin > > >---------- >From: Jim Myers [mailto:jimmyers@ncsa.uiuc.edu] >Sent: Wednesday, March 01, 2006 3:49 AM >To: Westhead, Martin (Martin); dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] CSV string worked example > >Martin - two types of comments - things I think are >typos/inconsistencies and an alternate logic: > >Clarifications: >are the initial definitions on the top element defining an order to >use subsequently or are they just there for us to see what you've defined? >Of the four there, you only explicitly (in a comment?) invoke one - >are the others implicit because of the order? >You use dfdl:tokenizer as a conversion later - is that supposed to >be split as well? >bytetochar is used implicitly before the first split? >chartostring is used implicitly before stringtoint which is >implicitly used to get the int element? >is EOS a returned value (and therefore of the type being returned) >or is it an exception? > >Logical - what happens if the rows are not in the logical model - >physically there are 10 rows with 5 elements, but the logical model >is 50 ints in a single sequence. To support this, you'd need to have >both tokenization steps in one sequence annotation with two separate >split separators - does the use of setLocal for split separator work >in this case? (Is this how byteorder is now used?) >Thinking about missing values - is it clear how a missing row versus >a missing element is now handled (I think so) - the split conversion >using comma can define a default input to use if the stream it >recieves is empty (from a \n\n pair) and the stringtoint conversion >can do likewise to cover a ,, pair. > > Jim > > >At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: > >Hi Folks, > >I have tried to work through the CSV example that Mike suggested a >couple of weeks ago. It has turned up some interesting issues which >I have tried to address. These are less about making the underlying >semantics work and more about providing a seamless default set up >that makes the easy things work just as you would like. > >I was pushed for time on this so I apologies if this is unclear in >places, but I wanted to put it out before tomorrow's meeting. > >Thanks, > >Martin > >James D. Myers >Associate Director, Cyberenvironments and Technologies, NCSA >1205 W. Clark St, MC-257 >Urbana, IL 61801 >217-244-1934 >jimmyers(a)ncsa.uiuc.edu James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

RE: [dfdl-wg] CSV string worked example
by Westhead, Martin (Martin) 01 Mar '06

01 Mar '06

Hi Jim, I think the others should be implicitly used because of order and type. Sorry tokenizer should be split - unpropagated change. Chartostring (which I called concatenate) is to be used first (because it is the more specific match). EOS is up for grabs I was thinking of it as a returned value (e.g. -1) but an exception might (or might not) be easier to make sense of. Regarding the new model. I don't think this is a problem at the level of your example. We could simply use a single sequence and a more complex "split" conversion. I imagine that the "split" conversion we would want to settle on should accept a regular expression (or at least a list of separators). In your example you just have to allow the separator to be a new line OR a comma and you are done. A note here this is intended as a rough sketch not a finished design. I am expecting the details to need to be worked out here. In particular I think Mike/IBM have some fairly complex ideas for separator/terminator/initiator/escape that we will have to try to seat in this framework. Thanks, Martin _____ From: Jim Myers [mailto:jimmyers@ncsa.uiuc.edu] Sent: Wednesday, March 01, 2006 3:49 AM To: Westhead, Martin (Martin); dfdl-wg(a)ggf.org Subject: Re: [dfdl-wg] CSV string worked example Martin - two types of comments - things I think are typos/inconsistencies and an alternate logic: Clarifications: are the initial definitions on the top element defining an order to use subsequently or are they just there for us to see what you've defined? Of the four there, you only explicitly (in a comment?) invoke one - are the others implicit because of the order? You use dfdl:tokenizer as a conversion later - is that supposed to be split as well? bytetochar is used implicitly before the first split? chartostring is used implicitly before stringtoint which is implicitly used to get the int element? is EOS a returned value (and therefore of the type being returned) or is it an exception? Logical - what happens if the rows are not in the logical model - physically there are 10 rows with 5 elements, but the logical model is 50 ints in a single sequence. To support this, you'd need to have both tokenization steps in one sequence annotation with two separate split separators - does the use of setLocal for split separator work in this case? (Is this how byteorder is now used?) Thinking about missing values - is it clear how a missing row versus a missing element is now handled (I think so) - the split conversion using comma can define a default input to use if the stream it recieves is empty (from a \n\n pair) and the stringtoint conversion can do likewise to cover a ,, pair. Jim At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: Hi Folks, I have tried to work through the CSV example that Mike suggested a couple of weeks ago. It has turned up some interesting issues which I have tried to address. These are less about making the underlying semantics work and more about providing a seamless default set up that makes the easy things work just as you would like. I was pushed for time on this so I apologies if this is unclear in places, but I wanted to put it out before tomorrow's meeting. Thanks, Martin James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0

Re: [dfdl-wg] CSV string worked example
by Jim Myers 01 Mar '06

01 Mar '06

Martin - two types of comments - things I think are typos/inconsistencies and an alternate logic: Clarifications: are the initial definitions on the top element defining an order to use subsequently or are they just there for us to see what you've defined? Of the four there, you only explicitly (in a comment?) invoke one - are the others implicit because of the order? You use dfdl:tokenizer as a conversion later - is that supposed to be split as well? bytetochar is used implicitly before the first split? chartostring is used implicitly before stringtoint which is implicitly used to get the int element? is EOS a returned value (and therefore of the type being returned) or is it an exception? Logical - what happens if the rows are not in the logical model - physically there are 10 rows with 5 elements, but the logical model is 50 ints in a single sequence. To support this, you'd need to have both tokenization steps in one sequence annotation with two separate split separators - does the use of setLocal for split separator work in this case? (Is this how byteorder is now used?) Thinking about missing values - is it clear how a missing row versus a missing element is now handled (I think so) - the split conversion using comma can define a default input to use if the stream it recieves is empty (from a \n\n pair) and the stringtoint conversion can do likewise to cover a ,, pair. Jim At 09:25 PM 2/28/2006, Westhead, Martin (Martin) wrote: >Hi Folks, > >I have tried to work through the CSV example that Mike suggested a >couple of weeks ago. It has turned up some interesting issues which >I have tried to address. These are less about making the underlying >semantics work and more about providing a seamless default set up >that makes the easy things work just as you would like. > >I was pushed for time on this so I apologies if this is unclear in >places, but I wanted to put it out before tomorrow's meeting. > >Thanks, > >Martin James D. Myers Associate Director, Cyberenvironments and Technologies, NCSA 1205 W. Clark St, MC-257 Urbana, IL 61801 217-244-1934 jimmyers(a)ncsa.uiuc.edu

1 0