- dfdl-wg - lists.ogf.org

Transformation between Binary data and XML
by Mike Beckerle 20 Mar '06

20 Mar '06

FYI: "Edmund Lai" <edmund.lai(a)gmail.com> 03/20/2006 02:37 AM To Mike Beckerle/Worcester/IBM@IBMUS cc Subject Transformation between Binary data and XML I have done some experiment on transformation between binary data and XML that maybe of interest to DFDL-WG. I am not following any existing spec, but rather I am trying to determine a set of commands that I feel is adequate and easy to learn. Only after I finish would I try to see it I can modify it to fit into any standard. The converter is web based and is in http://www.datamech.com/XMLForm/XMLandBinary.html The description is in http://www.datamech.com/XMLForm/aboutXmlAndBinary.html

1 0

How to deal with variable length elements?
by Robert E. McGrath 17 Mar '06

17 Mar '06

Folks, I'm trying to build up a story about how to handle arrays using DFDL. But first, I need to check if I understand the basics. Here is an example of a 1D array in XML, modeled as two elements, an integer indicating how many elements, followed by an array of zero or more floats. Looking at my XML textbooks, the following seems like the correct XML schema for this notion. +++  <xs: element type="float" name="floatType" \>  <xs:complexType name="floatArray"> <xs:sequence> <xs:element name="nelems type="int" /> <xs:element name="x" ref="floatType" minOccurs="0" maxOccurs="./nelems" /> </xs:sequence> </xs:complexType> +++ Do I have this correct? (I'm pretty sure the 'maxOccurs' isn't correct, so I hope someone will tell me the right way to do this.) If so, the next question will be "how do I annotate this with DFDL?", e.g., when the data is precisely one binary (or text encoded) int, followed by some binary (or text encoeded) floats.

3 5

RE: [dfdl-wg] Telcon tomorrow
by Steve Hanson 15 Mar '06

15 Mar '06

As discussed on the call, I'll issue the first draft of the properties without creating separator/terminator/initiator schemes, and we'll decide when we have evaluated the full property set. Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Steve Hanson/UK/IBM To 09/03/2006 10:16 "Westhead, Martin \(Martin\)" <westhead(a)avaya.com> cc dfdl-wg(a)ggf.org, owner-dfdl-wg(a)ggf.org Subject RE: [dfdl-wg] Telcon tomorrow (Document link: Steve Hanson) As it currently stands, the following properties apply to separators and there may well be more (given there is Encoding there should probably be byteOrder & BOM properties for Unicode). separator separatorEncoding finalSeparatorCanBeMissing separatorPosition separatorPolicyForAbsentElements Maybe it makes more sense to treat separators like quotes and have separator schemes that the user can define and re-use? Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Westhead, Martin \(Martin\)" <westhead(a)avaya.c To om> Steve Hanson/UK/IBM@IBMGB cc 08/03/2006 16:38 <dfdl-wg(a)ggf.org>, <owner-dfdl-wg(a)ggf.org> Subject RE: [dfdl-wg] Telcon tomorrow That seems sensible. Question: Can you/should you be able to extend sub properties. For example I introduce a new conversion based on the existing separator function that adds a tag prefix to each field...I would like to be able to add (when I introduce this conversion) a new sub-property to separator called "tagPrefix". Martin >-----Original Message----- >From: Steve Hanson [mailto:smh@uk.ibm.com] >Sent: Wednesday, March 08, 2006 2:07 AM >To: Westhead, Martin (Martin) >Cc: dfdl-wg(a)ggf.org; owner-dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] Telcon tomorrow > >Hi Martin > >Additional agenda item. This is something us IBMers discussed prior to the >F2F but I omitted to include it in my foils about properties. > >Several properties have a bunch of other sub-properties associated with >them. For example, separator has separatorEncoding and separatorIgnoreCase; >initiator and terminator likewise have equivalent sub-properties. It was >suggested that if sub-properties needed to be specified they should only be >specifiable on a sub-element, as below. This was to reduce the number of >properties on DFDL elements. > ><dfdl:sequence separator="@" /> > > or > ><dfdl:sequence> > <dfdl:separator encoding="850" ignoreCase="true">@</dfdl:separator> ></dfdl:sequence> > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM United Kingdom Ltd, Hursley, UK >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > "Westhead, Martin > (Martin)" > <westhead(a)avaya.c To > om> <dfdl-wg(a)ggf.org> > Sent by: cc > owner-dfdl-wg@ggf > .org Subject > [dfdl-wg] Telcon tomorrow > > 07/03/2006 22:03 > > > > > > > > >Preliminary Agenda >------------------ > > 1. Review outcome of F2F meeting > 2. Planning for next GGF > > >Conference call instructions: >Number: 1-800-664-0771 >At the automated answering system, press 0 for the operator. >Ask for the "Daffodil working group" call. > >Web Conference: https://webconferencing.avaya.com >Web Conference: Part. Code 785343; WebConf Reference 158672 >

1 0

Document on modelling uncertainty
by Geoff Judd 15 Mar '06

15 Mar '06

Hi, I have attached a document that discusses the support of uncertainty within DFDL. It is an extension of the choices document that was discussed at the last F2F but now includes choices, element wildcards, optionality, substitution groups and late binding of DFDL Schemas. There is an Alternatives Considered section at the end which contains ideas that were considered but rejected. Regards, Geoff Judd Websphere MQ Integrator Development IBM United Kingdom Ltd. Hursley Telephone: +44-1962-818461 E-mail: JUDDG(a)uk.ibm.com

1 0

RE: [dfdl-wg] Telcon tomorrow
by Steve Hanson 09 Mar '06

09 Mar '06

As it currently stands, the following properties apply to separators and there may well be more (given there is Encoding there should probably be byteOrder & BOM properties for Unicode). separator separatorEncoding finalSeparatorCanBeMissing separatorPosition separatorPolicyForAbsentElements Maybe it makes more sense to treat separators like quotes and have separator schemes that the user can define and re-use? Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Westhead, Martin \(Martin\)" <westhead(a)avaya.c To om> Steve Hanson/UK/IBM@IBMGB cc 08/03/2006 16:38 <dfdl-wg(a)ggf.org>, <owner-dfdl-wg(a)ggf.org> Subject RE: [dfdl-wg] Telcon tomorrow That seems sensible. Question: Can you/should you be able to extend sub properties. For example I introduce a new conversion based on the existing separator function that adds a tag prefix to each field...I would like to be able to add (when I introduce this conversion) a new sub-property to separator called "tagPrefix". Martin >-----Original Message----- >From: Steve Hanson [mailto:smh@uk.ibm.com] >Sent: Wednesday, March 08, 2006 2:07 AM >To: Westhead, Martin (Martin) >Cc: dfdl-wg(a)ggf.org; owner-dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] Telcon tomorrow > >Hi Martin > >Additional agenda item. This is something us IBMers discussed prior to the >F2F but I omitted to include it in my foils about properties. > >Several properties have a bunch of other sub-properties associated with >them. For example, separator has separatorEncoding and separatorIgnoreCase; >initiator and terminator likewise have equivalent sub-properties. It was >suggested that if sub-properties needed to be specified they should only be >specifiable on a sub-element, as below. This was to reduce the number of >properties on DFDL elements. > ><dfdl:sequence separator="@" /> > > or > ><dfdl:sequence> > <dfdl:separator encoding="850" ignoreCase="true">@</dfdl:separator> ></dfdl:sequence> > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM United Kingdom Ltd, Hursley, UK >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > "Westhead, Martin > (Martin)" > <westhead(a)avaya.c To > om> <dfdl-wg(a)ggf.org> > Sent by: cc > owner-dfdl-wg@ggf > .org Subject > [dfdl-wg] Telcon tomorrow > > 07/03/2006 22:03 > > > > > > > > >Preliminary Agenda >------------------ > > 1. Review outcome of F2F meeting > 2. Planning for next GGF > > >Conference call instructions: >Number: 1-800-664-0771 >At the automated answering system, press 0 for the operator. >Ask for the "Daffodil working group" call. > >Web Conference: https://webconferencing.avaya.com >Web Conference: Part. Code 785343; WebConf Reference 158672 >

1 0

Notes from Telcon 2006-3-8
by Westhead, Martin (Martin) 08 Mar '06

08 Mar '06

Present Martin Westhead Robert McGrath Steve Hanson Geoff Judd Tara Talbott Choices Geoff is working on the choices document. He has suggestions about how Schematron might be incorporated. We may see something before next week's session Separator We discussed some of the issues with the separator logic. Fundamentally we have formats that cannot be generally handled using the simplistic pipeline model that we have. In order to maintain modularity we are considering if there is a way to specify the separation semantics in a way orthogonal to the pipeline (i.e. bolted on the side). GGF It appears that no one from the main group of participants will be attending the next GGF in Tokyo. Since we were not present at the last GGF we may need to do something. A possibility would be to have someone we know (e.g. OGSA-DAI team) host the meeting and call in to give a remote presentation. (BTW - I may not be able to make next weeks call)

1 0

Telcon tomorrow
by Westhead, Martin (Martin) 08 Mar '06

08 Mar '06

Preliminary Agenda ------------------ 1. Review outcome of F2F meeting 2. Planning for next GGF Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

2 1

Notes from Face to Face day 2 and 3
by Westhead, Martin (Martin) 03 Mar '06

03 Mar '06

Present Martin Westhead Steve Hanson Suman Kalia Geoff Judd Actions The following actions were assigned: * Scoping and context [guards and tests] (Mike) * Properties and conversions (Steve) * Revisit conversions [Steve's questions, type matching, position, speculative parsing, where are we?](Martin) * Uncertainty [wildcards, substitution groups, choices, optionality] (Geoff) * Defaults and nulls (Geoff) * DFDL Schema (Suman) Open issues: * syntax (sequences, markup) * identifying core properties and conversions (if any) * round tripping * embedded XML Day 2 Notes Properties/conversions/grouping - Steve/Suman (Suman took notes on this topic) Steve will be re-writing the current document to take on board proposals from this meeting and to start the process (with Martin) of trying to converge the properties with the conversions material. Choices and wildcards - Geoff We explored several ideas around the framework that Geoff has. Geoff will be writing these up. Comments included: * Possible use of binhex to represent "deferred" choices * Modeled uncertainty with "any" should be treated as a choice * Substitution groups should be treated as a choice * Wildcards will be handled here and "open" wildcards (unmodeled uncertainty) could be handled by dropping in a string/binhex * Schematron-like assertions could be used as choice guards such assertions could also make general statements about data validity * Use of XML Schema validation to resolve speculative parsing raises issues. (Either all DFDL parsers must validate or would get different results) Which XML Schema constructs are supported - Steve We reviewed Steves slides made various modifications key decisions: * Attributes - wait and see Final documents We had an inconclusive discussion on the structure of the final documents. Day 3 Notes Instance Validation - Efficiency of XML Schema validation - Implementation difficult of XML Schema validation - Therefore cannot force the use of full DFDL parsing - so we cannot use XML Schema validation to resolve speculative parsing - property to state validation to be used could allow parser to produce a warning error if not available - possibility: could use require just facet/value checks (not structure) - XPath evaluation required - Therefore XPath assertions for choice discrimination possible - And XPath assertions for describing valid data possible - EDI community has strong requirement for such assertions to be available (such as coexistance rules) - Is there a requirement for content validation (since construction is driven by model) - Might be useful to have standard names for classes of error DFDL Schema Validation - XML Schema valid - Syntax and semantics of annotations (annotations must be consistent with schema) - XPath references are valid (standard should describe ways validation can be done and features that can be validated, validation should not be mandated).

1 0

Phone number for F2f
by Westhead, Martin (Martin) 02 Mar '06

02 Mar '06

Sorry I forgot to send out a reminder of the F2F call in numbers it is: 1-866-AVAYA-22 Part. Security Code 785343 Web coference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> WebConf Reference 158672 Part. Security Code 785343 Please let us know if you dial in I'll keep the bridge open all day, but email me if there any issues. Thanks, Martin

1 0

notes from F2F 2006-03-01
by Mike Beckerle 02 Mar '06

02 Mar '06

Attendees: Bob McGrath - NCSA Martin Westhead - Avaya Suman Kalia - IBM Geoff Judd - IBM Steve Hanson - IBM Mike Beckerle - IBM Agenda Review Context doc discussion - Martin Issues list: - write once context variables - expressiveness limit? - list valued context variables - define them then 'commit' to make them read-only? - default values for context variables? - breaks write-once behavior? - hidden: don't want paths to hidden peers to be any different from paths to non-hidden peers - mikeb - hidden: need to specify the order of traversal for parsing - hidden: on output, traversal order is very different - need inputCalc and outputCalc - hidden: currently says hidden can only appear where elements could appear - exception needed - simple types need hidden complex sub-structure - 'repDef' - maybe use hidden attributes and avoid element substructure for simple types. - or have an explicit repDef construct with a full indirection from a simple type to a complexType that is its representation - Roundtrip problem is not general data transformation - Separate tracker for round-trip to data issue - Copyright slugline example is sufficient Bundles: - is this just really for the top-level statements like conversion declarations? - revisit this once we've overviewed the scoping stuff - probably some features of one or the other are redundant. - bundles are like xs:include except can be used not-only at top-level. Conversions: - are an expository tool - organize properties into libraries - plug in conversions are allowed, but built-in property/conversions don't have to be implemented using the plug-in mechanism - ? DFDL systems might be allowed to not even implement plug-ins - need to explain XML all the way down idea more clearly. Stream of xs:byte, then streams of higher types. - test guard should select the conversion - once selected then requirements - if you can't find any conversion for something then the schema is invalid. - if the guard of a conversion is satisfied, but required properties are not present in the context, then the schema is invalid. - issue: can more than one conversion have the same name? - issue: what are the update rules for the conversion list in the context as you extend the context with new conversions - issue: type matching - match exactly? or ignore restrictions? E.g., string of length 1 matches general string, but general string doesn't match string of length 1. - issue: type A has child derived type B. Conversions for type B cannot be selected based on existance of conversions that have source/target type A. (isA relationship not involved in conversion selection). The type matching has only to do with restrictions actually. - issue: can different dfdl implementations use different search algorithms? Probably not. They must choose the same set of conversions so the standard can specify the selection algorithm exactly. Implementations are free to use different more efficient algorithms so long as they choose equivalent conversion chains. (need notion of equivalent conversion defined) - issue: context needs to contain the input-stream type (in the "C" context, or just in the informal context of the parser? E.g., you can ask what is the type of this element without that having to be in the "C" context. The type of the source stream could also be in this informal context. - property names should be namespace qualified when used in guard and required formulas of conversions. - conversions stuff doesn't cover all properties. E.g., choice discriminators aren't described how they use as part of conversions. - issue: alignment - can there be an alignment conversion Extensibility: - defineConversion - should express required properties test as well as input type and output type. - whitebox conversions - doubleInt example doesn't seem intuitive to some. suggestion is to name the complex type, separate the named conversion definition and just reference the type name - change keyword for guards to 'guard' not test, since test is used for statement conditional execution. - name conflicts: guard vs. selector vs. test - define terms specifically. - need tracker item re: output - steve hanson suggests using EDI security segments for the useStream example. - guard optional - if not specified means guard="true" Scoping: - simplify: don't allow base format on construct-specific annoation elements if inside dfdl:default or dfdl:defineFormat. variable defaults in the context: - if a variable has a default value, and you keep track of whether anyone has received the default value if they have received the default value then an attempt to set it is an error. This is consistent with single assignment behavior. variable declaration scope - a dfdl:declare on an xsd:element with maxOccurs > 1 is about the type, not the particle (not about the entire repeating element vector, but about each element within). external control of format: - should do overrides much like element references, i.e., external controls have override behavior. leaf contexts: - ? do these distinguish properties from variables Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA 01581 voice and FAX 508-599-7148 home/mobile office 508-915-4795

1 0