February 2006 - dfdl-wg - lists.ogf.org

DFDL properties - points to discuss at F2F
by Steve Hanson 26 Feb '06

26 Feb '06

I've been progressing the DFDL Properties tracker and several issues have come up that need resolving before I can suggest a complete list of DFDL properties. Please have a think about these and we can discuss at the F2F. 1) When parsing, we want to be flexible and tolerate data appearing different in forms. For example, allow initiators and terminators to be in upper or lower case, allow UTC timezone to be +00:00 or Z, allow thousand separators for decimal numbers, etc. But on output we must make a choice as to which form we use. We need some principles that we apply to all properties that exhibit this behaviour. 2) It looks like there are real-life examples of initiators and terminators being in a different character encoding to the accompanying data. Implies use of initiatorEncoding etc properties. Need to agree that this is the correct way to proceed. 3) Property precedence and the use of 'notApplicable' enums to say that a property is not to be interpreted on this object. Without this it is not possible to override a dfdl:default setting from higher up the scope. For example, how do I switch off the use of separator for a group when a dfdl:default has set it higher up. A value of notApplicable would solve that. Another example, I have a sequence where variable length fields have a terminator but fixed length fields do not. My terminator and most of my lengths are common, so I set both length and terminator at the group level using dfdl:default. When property scoping rules are applied, I will have a length and a terminator for each field, which do I use? I can't just say length takes precedence because every field has been given a length. I need a way of explicitly saying which to use. One approach is to use a special value for length - notApplicable' - which means ignore the property, and I would use that explicitly on all the variable length fields. Alternatively I can be more explicit and have a separate property to control which is used. 4) Use of 'Native' enum meaning use the locally defined value. Used for character encodings, time zones, endian-ness. The DFDL spec draft says: "There should be no ‘platform varying defaults’. For example, byteOrder should default to bigEndian, or littleEndian, or have no default at all (in which case leaving it unspecified will often cause an error except in all-text situations with non-endian character sets). What’s not acceptable is for byteOrder to default to some value based on the current platform or other environmental constraint. Similarly for locale-sensitive things". Does this imply that 'Native' can't be the default? I can see why we would not want something to default silently based on platform/locale, but by being clear that the default is 'Native' avoids that. 5) Agree on what offset facilities are to be offered for establishing the position of an item. Absolute and/or relative. Problem with absolute offsets is their fragility, a single variable length field breaks the scheme unless the offset is given by an expression. Are relative offsets all we really need? And what are they relative to? Last field? 6) Binary versus text. At the moment the Schema for DFDL has the text model inheriting from the binary model. So text isn't really text, it's text or binary. I think this is wrong and that repType=text and repType=binary really imply separate semantics. It still means properties can be shared, but it clarifies things like the behavior of built-in prefixed lengths - a length prefix for a binary string would be a physical integer in the data, a length prefix for a text string would be a physical numeric string in the data. 7) Padding character - does setting this property imply trim on parse as well as pad on output? Do we need a control for this (pad/trim/both) 8) Justification, date/time format, and some other properties have a default that varies depending on the logical type of the element. How does this default interact with setting a default via scoping rules? We have said that there are no hard-coded defaults, implying a dfdl:defineFormat block must exist and must define values for all properties. This contradicts deriving the default from the logical type. For such properties, we could have an accompanying dfdl:defineFormat-only property that says how it defaults. Or we could exempt these from having to have a default set in dfdl:defineFormat. 9) There are many properties that are inherited from the CAM binary model. Suman is establishing with IBM compiler people exactly what ones are needed and what their semantics are. I find some of the names unhelpful and suggest that renaming such properties is a good idea. 10) Decimal properties. There are a host of properties for controlling decimal formats, it's not clear to me what some of them mean. I'd like us to agree on a finished set. 11) Group level properties. We don't have many properties that define how the members of a group behave. We have separator for instance that says all child fields will be separated. Is there benefit in having other properties like this, for example 'initiated' meaning all child fields must have a unique initiator, or 'fixed' meaning all child fields must have a pre-computable length. Such properties have a secondary benefits - they communicate the nature of the group without recourse to examination of all children, and they enable DFDL editors to validate that all intended properties are present. Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

F2F attendance
by Westhead, Martin (Martin) 22 Feb '06

22 Feb '06

(Dates: 1-3rd March 8.30am start on Wed morning.) Hi Folks, I need to give our security a list of people coming. If you are not on this list and planning to attend the f2f next week (in person) please contact me. Mike Beckerle Geoff Judd Steve Hanson Suman Kalia Robert McGrath Vegetarians - I am arranging some lunch to be provided. If you want me to get vegetarian (or other dietary need) food please let me know. FYI - there is a cafeteria very close to our meeting room it is open in the morning for coffee and pastries, fruit salad etc. I am also planning to host a meal for Wed night at a local Tai restaurant please let me know if you do not like Tai food.

1 0

Telcon notes 2006-2-22
by Westhead, Martin (Martin) 22 Feb '06

22 Feb '06

Present Martin Westhead Tara Talbott Suman Kalia Bob McGrath Geoff Judd Steve Hanson Jim Myers 1. F2F Logistics and Agenda Logistics: aim for an 8.30 am start on March 1st. Conference phone dialin: 1- 866-AVAYA-22 Participant Code: 785343 WebConf Reference: 158672 Web conf url: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> (Note that the participant code for the phone and web conference is the same.) 158672 The discussion about variables and the context seems to be largely resolved, otherwise the Agenda 2. Reports from trackers Some progress is being made on several trackers (scoping, defaults, choices...etc.) for presentation at the F2F. 3. Variables and Context We reviewed a discussion between Martin and Geoff over the semantics of variables and the context. Geoff had an important requirement that could not be handled by contexts illustrated by the following example: <xsd:element name="outerElem"> <xsd:complexType> <xsd:sequence> <xsd:element name="innerElem" maxOccurs="10"> <xsd:complexType> <xsd:sequence> <xsd:element name="numberOfEntries" type="xsd:int" dfdl:setGlobal="COUNT"/> <xsd:element name="repString" type="xsd:string" minOccurs="0" maxOccurs="unbounded" dfdl:repeatCount="$COUNT"/> </xsd:sequence> <xsd:complexType> </xsd:sequence> </xsd:complexType> </xsd:element> With the current semantics this would fail because a global context value can only be set once. Setting in the local context would not work because COUNT would not be in the context of the repString element. A resolution to this issue was agreed involving introducing a declaration statement which could declare but not set a variable. When a local context set was used, if the identifier referred to such a declared but unassigned variable it would have the effect of assigning a value to that instead of setting the local context. In this way we could control the depth at which the context value was defined and ensure that it occurred at the right level. 4. Conversions etc. No new material here. We are still waiting for a more fully worked example using some sort of string separator. Hopefully this will be ready for the F2F.

1 0

Telcon tomorrow
by Westhead, Martin (Martin) 22 Feb '06

22 Feb '06

Please note: I have to get into the office for the call this week so I may be late (stuck in 101 traffic). If so please start without me. - Martin Preliminary Agenda ------------------ 1. F2F Logistics and Agenda (see attached from last week's notes) 2. Reports from trackers 3. Variables and Context 4. Conversions etc. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672 F2F Agenda DFDL underlying mechanisms - Martin Conversions/context/hidden elements - Martin - Relationship to variables - Geoff/Martin Scoping document - Mike Properties/conversions/grouping - Steve/Suman Choices and wildcards - Geoff Null and default values - Geoff Validation Actions to lead to a final document set. - Martin - Which XML Schema constructs are supported - Steve - Review of trackers - All Validation * Validation of the DFDL schema (consistent properties) * Validation of the DFDL document * Choices based on failure

1 0

Notes from telcon 2006-2-15
by Westhead, Martin (Martin) 15 Feb '06

15 Feb '06

Present Martin Westhead Suman Kalia Mike Beckerle Tara Talbott Bob McGrath Geoff Judd F2F Agenda DFDL underlying mechanisms - Martin Conversions/context/hidden elements - Martin - Relationship to variables - Geoff/Martin Scoping document - Mike Properties/conversions/grouping - Steve/Suman Choices and wildcards - Geoff Null and default values - Geoff Validation Actions to lead to a final document set. - Martin - Which XML Schema constructs are supported - Steve - Review of trackers - All Validation * Validation of the DFDL schema (consistent properties) * Validation of the DFDL document * Choices based on failure Discussion We spent the last 15 minutes of the call discussing issues around the "conversions" document. Mike requested an example which showed how separator conversions could be written in this way.

1 0

Telcon agenda for tomorrow
by Westhead, Martin (Martin) 15 Feb '06

15 Feb '06

Preliminary Agenda ------------------ 1. Agenda for F2F 2. Reports from trackers 3. Ongoing review "Conversions" document 4. Review "Bundles" document Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference Reference: 158672 Security Code : 785343

2 1

Telcon notes 2006-02-08
by Westhead, Martin (Martin) 08 Feb '06

08 Feb '06

Present: -------- Martin Westhead Tara Talbott Bob McGrath Geoff Judd Mike Bekerele Discussion ---------- 1. F2F Agenda The following topics/goals were raised for inclusion on the agenda: Conversions - basics: do we have a common understanding? Do we have consensus as this as a direction? Conversions - does the current proposal sufficiently address our extensibility needs (e.g. arrays, other trackers)? Conversions - can we capture the meaning of the existing properties effectively within this framework. Process - arriving at a clear set of milestones/actions with people attached to them required to reach a standards document. 2. Conversions discussion Geoff raised an issue that he was concerned about ordinary users being exposed to a lot of boiler-plate. Although this could be avoided altogether it was agreed that including a standard definitions file might be a sensible way to do it. Tara wanted to know if the whole stack of conversions could be specified. This seems like a sensible suggestion. It should be straight forward to specify all or even part of the conversion stack.

1 0

Updated documents
by Westhead, Martin (Martin) 06 Feb '06

06 Feb '06

Hi folks, I have taken another pass at the conversions material. Here are two documents that came from the document that was "Conversions". The first is still called "Conversions" and focuses entirely on the semantics of conversions and how they are chosen. The second document contains all the material that could be considered "extensions" (so that's what I called it :-) ) The intention here is to separate out the motivational issues. IMO we need the material in the first document in order to explain the operation of a DFDL parser. (I do not claim that this is the right material just that we need an explanation of how this works and this is a proposal to meet that need). The second document contains material that may be regarded differently in terms of implementation cost/benefit by different groups in the WG and we should discuss which bits make sense as required (for all implementations) and which might be optional. I have included new examples in both documents, inspired by Steve's suggestions. I am aware that all of this material needs the details attending to. There are a number of design decisions that I made arbitrarily in order to get a document out to try to communicate the broad ideas. Assuming the ideas are adopted we will need to work through all the details of the syntax etc. Mike brought up a number of such issues in his response to the first draft of this stuff last week. I have not attempted to really answer these yet, I am still working at communicating the ideas to the group as a whole. I did make some minor changes to the syntax largely to try to make the ideas clearer. Please let me know what you think. Thanks, Martin

2 1

RE: [dfdl-wg] More documents
by Westhead, Martin (Martin) 04 Feb '06

04 Feb '06

Wouldn't repType="xml" make more sense? >-----Original Message----- >From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of >Steve Hanson >Sent: Thursday, February 02, 2006 7:06 AM >To: dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] More documents > >I'd forgotten the isXML property - that covers this nicely. > >I'll stick with the remaining two: > >- A portion of the data is encrypted, with fields in the message prior to >the encrypted section providing the decryption keys etc. (X12 security >segment motivates this) >- Data where decimal fields (say) are in a wacky encoding not supported by >stock DFDL properties (TLOG retail standard motivates this) > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM Hursley, England >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > Mike Beckerle > <beckerle(a)us.ibm. > com> To > Steve Hanson/UK/IBM@IBMGB > 02/02/2006 13:41 cc > dfdl-wg(a)ggf.org, > owner-dfdl-wg(a)ggf.org > Subject > Re: [dfdl-wg] More documents > > > > > > > > > > > >I would second this approach. A payload string of XML data is just a string >of value content to us. > >Note however that in our proposed set of properties there is one "isXML" >which is intended to facilitate the usage pattern of XML payload strings. >This property is a boolean you can set to say that the string's content is >a well formed XML document or a well-formed fragment of XML. This is just a >shorthand for what would otherwise be a large set of quoting/escaping >conventions, the use of a dynamic character set selected based on the >encoding attribute in the <?xml version="1.0" encoding="US-ASCII"?> slug >line (if present), etc. > >(We would need to specify what the concept of "well formed fragment of XML" >means. I think intuitively people know what this means, something >intelligible to an XML parser, but we need to be explicit. It means a >fragment of XML that begins and ends between two elements. Hence, is not a >fragment that starts in the middle of any quoting construct, nor in the >middle of a tag or attribute, etc. ) > >Mike Beckerle >STSM, Architect, Scalable Computing >IBM Software Group >Information Integration Solutions >Westborough, MA 01581 >voice and FAX 508-599-7148 >home/mobile office 508-915-4767 > > > > > Steve Hanson <smh(a)uk.ibm.com> > Sent by: owner-dfdl-wg(a)ggf.org > > To > 02/01/2006 01:27 PM dfdl-wg(a)ggf.org > cc > > Subject > Re: [dfdl-wg] More > documents > > > > > > > > > > > > >I'll see what I can come up with. > >As far as the embedded XML goes, I put it there as we will be asked this >question. Thinking it through, maybe we should simply treat it as a BLOB >and leave it to the user to take and parse using an XML parser as an >independent operation. This is symmetric with an XML document containing a >non-XML BLOB as CDATA that needed to be parsed using DFDL. > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM Hursley, England >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > "Robert E. > McGrath" > <mcgrath(a)ncsa.uiu To > c.edu> dfdl-wg(a)ggf.org > Sent by: cc > owner-dfdl-wg@ggf > .org Subject > Re: [dfdl-wg] More documents > > 01/02/2006 15:03 > > > > > > > > >On Wednesday 01 February 2006 07:03, Steve Hanson wrote: > >> - A portion of the data is encrypted, with fields in the message prior to >> the encrypted section providing the decryption keys etc. (X12 security >> segment motivates this) >> - Data where some XML is embedded in the middle >> - Data where decimal fields (say) are in a wacky encoding not supported >by >> stock DFDL properties (TLOG retail standard motivates here) >> > >Thease are great examples. Can someone give me fully documented >data files from which to try to construct such examples? > >By the way, I'm not sure whether embedded XML is within the scope >of DFDL--it gets insanely hairy. > >But the others are exactly the kinds of things that the core standard >must either cover or have an extension mechanism that covers. > > >-- >--- >Robert E. McGrath, Ph.D. >National Center for Supercomputing Applications >University of Illinois, Urbana-Champaign >1205 West Clark >Urbana, Illinois 61801 >(217)-333-6549 > >mcgrath(a)ncsa.uiuc.edu > > > >

1 0

RE: [dfdl-wg] More documents
by Westhead, Martin (Martin) 02 Feb '06

02 Feb '06

Sorry to be a latecomer to this thread. I don't think embedded XML should provide any problems at the description level. I gave it some thought when I was writing the document. There is a very straightforward way to handle it within the set of mechanisms proposed. All you need is a conversion that does bytes -> XML (i.e. an XML parser) you end up with a stream of the root XML type. The conversion system will look at this and compare it to what it is expecting. In the absence of any alternative conversions, and assuming the types match it will apply the "null" conversion and drop the XML in place. I can include this example. Cheers, Martin >-----Original Message----- >From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On Behalf Of >Robert E. McGrath >Sent: Thursday, February 02, 2006 8:14 AM >To: Jim Myers; dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] More documents > >The main thing about XML is that it is very difficult to interpret >an arbitrary fragment within the context of another XML. E.g., >suppose there was a dfdl annotion inside the fragment? What >would that mean? DFDL to describe DFDL? Ick! > >Some kind of BLOB plus interpreter gives the necessary >mechanics. E.g., where conventions exist for such fragments, >standard code can check and dig out the stuff--this would be >published as a DFDL extension. (I think I'm just agreeing with Jim.) > >On Thursday 02 February 2006 09:29, Jim Myers wrote: >> One could certainly imagine a case where there's a text integer in >> the XML that is required for parsing subsequent data... I don't know >> that we have to support this case directly/to start, but I think we >> should consider a use case where Joe User is very excited by the >> power of DFDL to deal with his complex information and has now run >> across a case where info within an XML fragment in his doc is >> required for further parsing, he wants to use DFDL to get the >> fragment, has an XML parser he could connect (assuming a DFDL parser >> that supported extensions), and then needs a way to use one of the >> nodes identified as a source for further DFDL processing (the >> /mydata/length node contains a string representing the length of a >> subsequent array and he needs to get that string, convert to int >> using standard DFDL and use that as an input for the size of the >> array). The use case without extensibility is that Joe waits for DFDL >> 2.0 when we add an XML parser and he'd now like to do the same thing >> using his knowledge of DFDL 1.0 to process the string to int and use >> it as a repeat count for an array. >> >> Jim >> >> At 07:41 AM 2/2/2006, Mike Beckerle wrote: >> >I would second this approach. A payload string of XML data is just a >> >string of value content to us. >> > >> >Note however that in our proposed set of properties there is one >> >"isXML" which is intended to facilitate the usage pattern of XML >> >payload strings. This property is a boolean you can set to say that >> >the string's content is a well formed XML document or a well-formed >> >fragment of XML. This is just a shorthand for what would otherwise >> >be a large set of quoting/escaping conventions, the use of a dynamic >> >character set selected based on the encoding attribute in the <?xml >> >version="1.0" encoding="US-ASCII"?> slug line (if present), etc. >> > >> >(We would need to specify what the concept of "well formed fragment >> >of XML" means. I think intuitively people know what this means, >> >something intelligible to an XML parser, but we need to be explicit. >> >It means a fragment of XML that begins and ends between two >> >elements. Hence, is not a fragment that starts in the middle of any >> >quoting construct, nor in the middle of a tag or attribute, etc. ) >> > >> >Mike Beckerle >> >STSM, Architect, Scalable Computing >> >IBM Software Group >> >Information Integration Solutions >> >Westborough, MA 01581 >> >voice and FAX 508-599-7148 >> >home/mobile office 508-915-4767 >> > >> > >> > >> > >> >Steve Hanson <smh(a)uk.ibm.com> >> >Sent by: owner-dfdl-wg(a)ggf.org >> > >> >02/01/2006 01:27 PM >> >To >> >dfdl-wg(a)ggf.org >> >cc >> >Subject >> >Re: [dfdl-wg] More documents >> > >> > >> > >> > >> >I'll see what I can come up with. >> > >> >As far as the embedded XML goes, I put it there as we will be asked this >> >question. Thinking it through, maybe we should simply treat it as a BLOB >> >and leave it to the user to take and parse using an XML parser as an >> >independent operation. This is symmetric with an XML document containing >a >> >non-XML BLOB as CDATA that needed to be parsed using DFDL. >> > >> >Regards, Steve >> > >> >Steve Hanson >> >WebSphere Message Brokers, >> >IBM Hursley, England >> >Internet: smh(a)uk.ibm.com >> >Phone (+44)/(0) 1962-815848 >> > >> > >> > >> > "Robert E. >> > McGrath" >> > <mcgrath(a)ncsa.uiu >To >> > c.edu> dfdl-wg(a)ggf.org >> > Sent by: >cc >> > owner-dfdl-wg@ggf >> > .org >Subject >> > Re: [dfdl-wg] More documents >> > >> > 01/02/2006 15:03 >> > >> >On Wednesday 01 February 2006 07:03, Steve Hanson wrote: >> > > - A portion of the data is encrypted, with fields in the message >prior >> > > to the encrypted section providing the decryption keys etc. (X12 >> > > security segment motivates this) >> > > - Data where some XML is embedded in the middle >> > > - Data where decimal fields (say) are in a wacky encoding not >supported >> > >> >by >> > >> > > stock DFDL properties (TLOG retail standard motivates here) >> > >> >Thease are great examples. Can someone give me fully documented >> >data files from which to try to construct such examples? >> > >> >By the way, I'm not sure whether embedded XML is within the scope >> >of DFDL--it gets insanely hairy. >> > >> >But the others are exactly the kinds of things that the core standard >> >must either cover or have an extension mechanism that covers. >> > >> > >> >-- >> >--- >> >Robert E. McGrath, Ph.D. >> >National Center for Supercomputing Applications >> >University of Illinois, Urbana-Champaign >> >1205 West Clark >> >Urbana, Illinois 61801 >> >(217)-333-6549 >> > >> >mcgrath(a)ncsa.uiuc.edu >> >> James D. Myers >> Associate Director, Collaborative Systems, NCSA >> 1205 W. Clark St, MC-257 >> Urbana, IL 61801 >> 217-244-1934 >> jimmyers(a)ncsa.uiuc.edu > >-- >--- >Robert E. McGrath, Ph.D. >National Center for Supercomputing Applications >University of Illinois, Urbana-Champaign >1205 West Clark >Urbana, Illinois 61801 >(217)-333-6549 > >mcgrath(a)ncsa.uiuc.edu

1 0