- dfdl-wg - lists.ogf.org

Notes from Telcon 2006-04-06
by Westhead, Martin (Martin) 05 Apr '06

05 Apr '06

Present Martin Westhead Tara Talbott Mike Beckerle Steven Hanson Geoff Judd Properties document Some things that came out of the discussion (mainly from Mike who took better notes than me) - clarification on Opaque vs. binary (details below) - Generally the naming scheme looks ok, even the very long names like "syntaxFinalSeparatorCanBeMissing" are ok. - Adding a conceptual UML taxonomy of the various represetnation kinds would help to motivate the naming conventions used for the properties. - Want to break up big enums like repType and make them extensible. So that a library can add values to it. - Proposal: repType is a QName, not an enum, and a DFDL processor adds libraries that register the valid strings for it as part of their definitions. This allows refactoring the properties list into libraries. - Issues with library fragmentation: how small a subset of the libraries can you have and still call it DFDL? Discussion: not too small because you won't get interoperability. However, one can have data that uses only a subset of DFDL features, and then you can have software that only implements what you need. Your data and schemas will be DFDL conformant in that anyone with a conforming DFDL implementation will be able to read your data. However, you will not have software that can process data you receive from others which uses more features. This is an ok situation: conforming data and schemas, but subset implementation. - It would be good to be able to formally describe the subset of DFDL constructs that a subset implementation provides. E.g., an XML file which characterizes the subset formally. - Action for all to look at the paragraph in Steve's email about default values. Binary/Opaque issue: >From last week's notes: 1. Keep the group Binary. It covers situations where a binary representation in the logical model is either what we really want or is simply the best we can do. 2. Add the reptype 'text' to the group Binary. This is to cover text representations of binary data. i.e. we want to interpret the bytes on the disk as a text representation of binary data. [Question: what if the representation is base64 but we would like the logical model to represent it as xs:hexBinary? What if it is uuencoded?] 3. Add a new Group called Opaque. This would have the logical type xs:any and the representations 'text' or 'binaryStream'. A text representation means that this is an opaque string and it should be included in the logical model as CDATA. A binary representation means that it is an opaque Blob and it should be included in the logical model using a binary representation. We can choose any binary representation let us say hexBinary. Clarifications: an element of type hexBinary or base64Binary would behave, at an API level as both a string (can deliver hex characters in the case of hexBinary) and as a vector of bytes. An element of type Opaque would have no operations at an API level other than copy(), and size(). Could also add as a representation kind for binary data: uuencode, base100Binary, in addition to base64 and hex. Items for next week - issue of defaults - arrays discussion - outstanding property document discussion items

1 0

Telcon tomorrow
by Westhead, Martin (Martin) 04 Apr '06

04 Apr '06

Preliminary Agenda 1. Continue last weeks discussion on Steve's document on Properties. In particular consider the Opaque/Binary data proposal from last week's notes. 2. Arrays - comments on length calculation in recent email from Mike. Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

1 0

Telcon Notes
by Westhead, Martin (Martin) 29 Mar '06

29 Mar '06

Present Geoff Judd Bob McGrath Mike Beckerle Martin Westhead Steve Hanson Uncertainty Document No more comments at this point. Some minor IBM modifications. Properties Document - Need to add physical reps for binHex and base64Encoded - Discussion of opaque data. There was a lengthy discussion of the properties and conversions surrounding opaque and binary data. Martin made a proposal that needs careful consideration, it is as follows: Steve's doc has groups of logical representations and then repType properties which correspond to physical representations of those groups. It currently contains the group Binary which has logical representations xs:hexBinary and xs:base64Binary and the representation type 'binaryStream'. The proposal is as follows: 1. Keep the group Binary. It covers situations where a binary representation in the logical model is either what we really want or is simply the best we can do. 2. Add the reptype 'text' to the group Binary. This is to cover text representations of binary data. i.e. we want to interpret the bytes on the disk as a text representation of binary data. [Question: what if the representation is base64 but we would like the logical model to represent it as xs:hexBinary? What if it is uuencoded?] 3. Add a new Group called Opaque. This would have the logical type xs:any and the representations 'text' or 'binaryStream'. A text representation means that this is an opaque string and it should be included in the logical model as CDATA. A binary representation means that it is an opaque Blob and it should be included in the logical model using a binary representation. We can choose any binary representation let us say hexBinary. Arrays Update - Bob drew attention to a length calculation proposal in Mike's recent emails. Do we like it?

1 0

Fw: [wg-all] New GGF document draft for chairs/editors/authors
by Mike Beckerle 27 Mar '06

27 Mar '06

There is a new document template for GGF documents. I suggest we convert the next round of all our draft memoranda/docs to this format to allow easier merging later. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Integration Solutions Westborough, MA 01581 voice and FAX 508-599-7148 home/mobile office 508-915-4795 ----- Forwarded by Mike Beckerle/Worcester/IBM on 03/27/2006 08:04 AM ----- "Gregory Newby" <newby(a)arsc.edu> Sent by: owner-wg-all(a)ggf.org 03/26/2006 09:00 PM To wg-all(a)ggf.org cc Subject [wg-all] New GGF document draft for chairs/editors/authors This information is mostly of interest to group chairs, but there are others in the GGF community involved with writing documents who might also want to use this template. There is a new GGF document template available for use by authors/editors for the document series. This template has some wording changes we have adopted (such as calling documents "documents" rather than "memos"), some recommendations on the use of the required "security considerations" section, and sample references. It also includes the new copyright and intellectual property statements as published in GFD-CP.063 (this document will be online shortly). Authors should be careful to use these new statements, rather than the old ones. Find the document in the Editor group area on http://forge.gridforum.org under the "Author Guidelines and Templates" documents. https://forge.gridforum.org/projects/ggf-editor Here is a direct, but longer, link: https://forge.gridforum.org/docman2/ViewCategory.php?group_id=90&category_i… Enjoy! -- Greg Newby (GGF Editor)

1 0

Draft of Properties document
by Steve Hanson 23 Mar '06

23 Mar '06

The attached document is the first draft of the revised set of DFDL properties. The following should be borne in mind when reviewing: - I have divided the set of properties in two, those that are concerned with physical representation of the data (rep properties), and those that aren't (non-rep properties). - A number of physical rep types have been identified (text, binaryStream, binaryInteger, binaryFloat, zonedDecimal, packedDecimal, binaryCodedDecimal, xml). - XML Schema logical types have been gathered into type groups (number, string, binary, boolean, calendar). For each logical type group, only certain physical rep types are allowed (eg, for string, only text & xml). - There are a group of rep properties that are common to many physical rep types (eg, length). The majority of rep properties are specific to a physical rep type, I have decorated such property names accordingly (eg, integerSigned). Some of these are further specific to logical type group, I have further decorated such property names accordingly (eg, textCalendarScheme). - I have assumed for now that all physical rep types are part of the core standard, we need to revisit this as clearly some of the decimal reps are not universally used. - I have not (yet) attempted to assign properties to conversions. I'd rather we reviewed the properties first in terms of the general approach to organisation, naming, and so on, before starting this exercise. - The issue of defaults poses some questions. We have agreed that hard-wired model defaults are not desirable as it means the absence of a property implies a behaviour. If we wish to change this behaviour then we are stuck because existing DFDL Schema then behave differently. So the proposal in the scoping document is that all properties that are used during a parse/serialise must have an explicit value defined somewhere in the DFDL Schema, typically in a dfdl:defineFormat annotation. However, there are properties where we don't want a one-value-fits-all value, but at the same time do not want to specify a value on every element. Example: justification, where typically all strings are left justfied, and all numbers are right justified. Example: calendarPattern, where the patterns for a date, dateTime, time, monthDay, etc are different. One approach is to duplicate the properties - so we would have textStringJustification, textNumberJustification, and so on. You will see that this is what I have done in the document for justification. An alternative approach is to have one property, but to add an additional enum 'schema', which means derive the default from the logical type. You will see that this is what I have done for calendarPattern. So calendarPatternKind set to 'schema' for xsd:date would yield "yyyy-MM-dd" and for xsd:time would yield "hh:mm:ss.sss". Of course, 'schema' can be considered a form of hard-wiring, so maybe we also need some properties that define what these defaults are (they'd only ever be set at dfdl:defineFormat level)? We need to decide which of these approaches is preferred, whether they both make sense, or if there is a better way. - I have not attempted to define properties for multi-dimensional arrays, I have been leaving this until the tracker for this is resolved. - I have not attempted to define properties for physical rep type xml. I think we need a broader discussion on how XML is handled within DFDL first. - Certain properties come with a whole bunch of related properties. Examples are patterns for text numbers, patterns for text calendars, separators, initiators, terminators, occurs. For the patterns I have created schemes under which to group the related properties, like has been proposed for escapes/quotes. I have not done this for separators etc, partly so you can see the two different approaches. We need to come up with a consistent design for when to use schemes and when not to use schemes. - For each property I've suggested which DFDL annotations are applicable, this also requires checking. - My escape scheme is deliberately simple, it may well need to be improved, and we need to decide on whether alternate and/or nested schemes are required. - I've tried to be consistent with property names. For example, properties that control how another property is interpreted is suffixed with 'kind' (eg, length and lengthKind). Similarly with enum values that have the same meaning across properties (eg, always, never, output, input) for properties that say whether something is applicable to input and/or output. - Mike, Geoff, Suman and myself spent some time prior to the F2F in designing some of the properties. They may notice that I have deviated in places from what was proposed. This is invariably due to discovery of some scenario not covered by our previous discussion. For example, it turns out we need to have a separate control for trimming fixed length text, instead of deducing a behaviour from justification (because MRM has a little known but useful property that controls this). They may also notice that not all property behaviour has been stated. This is simply down to a desire to keep the property descriptions concise at this stage. - I've incorporated Geoff's latest thinking on default value and null value properties, although the document describing the theory behind these has not yet been reviewed by anyone other than Geoff and myself, and there some open questions on things like null indicator fields. I expect there to be considerable discussion in this area. - For some properties I've indicated where they can take either a literal value or an XPath. The list of such properties requires revision as I have not been exhaustive in this, mainly allowing XPath where I know that IBM's models allow it. - Although I have tried to make sure that DFDL properties encompass IBM's models, I can not guarantee this at this point in time. Review is required by IBMers beyond the DFDL WG IBMers. I have initiated this process. (See attached file: DFDL_Properties_v004.doc) Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

Telcon Notes 2006-3-22
by Westhead, Martin (Martin) 22 Mar '06

22 Mar '06

Present: Martin Westhead Bob McGrath Steve Hanson Geoff Judd Reviewed Geoff's document on "Support of Uncertainty". This is looking in good shape. Considered Steve's email on escape characters. It was observed that we need to have a way to handle XML escaping although this is a sufficiently important special case that perhaps it can be handled just as a stand alone special case. Jim raised the issue of DFDL implementations and suggested that moving forward he was going to try to focus his DFDL time on driving implementations rather than on all the language details. It was generally agreed that bringing Defudle up to date would be a big contribution to the group. Coming soon: Nulls - Geoff Properties doc - Steve

1 0

Telcon Tomorrow
by Westhead, Martin (Martin) 22 Mar '06

22 Mar '06

Sorry I missed last week and so I'm a little out of the loop I have no agenda items at this point. Contributions welcomed? Conference call instructions: Number: 1-800-664-0771 At the automated answering system, press 0 for the operator. Ask for the "Daffodil working group" call. Web Conference: https://webconferencing.avaya.com <https://webconferencing.avaya.com/> Web Conference: Part. Code 785343; WebConf Reference 158672

3 2

Transformation between Binary data and XML
by Mike Beckerle 20 Mar '06

20 Mar '06

FYI: "Edmund Lai" <edmund.lai(a)gmail.com> 03/20/2006 02:37 AM To Mike Beckerle/Worcester/IBM@IBMUS cc Subject Transformation between Binary data and XML I have done some experiment on transformation between binary data and XML that maybe of interest to DFDL-WG. I am not following any existing spec, but rather I am trying to determine a set of commands that I feel is adequate and easy to learn. Only after I finish would I try to see it I can modify it to fit into any standard. The converter is web based and is in http://www.datamech.com/XMLForm/XMLandBinary.html The description is in http://www.datamech.com/XMLForm/aboutXmlAndBinary.html

1 0

How to deal with variable length elements?
by Robert E. McGrath 17 Mar '06

17 Mar '06

Folks, I'm trying to build up a story about how to handle arrays using DFDL. But first, I need to check if I understand the basics. Here is an example of a 1D array in XML, modeled as two elements, an integer indicating how many elements, followed by an array of zero or more floats. Looking at my XML textbooks, the following seems like the correct XML schema for this notion. +++  <xs: element type="float" name="floatType" \>  <xs:complexType name="floatArray"> <xs:sequence> <xs:element name="nelems type="int" /> <xs:element name="x" ref="floatType" minOccurs="0" maxOccurs="./nelems" /> </xs:sequence> </xs:complexType> +++ Do I have this correct? (I'm pretty sure the 'maxOccurs' isn't correct, so I hope someone will tell me the right way to do this.) If so, the next question will be "how do I annotate this with DFDL?", e.g., when the data is precisely one binary (or text encoded) int, followed by some binary (or text encoeded) floats.

3 5

RE: [dfdl-wg] Telcon tomorrow
by Steve Hanson 15 Mar '06

15 Mar '06

As discussed on the call, I'll issue the first draft of the properties without creating separator/terminator/initiator schemes, and we'll decide when we have evaluated the full property set. Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Steve Hanson/UK/IBM To 09/03/2006 10:16 "Westhead, Martin \(Martin\)" <westhead(a)avaya.com> cc dfdl-wg(a)ggf.org, owner-dfdl-wg(a)ggf.org Subject RE: [dfdl-wg] Telcon tomorrow (Document link: Steve Hanson) As it currently stands, the following properties apply to separators and there may well be more (given there is Encoding there should probably be byteOrder & BOM properties for Unicode). separator separatorEncoding finalSeparatorCanBeMissing separatorPosition separatorPolicyForAbsentElements Maybe it makes more sense to treat separators like quotes and have separator schemes that the user can define and re-use? Regards, Steve Steve Hanson WebSphere Message Brokers, IBM United Kingdom Ltd, Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Westhead, Martin \(Martin\)" <westhead(a)avaya.c To om> Steve Hanson/UK/IBM@IBMGB cc 08/03/2006 16:38 <dfdl-wg(a)ggf.org>, <owner-dfdl-wg(a)ggf.org> Subject RE: [dfdl-wg] Telcon tomorrow That seems sensible. Question: Can you/should you be able to extend sub properties. For example I introduce a new conversion based on the existing separator function that adds a tag prefix to each field...I would like to be able to add (when I introduce this conversion) a new sub-property to separator called "tagPrefix". Martin >-----Original Message----- >From: Steve Hanson [mailto:smh@uk.ibm.com] >Sent: Wednesday, March 08, 2006 2:07 AM >To: Westhead, Martin (Martin) >Cc: dfdl-wg(a)ggf.org; owner-dfdl-wg(a)ggf.org >Subject: Re: [dfdl-wg] Telcon tomorrow > >Hi Martin > >Additional agenda item. This is something us IBMers discussed prior to the >F2F but I omitted to include it in my foils about properties. > >Several properties have a bunch of other sub-properties associated with >them. For example, separator has separatorEncoding and separatorIgnoreCase; >initiator and terminator likewise have equivalent sub-properties. It was >suggested that if sub-properties needed to be specified they should only be >specifiable on a sub-element, as below. This was to reduce the number of >properties on DFDL elements. > ><dfdl:sequence separator="@" /> > > or > ><dfdl:sequence> > <dfdl:separator encoding="850" ignoreCase="true">@</dfdl:separator> ></dfdl:sequence> > >Regards, Steve > >Steve Hanson >WebSphere Message Brokers, >IBM United Kingdom Ltd, Hursley, UK >Internet: smh(a)uk.ibm.com >Phone (+44)/(0) 1962-815848 > > > > "Westhead, Martin > (Martin)" > <westhead(a)avaya.c To > om> <dfdl-wg(a)ggf.org> > Sent by: cc > owner-dfdl-wg@ggf > .org Subject > [dfdl-wg] Telcon tomorrow > > 07/03/2006 22:03 > > > > > > > > >Preliminary Agenda >------------------ > > 1. Review outcome of F2F meeting > 2. Planning for next GGF > > >Conference call instructions: >Number: 1-800-664-0771 >At the automated answering system, press 0 for the operator. >Ask for the "Daffodil working group" call. > >Web Conference: https://webconferencing.avaya.com >Web Conference: Part. Code 785343; WebConf Reference 158672 >

1 0