- dfdl-wg - lists.ogf.org

DFDL - Affiliation changing for Mike Beckerle
by Mike Beckerle 12 Dec '07

12 Dec '07

DFDLers, I am leaving IBM to join a smaller company. The good news is that I will continue to be involved in DFDL-related work in my new capacity. I will forward contact info for my new position soon. In the mean time I can be reached at the address here: (encoded to avoid spam address scrapers) michael dot beckerle at alum dot mit dot edu ...mikeb

1 0

Can't make call today. Sorry
by Mike Beckerle 12 Dec '07

12 Dec '07

Sorry everyone. I cannot make the call today. ...mikeb

1 0

DFDL infoset and wildcards
by Mike Beckerle 07 Dec '07

07 Dec '07

Our Infoset section of the spec is silent about wildcards. Do we need an infoset information item that corresponds to things that match a wildcard? Or can wildcards just be element info items that lack [namespace] [dataType] and where the [schema] refers to the schema component that defines the wildcard, not to an element declaration. Comments? Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

OGF DFDL WG minutes 2007-12-05 call
by Mike Beckerle 06 Dec '07

06 Dec '07

OGF DFDL WG minutes 2007-12-05 call Suman Kalia, Simon Parker, Alan Powell, Mike Beckerle (who else? - was someone else on also) We discussed Output issues in the DFDL expression language: E.g.., an outputValueCalc for a field in the header of a data stream may contain information that requires you to know the rep, or length of the rep, of the whole data item. We concluded that this kind of thing can't be ruled out. Some formats just require buffering and are not streamable; however, implementations can vary on just how large a data item they're able to cope with here. Expression language section will include a subsection highlighting this issue and that implementations can vary here. Alan will update his expression language proposal and include this. Also suggested was a path length-from-to function that takes 2 path expressions and gives you the size of the represntation between them. (start of first, to last bit before start of 2nd). (I don't think we discussed a clear use case motivating this, but there may be one. We did discuss applications trying to fit data into limited size boxes, but the use case is not clear. Also note that all representation lengths are subject to change due to different starting alignments.) Nillable and Default: We also discussed the interaction of nillable and having a default. The sense of the group on the call is that we can restrict these so that if something is nillable it cannot also have a default value, and that the behavior of DFDL on output for a required element that is nillable but not in the logical data, is to create a null value. Everyone agreed that there is no need for a property useNullValueForDefault because this should always be the behavior. Mike will forward a proposal. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 2

OGF DFDL - useNullValueForDefault
by Mike Beckerle 03 Dec '07

03 Dec '07

I'm trying to simplify something in the nulls/defaults space. There's this property proposed: useNullValueForDefault. This is a boolean. I'm trying to understand what this is for. On input, when the element is required but no content is found, and you therefore want to create a default value, you would look, see that the element is nillable, see that useNullValueForDefault is true, and set the logical value of that element to null. On output when no logical value is provided but the element is required, you therefore want to create a default value, so you look and see that the element is nillable and see that useNullValueForDefault is true and so you provide the logical value null as the value. Then you output as if the logical value was null in the first place, so you would output the first of the dfdl:nullValues list as the representation. These two are not symmetric. I understand wanting empty content to cause null as the logical value of a nillable element. Symmetry argues that one wants in this case for the output side to output empty content as the representation of null logical value. The way I would express this is to provide a means for the list of dfdl:nullValues to be able to contain the empty string as its first member of the list. E.g., some syntax like: dfdl:nullValues=" '' 'null' 'NULL' '\ ' " Once you have this, there is no need for useNullValueForDefault. It's functionality is subsumed by allowing nullValues to contain empty-string as a null representation. This leaves the case where an element is both nullable and has a non-null default value. Is this case important or can we just say that an element is either nullable or has default values, but not both. We already have some constraints on this stuff in that we don't allow nullable complex types, only simple. I believe allowing all combinations of nullability with default values is probably overkill. Comments? I understand wanting a missing Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 3

OGF DFDL core draft 030
by Mike Beckerle 30 Nov '07

30 Nov '07

It's located here: https://forge.gridforum.org/sf/docman/do/viewDocument/projects.dfdl-wg/docm… Roughly this is what I put in: v030 - binary/opaque types resolved - also anySimpleType - facets discussion integrated in - choiceType enum and length props on choice - prefix length strings (needs a bit more still) - wildcards with dimension fix - xs:integer type added -some on nulls/default/optionals -a few other small bits (see change history) in prep for regexp stuff. -a little bit on value calc This is a major revision since the last OGF DFDL WG draft core (which was version 025 - from back in September). I'm quite dissatisfied now with the flow of the spec. Everything seems to be needed to describe everything else. E.g., initiators can't be described without describing nulls and default values that they interact with, but those are much later in the spec and involve some very nitpicky properties like nullValueInitiatorPolicy, ..... so all the information about initiators isn't in one place. A big job ahead is revising all the property descriptions to refer to the grammar and the various syntactic regions defined by it. Planned (Roughly) vX+1 - improve (finish?) nulls/default/optionals - 2-level description (schema components) - regular expressions for lengths - expression language - valueCalc vX+2 - assertions/discriminators and choice - how speculative parsing works (combining choice and variable-occurrence - currently these are separate) - reordering the properties discussion (move representation earlier, improve flow of topics.) vX+3 - escapeSchemes - string xml type - variables and selectors - improvments on property descriptions Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

DFDL: Minutes from OGF WG call, 28 Nov 2007
by Ian W Parkinson 29 Nov '07

29 Nov '07

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 28 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Ian Parkinson (IBM) Alan Powell (IBM) 1. Introduction The agenda, distributed by Mike prior to the call, covered two main items for discussion. A proposal has been circulated to the DFDL-WG mailing list, describing the use of annotated simple types for describing length prefixes, which had already been discussed within IBM. In the absense of Simon Parker or representatives from parties other than IBM, the meeting opted not to discuss this proposal at this time. 2. Plan for contents of next draft The distributed agenda included a draft list of items to be included in the next draft of the DFDL specification. Steve queried the 'integer type added' item: Mike explained there had been a debate over support for both the integer XSD type and the decimal type. We had previously opted to support both, as the canonical representation for decimal includes a trailing ".0" on integers, which may be confusing for users. Mike would like to prepare, along similar lines, a plan covering several draft versions of the specification. Some work items, such as the SWIFT example, would be omitted from the plan as they do not directly concern the text of the specification. Mike's plan will be distributed via the DFDL-WG mailing list. Alan asked if we should set a target date for completion of the specification. Mike observed that we had previously set target dates that had not been achieved; however Steve felt that as we are in a position to plan the next few drafts of the specification, a target date is likely to be more reasonable. 3. Other business Steve has been revisiting the decimal supplement, and has found some errors. Mike observed that it would be desirable for DFDL's support of packed decimal numbers and floating point numbers to be analogous. One complexity lies in the number of types of packed decimal that DFDL needs to support (including 'pure', unsigned packed, IBM packed, and zoned decimal), along with variants, which Mike felt might best be addressed through a small pattern language. Steve would investigate this further. Meeting closed, 17:30 GMT Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL plans for next few spec versions
by Mike Beckerle 28 Nov '07

28 Nov '07

I'd like to rough out targets for content for the next several spec drafts: vNext - (I hope to get this all in this week) - binary/opaque types resolved - also anySimpleType - facets discussion integrated in - choiceType enum and length props on choice - prefix lengths - wildcards with dimension fix - xs:integer type added vX+1 - 2-level description (schema components) - nulls/default/optionals - regular expressions for lengths - expression language - valueCalc vX+2 - assertions/discriminators and choice - how speculative parsing works - reordering properties discussion (move representation earlier) vX+3 - escapeSchemes - string xml type - variables and selectors - improvments on property descriptions

1 0

OGF DFDL WG call 2007-11-28
by Mike Beckerle 27 Nov '07

27 Nov '07

Agenda for the call: 1) prefix length strings - as circulated in email 2) plan for contents of next draft I suggest this as a starting point: - binary/opaque should be resolved - null/defaults - facets discussion (which we use and how) integrated - choiceType enum + length properties on choice - prefix lengths - wildcards with dimensions - integer type added 3) Any other items ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

DFDL: Minutes from OGF WG call, 21 Nov 2007
by Ian W Parkinson 22 Nov '07

22 Nov '07

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 21 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Suman Kalia (IBM) Simon Parker (PolarLake) Ian Parkinson (IBM) 1. Introduction Mike would like, in this meeting, to cover the hexBinary and base64Binary debate, and also to discuss the use of 'any' wildcards with minOccurs and maxOccurs. 2. 'Any' wildcards with minOccurs and maxOccurs The DFDL specification presently disallows the use of minOccurs and maxOccurs on 'any' wildcards, in contrast to XML schema. Simon stated that he sees no reason to forbid this, and that these properties might be useful, for example when structures could be followed by arbitrary extensions. Suman felt that the most common use case would be minOccurs="0" and maxOccurs="1". Mike wondered whether this would be useful within unordered groups, and thought it would be a good way to model an 'all' group containing some known fields and a number of unknown fields. Mike took an action item to investigate this further. 3. hexBinary and base64Binary The working group has been discussing, via email, the use of hexBinary and base64Binary types, along with 'enumeration' and 'pattern' properties. Mike asked the group whether we should disallow the 'enumeration' and 'pattern' properties on binary types (as they are difficult to use, in particular with base64Binary), and whether we should remove support for base64Binary (as it shares a value space with hexBinary, and is therefore a synonym of hexBinary in DFDL). Simon felt that this distinction remained a useful hint to any component emitting XML based on a DFDL infoset. He also observed that base64Binary is more commonly used than hexBinary, and is preferred. Mike argued that while base 64 might be preferred to hexadecimal in terms of space, hexadecimal is more readable. Steve observed that hexBinary is commonly used. While Mike felt that supporting fixed and default binary values might lead to requiring support for patterns and enumeration, the meeting agreed that there are use cases for default and fixed values - for example, some file formats use "eyecatchers" which are best expressed in hexadecimal. Mike pointed out that this could be achieved usng a string type, but suggested allowing 'default' and 'fixed' for hexBinary. Simon suggested that something similar would be necesssary for base64Binary, as some values (such as passwords and identifiers) are frequently expressed in base 64. Sandy Gao (IBM) has been asked to comment on whether there are any use cases where patterns are used with hexBinary of base64Binary. To conclude this discussion, Mike proposed the following: to retain support for both base64Binary and hexBinary, with identical content in DFDL; to allow both 'fixed' and 'default' for both base64Binary and hexBinary, but (pending further information from Sandy) to disallow 'pattern' and 'enum'. [Simon and Suman left the meeting] 4. Array Prefixes and Suffixes Mike asked whether the group was happy with the omission of array prefixes and suffixes. We know how to add these back should we ever need to, and there is a concern that including them would lead to many more array properties being necessary. Steve was happy with the present proposal. 5. Choice type and Length properties on Choice Mike observed that there is a need to distinguish between choice groups which are of constant length, and choice groups where the length is determined by the relevent subelement. Where the choice is unresolvable, it is not possible to have a choice of variable length. Steve felt that as we are able to make assertions, there would be very few cases where a choice is unresolvable. Mike pointed out that in an unresolvable choice group, each arm would need consistant enough syntax for a parser to be able to determine the end; and that this could be modelled as arms with enough information to discriminate. Geoff suggested that experience with IBM's MRM technology shows this to be unusual. The meeting considered two options. We could specify two properties, one to select between constant length and variable length; and one to select between resolvable and unresolvable. In this option, the combination variable-length/unresolvable would be disallowed. In the second option, we would have a single property with three possible values: constant length, variable length or unresolvable. The meeting agreed upon the second option. When experimenting with the DFDL language recently, Steve found specifying length on structures to be awkward. He proposed removing 'lengthKind' on choice elements, and Mike added that we would also wish to remove other associated properties such as 'intitator' and 'terminator'. On reflection, the meeting decided to keep these properties, noting that using these on a choice element is identical to wrapping the choice element in a sequence element with the same properties. 6. Other business There has, internally within IBM, been a discussion regarding length prefixes on strings. Mike will circulate a proposal to the working group, to allow prefix formats to be described through annotations on simpleType definitions. Meeting closed, 17:55 GMT Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0