November 2007 - dfdl-wg - lists.ogf.org

OGF DFDL - useNullValueForDefault
by Mike Beckerle 03 Dec '07

03 Dec '07

I'm trying to simplify something in the nulls/defaults space. There's this property proposed: useNullValueForDefault. This is a boolean. I'm trying to understand what this is for. On input, when the element is required but no content is found, and you therefore want to create a default value, you would look, see that the element is nillable, see that useNullValueForDefault is true, and set the logical value of that element to null. On output when no logical value is provided but the element is required, you therefore want to create a default value, so you look and see that the element is nillable and see that useNullValueForDefault is true and so you provide the logical value null as the value. Then you output as if the logical value was null in the first place, so you would output the first of the dfdl:nullValues list as the representation. These two are not symmetric. I understand wanting empty content to cause null as the logical value of a nillable element. Symmetry argues that one wants in this case for the output side to output empty content as the representation of null logical value. The way I would express this is to provide a means for the list of dfdl:nullValues to be able to contain the empty string as its first member of the list. E.g., some syntax like: dfdl:nullValues=" '' 'null' 'NULL' '\ ' " Once you have this, there is no need for useNullValueForDefault. It's functionality is subsumed by allowing nullValues to contain empty-string as a null representation. This leaves the case where an element is both nullable and has a non-null default value. Is this case important or can we just say that an element is either nullable or has default values, but not both. We already have some constraints on this stuff in that we don't allow nullable complex types, only simple. I believe allowing all combinations of nullability with default values is probably overkill. Comments? I understand wanting a missing Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 3

OGF DFDL core draft 030
by Mike Beckerle 30 Nov '07

30 Nov '07

It's located here: https://forge.gridforum.org/sf/docman/do/viewDocument/projects.dfdl-wg/docm… Roughly this is what I put in: v030 - binary/opaque types resolved - also anySimpleType - facets discussion integrated in - choiceType enum and length props on choice - prefix length strings (needs a bit more still) - wildcards with dimension fix - xs:integer type added -some on nulls/default/optionals -a few other small bits (see change history) in prep for regexp stuff. -a little bit on value calc This is a major revision since the last OGF DFDL WG draft core (which was version 025 - from back in September). I'm quite dissatisfied now with the flow of the spec. Everything seems to be needed to describe everything else. E.g., initiators can't be described without describing nulls and default values that they interact with, but those are much later in the spec and involve some very nitpicky properties like nullValueInitiatorPolicy, ..... so all the information about initiators isn't in one place. A big job ahead is revising all the property descriptions to refer to the grammar and the various syntactic regions defined by it. Planned (Roughly) vX+1 - improve (finish?) nulls/default/optionals - 2-level description (schema components) - regular expressions for lengths - expression language - valueCalc vX+2 - assertions/discriminators and choice - how speculative parsing works (combining choice and variable-occurrence - currently these are separate) - reordering the properties discussion (move representation earlier, improve flow of topics.) vX+3 - escapeSchemes - string xml type - variables and selectors - improvments on property descriptions Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

DFDL: Minutes from OGF WG call, 28 Nov 2007
by Ian W Parkinson 29 Nov '07

29 Nov '07

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 28 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Ian Parkinson (IBM) Alan Powell (IBM) 1. Introduction The agenda, distributed by Mike prior to the call, covered two main items for discussion. A proposal has been circulated to the DFDL-WG mailing list, describing the use of annotated simple types for describing length prefixes, which had already been discussed within IBM. In the absense of Simon Parker or representatives from parties other than IBM, the meeting opted not to discuss this proposal at this time. 2. Plan for contents of next draft The distributed agenda included a draft list of items to be included in the next draft of the DFDL specification. Steve queried the 'integer type added' item: Mike explained there had been a debate over support for both the integer XSD type and the decimal type. We had previously opted to support both, as the canonical representation for decimal includes a trailing ".0" on integers, which may be confusing for users. Mike would like to prepare, along similar lines, a plan covering several draft versions of the specification. Some work items, such as the SWIFT example, would be omitted from the plan as they do not directly concern the text of the specification. Mike's plan will be distributed via the DFDL-WG mailing list. Alan asked if we should set a target date for completion of the specification. Mike observed that we had previously set target dates that had not been achieved; however Steve felt that as we are in a position to plan the next few drafts of the specification, a target date is likely to be more reasonable. 3. Other business Steve has been revisiting the decimal supplement, and has found some errors. Mike observed that it would be desirable for DFDL's support of packed decimal numbers and floating point numbers to be analogous. One complexity lies in the number of types of packed decimal that DFDL needs to support (including 'pure', unsigned packed, IBM packed, and zoned decimal), along with variants, which Mike felt might best be addressed through a small pattern language. Steve would investigate this further. Meeting closed, 17:30 GMT Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL plans for next few spec versions
by Mike Beckerle 28 Nov '07

28 Nov '07

I'd like to rough out targets for content for the next several spec drafts: vNext - (I hope to get this all in this week) - binary/opaque types resolved - also anySimpleType - facets discussion integrated in - choiceType enum and length props on choice - prefix lengths - wildcards with dimension fix - xs:integer type added vX+1 - 2-level description (schema components) - nulls/default/optionals - regular expressions for lengths - expression language - valueCalc vX+2 - assertions/discriminators and choice - how speculative parsing works - reordering properties discussion (move representation earlier) vX+3 - escapeSchemes - string xml type - variables and selectors - improvments on property descriptions

1 0

OGF DFDL WG call 2007-11-28
by Mike Beckerle 27 Nov '07

27 Nov '07

Agenda for the call: 1) prefix length strings - as circulated in email 2) plan for contents of next draft I suggest this as a starting point: - binary/opaque should be resolved - null/defaults - facets discussion (which we use and how) integrated - choiceType enum + length properties on choice - prefix lengths - wildcards with dimensions - integer type added 3) Any other items ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

DFDL: Minutes from OGF WG call, 21 Nov 2007
by Ian W Parkinson 22 Nov '07

22 Nov '07

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 21 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Suman Kalia (IBM) Simon Parker (PolarLake) Ian Parkinson (IBM) 1. Introduction Mike would like, in this meeting, to cover the hexBinary and base64Binary debate, and also to discuss the use of 'any' wildcards with minOccurs and maxOccurs. 2. 'Any' wildcards with minOccurs and maxOccurs The DFDL specification presently disallows the use of minOccurs and maxOccurs on 'any' wildcards, in contrast to XML schema. Simon stated that he sees no reason to forbid this, and that these properties might be useful, for example when structures could be followed by arbitrary extensions. Suman felt that the most common use case would be minOccurs="0" and maxOccurs="1". Mike wondered whether this would be useful within unordered groups, and thought it would be a good way to model an 'all' group containing some known fields and a number of unknown fields. Mike took an action item to investigate this further. 3. hexBinary and base64Binary The working group has been discussing, via email, the use of hexBinary and base64Binary types, along with 'enumeration' and 'pattern' properties. Mike asked the group whether we should disallow the 'enumeration' and 'pattern' properties on binary types (as they are difficult to use, in particular with base64Binary), and whether we should remove support for base64Binary (as it shares a value space with hexBinary, and is therefore a synonym of hexBinary in DFDL). Simon felt that this distinction remained a useful hint to any component emitting XML based on a DFDL infoset. He also observed that base64Binary is more commonly used than hexBinary, and is preferred. Mike argued that while base 64 might be preferred to hexadecimal in terms of space, hexadecimal is more readable. Steve observed that hexBinary is commonly used. While Mike felt that supporting fixed and default binary values might lead to requiring support for patterns and enumeration, the meeting agreed that there are use cases for default and fixed values - for example, some file formats use "eyecatchers" which are best expressed in hexadecimal. Mike pointed out that this could be achieved usng a string type, but suggested allowing 'default' and 'fixed' for hexBinary. Simon suggested that something similar would be necesssary for base64Binary, as some values (such as passwords and identifiers) are frequently expressed in base 64. Sandy Gao (IBM) has been asked to comment on whether there are any use cases where patterns are used with hexBinary of base64Binary. To conclude this discussion, Mike proposed the following: to retain support for both base64Binary and hexBinary, with identical content in DFDL; to allow both 'fixed' and 'default' for both base64Binary and hexBinary, but (pending further information from Sandy) to disallow 'pattern' and 'enum'. [Simon and Suman left the meeting] 4. Array Prefixes and Suffixes Mike asked whether the group was happy with the omission of array prefixes and suffixes. We know how to add these back should we ever need to, and there is a concern that including them would lead to many more array properties being necessary. Steve was happy with the present proposal. 5. Choice type and Length properties on Choice Mike observed that there is a need to distinguish between choice groups which are of constant length, and choice groups where the length is determined by the relevent subelement. Where the choice is unresolvable, it is not possible to have a choice of variable length. Steve felt that as we are able to make assertions, there would be very few cases where a choice is unresolvable. Mike pointed out that in an unresolvable choice group, each arm would need consistant enough syntax for a parser to be able to determine the end; and that this could be modelled as arms with enough information to discriminate. Geoff suggested that experience with IBM's MRM technology shows this to be unusual. The meeting considered two options. We could specify two properties, one to select between constant length and variable length; and one to select between resolvable and unresolvable. In this option, the combination variable-length/unresolvable would be disallowed. In the second option, we would have a single property with three possible values: constant length, variable length or unresolvable. The meeting agreed upon the second option. When experimenting with the DFDL language recently, Steve found specifying length on structures to be awkward. He proposed removing 'lengthKind' on choice elements, and Mike added that we would also wish to remove other associated properties such as 'intitator' and 'terminator'. On reflection, the meeting decided to keep these properties, noting that using these on a choice element is identical to wrapping the choice element in a sequence element with the same properties. 6. Other business There has, internally within IBM, been a discussion regarding length prefixes on strings. Mike will circulate a proposal to the working group, to allow prefix formats to be described through annotations on simpleType definitions. Meeting closed, 17:55 GMT Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL & length prefixes - proposal
by Mike Beckerle 21 Nov '07

21 Nov '07

For the broader DFDL workgroup (Martin, Simon, and any other onlookers), this proposal was circulated among the IBM DFDLers and we liked it, so we wanted others to take a look. ------------------------------------------------------------------------ Prefix lengths are common, but unfortunately there are many prefix length formats. For flexibility we want to be able to use a hidden field and calc properties, but this is clumsy enough to be very unattractive except for extreme cases. However, without it we have the keyword proliferation issue. Here's a way to get the best of both worlds. <element name="myString" type="string" dfdl:lengthKind="prefixed" dfdl:lengthPrefixType="packed4"/> here's the trick. Named types for length prefixes are just a real schema type definition of an integer type with dfdl annotations. . <simpleType name="packed4"> <restriction base="integer" dfdl:representation="decimal" dfdl:decimalType="packed" dfdl:numberFormat="9999" dfdl:decimalSigned="false" dfdl:prefixIncludesPrefixLength="false"/> </simpleType> The behavior is simple. Imagine that a hidden length prefix element is defined immediately before your element, and it's type is the one given here. You don't have to model that hidden element. DFDL knows how to do that for you. Just tell us it's type, including DFDL properties for the representation of that type. We could build in a few of these to some predefined profiles. E.g., dfdl:prefix2, and dfdl:prefix4 might be the typical 2 and 4 byte length prefix definitions you see in PL1 for example. I suggest the above because of the slippery slope where we have to gradually bifurcate all the integer properties to have lengthPrefix flavors of them e.g., lengthPrefixRepresentation = text or binary or decimal lengthPrefixLengthUnits = bits, bytes, or characters lengthPrefixLength = an integer lengthPrefixLengthIncludesPrefix (boolean) lengthPrefixTextNumberJustification lengthPrefixTextNumberFormat Anywhere you stop here there will be many cases that aren't covered.

1 0

DFDL wildcards with dimension
by Mike Beckerle 21 Nov '07

21 Nov '07

I did some investigation. 1) xs:any wildcards are not allowed inside xs:all groups at all. 2) The schema below is valid, and admits the instance document following it: <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="x"> <xsd:complexType> <xsd:sequence> <xsd:element name="name"/> <xsd:element name="address"/> <xsd:any processContents="lax" minOccurs='0' maxOccurs='unbounded'/> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:schema> Here's the instance: <?xml version="1.0"?> <x> <name/> <address/> <foo/> <bar/> <quux/> <frobboz/> </x> 3) Note however, the only way to create an XML Schema which will validate both the instance above and the one below where the order is all scrambled, is an array-of-choice schema: <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="x"> <xsd:complexType> <xsd:choice maxOccurs='unbounded'>  <xsd:element name="name"/> <xsd:element name="address"/> <xsd:any processContents="lax"/> </xsd:choice> </xsd:complexType> </xsd:element> </xsd:schema> This array-of-choice schema will accept this instance document also: <?xml version="1.0"?> <x> <foo/> <address/> <bar/> <name/> <quux/> <frobboz/> </x> The above array-of-choice schema obviously will also accept things where there are multiple name and address entries, which is undesirable, but the XPath expressions x/name and x/address will still work. <?xml version="1.0"?> <x> <address/> <address/> <foobar/> </x> Implications for DFDL: I suggest we should allow dimension on any wildcards because this unordered sequence DFDL fragment: <xsd:sequence dfdl:ordered="false"> <xsd:element name="name"/> <xsd:element name="address"/> <xsd:any id="unknown" processContents="lax" minOccurs='0' maxOccurs='unbounded'/> </xsd:sequence> describes something quite useful which otherwise requires an array of choice to model. Array-of-choice is generally undesirable as a model because it loses the cardinality information that says that name is required and address is required and there can be exactly 1 instance of each. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

Fw: OGF DFDL WG call today 2007-11-21
by Suman Kalia 21 Nov '07

21 Nov '07

Some interesting discussion on the hexBinary and base64Binary.. Schema authors also had reservations on having these 2 types with same value space and considered a possibility of having one binary type with encoding facet to specify hex and base64 which would have been a much better solution. I am equally perplexed on the use case for having pattern (and also enumeration) facet on the hexBinary and base64Binary as the facet work on the lexical space. May be the conservative approach for us is to treat these built-in schema type as is and only allow the dfdl set of annotations pertaining to allowable facets.. By copy to Sandy - do you know of a use case where pattern facet is used on hexBinary and base64Binary http://lists.w3.org/Archives/Public/www-xml-schema-comments/2001AprJun/0067… Some excerpts from discussion There are two possible ways to fix this before going to recommendation: 1. return to the CR status of a binary datatype with an encoding facet that could be either 'hex' or 'base64'. 2. eliminate one of hexBinary or base64Binary, and call the remaining type 'binary'. *Note that this is the most conservative decision*. The missing encoding could always be added back, but once released as an official recommendation parsers would be required to support both encodings for all time. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 11/21/2007 11:50 AM ----- Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 11/21/2007 11:08 AM To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] OGF DFDL WG call today 2007-11-21 We may or may not achieve quorum today because of the US holiday tomorrow and big travel day today. I will join the call at 12noon US.ET, and if we have enough people by 12:05 then I'd like to discuss one or more of these topics * hexBinary and base64Binary - I still find these confusing. * array prefix and suffix - just review resolution of this issue - leaving out for now just to be conservative. Could put back in fairly easily. * choiceType and length properties on xs:choice My latest musings on hexBinary and base64Binary .... E.g., XSD allows pattern and enumeration facets on these <element name="aThing" type="base64Binary" length="3" pattern="AAAA|////" /> I think that pattern is a regexp for a base64 string matching 3 bytes of zeros or 3 bytes of all ones, but my regexp syntax is no doubt incorrectly escapified. ("A" is 6 bits of zero, "/" is 6 bits of 1 in base64). To me this is marvelously confusing. It feels downright silly to allow pattern and enumeration on these things. I'd like to adopt the strictest possible sensible thing. - hexBinary only, binary representation only, no pattern or enumeration facets supported or allowed for this type. There is still the issue of default/fixed. E.g., <element name="unknownStuff" type="hexBinary" fixed="F41306C0" dfdl:lengthKind="implicit" /> The user specifies what the bytes are as a hex string, and since they said it's hexBinary, they use hex to express the literal content. This is a way to say "right here in the data there's this blob of stuff I don't understand, but it always contains these data bytes". I've certainly seen the need for this sort of thing. It's being ignored on input, but on output it generates the fixed bytes of data so as to create valid output even when you don't understand this part of the data format. In fact, the cases I've seen are using this to skip over decimal numbers the wierd format of which isn't understood. The above pattern could be one or more decimal numbers in strange formats. However, I can see the slippery slope from here to allowing pattern and enumeration facets, so I'd be happy ruling out use of default and fixed on hexBinary type also, just to make it even simpler. Also, this element achieves the same end using our "%" escapes in strings. <element name="unknownStuff" type="string" dfdl:encoding="ascii" fixed="%F4%13%06%C0" dfdl:lengthKind="implicit"/> This works for any single-byte-wide character-set encoding. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

OGF DFDL WG call today 2007-11-21
by Mike Beckerle 21 Nov '07

21 Nov '07

We may or may not achieve quorum today because of the US holiday tomorrow and big travel day today. I will join the call at 12noon US.ET, and if we have enough people by 12:05 then I'd like to discuss one or more of these topics * hexBinary and base64Binary - I still find these confusing. * array prefix and suffix - just review resolution of this issue - leaving out for now just to be conservative. Could put back in fairly easily. * choiceType and length properties on xs:choice My latest musings on hexBinary and base64Binary .... E.g., XSD allows pattern and enumeration facets on these <element name="aThing" type="base64Binary" length="3" pattern="AAAA|////" /> I think that pattern is a regexp for a base64 string matching 3 bytes of zeros or 3 bytes of all ones, but my regexp syntax is no doubt incorrectly escapified. ("A" is 6 bits of zero, "/" is 6 bits of 1 in base64). To me this is marvelously confusing. It feels downright silly to allow pattern and enumeration on these things. I'd like to adopt the strictest possible sensible thing. - hexBinary only, binary representation only, no pattern or enumeration facets supported or allowed for this type. There is still the issue of default/fixed. E.g., <element name="unknownStuff" type="hexBinary" fixed="F41306C0" dfdl:lengthKind="implicit" /> The user specifies what the bytes are as a hex string, and since they said it's hexBinary, they use hex to express the literal content. This is a way to say "right here in the data there's this blob of stuff I don't understand, but it always contains these data bytes". I've certainly seen the need for this sort of thing. It's being ignored on input, but on output it generates the fixed bytes of data so as to create valid output even when you don't understand this part of the data format. In fact, the cases I've seen are using this to skip over decimal numbers the wierd format of which isn't understood. The above pattern could be one or more decimal numbers in strange formats. However, I can see the slippery slope from here to allowing pattern and enumeration facets, so I'd be happy ruling out use of default and fixed on hexBinary type also, just to make it even simpler. Also, this element achieves the same end using our "%" escapes in strings. <element name="unknownStuff" type="string" dfdl:encoding="ascii" fixed="%F4%13%06%C0" dfdl:lengthKind="implicit"/> This works for any single-byte-wide character-set encoding. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0