- dfdl-wg - lists.ogf.org

DFDL & length prefixes - proposal
by Mike Beckerle 21 Nov '07

21 Nov '07

For the broader DFDL workgroup (Martin, Simon, and any other onlookers), this proposal was circulated among the IBM DFDLers and we liked it, so we wanted others to take a look. ------------------------------------------------------------------------ Prefix lengths are common, but unfortunately there are many prefix length formats. For flexibility we want to be able to use a hidden field and calc properties, but this is clumsy enough to be very unattractive except for extreme cases. However, without it we have the keyword proliferation issue. Here's a way to get the best of both worlds. <element name="myString" type="string" dfdl:lengthKind="prefixed" dfdl:lengthPrefixType="packed4"/> here's the trick. Named types for length prefixes are just a real schema type definition of an integer type with dfdl annotations. . <simpleType name="packed4"> <restriction base="integer" dfdl:representation="decimal" dfdl:decimalType="packed" dfdl:numberFormat="9999" dfdl:decimalSigned="false" dfdl:prefixIncludesPrefixLength="false"/> </simpleType> The behavior is simple. Imagine that a hidden length prefix element is defined immediately before your element, and it's type is the one given here. You don't have to model that hidden element. DFDL knows how to do that for you. Just tell us it's type, including DFDL properties for the representation of that type. We could build in a few of these to some predefined profiles. E.g., dfdl:prefix2, and dfdl:prefix4 might be the typical 2 and 4 byte length prefix definitions you see in PL1 for example. I suggest the above because of the slippery slope where we have to gradually bifurcate all the integer properties to have lengthPrefix flavors of them e.g., lengthPrefixRepresentation = text or binary or decimal lengthPrefixLengthUnits = bits, bytes, or characters lengthPrefixLength = an integer lengthPrefixLengthIncludesPrefix (boolean) lengthPrefixTextNumberJustification lengthPrefixTextNumberFormat Anywhere you stop here there will be many cases that aren't covered.

1 0

DFDL wildcards with dimension
by Mike Beckerle 21 Nov '07

21 Nov '07

I did some investigation. 1) xs:any wildcards are not allowed inside xs:all groups at all. 2) The schema below is valid, and admits the instance document following it: <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="x"> <xsd:complexType> <xsd:sequence> <xsd:element name="name"/> <xsd:element name="address"/> <xsd:any processContents="lax" minOccurs='0' maxOccurs='unbounded'/> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:schema> Here's the instance: <?xml version="1.0"?> <x> <name/> <address/> <foo/> <bar/> <quux/> <frobboz/> </x> 3) Note however, the only way to create an XML Schema which will validate both the instance above and the one below where the order is all scrambled, is an array-of-choice schema: <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="x"> <xsd:complexType> <xsd:choice maxOccurs='unbounded'>  <xsd:element name="name"/> <xsd:element name="address"/> <xsd:any processContents="lax"/> </xsd:choice> </xsd:complexType> </xsd:element> </xsd:schema> This array-of-choice schema will accept this instance document also: <?xml version="1.0"?> <x> <foo/> <address/> <bar/> <name/> <quux/> <frobboz/> </x> The above array-of-choice schema obviously will also accept things where there are multiple name and address entries, which is undesirable, but the XPath expressions x/name and x/address will still work. <?xml version="1.0"?> <x> <address/> <address/> <foobar/> </x> Implications for DFDL: I suggest we should allow dimension on any wildcards because this unordered sequence DFDL fragment: <xsd:sequence dfdl:ordered="false"> <xsd:element name="name"/> <xsd:element name="address"/> <xsd:any id="unknown" processContents="lax" minOccurs='0' maxOccurs='unbounded'/> </xsd:sequence> describes something quite useful which otherwise requires an array of choice to model. Array-of-choice is generally undesirable as a model because it loses the cardinality information that says that name is required and address is required and there can be exactly 1 instance of each. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

Fw: OGF DFDL WG call today 2007-11-21
by Suman Kalia 21 Nov '07

21 Nov '07

Some interesting discussion on the hexBinary and base64Binary.. Schema authors also had reservations on having these 2 types with same value space and considered a possibility of having one binary type with encoding facet to specify hex and base64 which would have been a much better solution. I am equally perplexed on the use case for having pattern (and also enumeration) facet on the hexBinary and base64Binary as the facet work on the lexical space. May be the conservative approach for us is to treat these built-in schema type as is and only allow the dfdl set of annotations pertaining to allowable facets.. By copy to Sandy - do you know of a use case where pattern facet is used on hexBinary and base64Binary http://lists.w3.org/Archives/Public/www-xml-schema-comments/2001AprJun/0067… Some excerpts from discussion There are two possible ways to fix this before going to recommendation: 1. return to the CR status of a binary datatype with an encoding facet that could be either 'hex' or 'base64'. 2. eliminate one of hexBinary or base64Binary, and call the remaining type 'binary'. *Note that this is the most conservative decision*. The missing encoding could always be added back, but once released as an official recommendation parsers would be required to support both encodings for all time. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 11/21/2007 11:50 AM ----- Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 11/21/2007 11:08 AM To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] OGF DFDL WG call today 2007-11-21 We may or may not achieve quorum today because of the US holiday tomorrow and big travel day today. I will join the call at 12noon US.ET, and if we have enough people by 12:05 then I'd like to discuss one or more of these topics * hexBinary and base64Binary - I still find these confusing. * array prefix and suffix - just review resolution of this issue - leaving out for now just to be conservative. Could put back in fairly easily. * choiceType and length properties on xs:choice My latest musings on hexBinary and base64Binary .... E.g., XSD allows pattern and enumeration facets on these <element name="aThing" type="base64Binary" length="3" pattern="AAAA|////" /> I think that pattern is a regexp for a base64 string matching 3 bytes of zeros or 3 bytes of all ones, but my regexp syntax is no doubt incorrectly escapified. ("A" is 6 bits of zero, "/" is 6 bits of 1 in base64). To me this is marvelously confusing. It feels downright silly to allow pattern and enumeration on these things. I'd like to adopt the strictest possible sensible thing. - hexBinary only, binary representation only, no pattern or enumeration facets supported or allowed for this type. There is still the issue of default/fixed. E.g., <element name="unknownStuff" type="hexBinary" fixed="F41306C0" dfdl:lengthKind="implicit" /> The user specifies what the bytes are as a hex string, and since they said it's hexBinary, they use hex to express the literal content. This is a way to say "right here in the data there's this blob of stuff I don't understand, but it always contains these data bytes". I've certainly seen the need for this sort of thing. It's being ignored on input, but on output it generates the fixed bytes of data so as to create valid output even when you don't understand this part of the data format. In fact, the cases I've seen are using this to skip over decimal numbers the wierd format of which isn't understood. The above pattern could be one or more decimal numbers in strange formats. However, I can see the slippery slope from here to allowing pattern and enumeration facets, so I'd be happy ruling out use of default and fixed on hexBinary type also, just to make it even simpler. Also, this element achieves the same end using our "%" escapes in strings. <element name="unknownStuff" type="string" dfdl:encoding="ascii" fixed="%F4%13%06%C0" dfdl:lengthKind="implicit"/> This works for any single-byte-wide character-set encoding. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

OGF DFDL WG call today 2007-11-21
by Mike Beckerle 21 Nov '07

21 Nov '07

We may or may not achieve quorum today because of the US holiday tomorrow and big travel day today. I will join the call at 12noon US.ET, and if we have enough people by 12:05 then I'd like to discuss one or more of these topics * hexBinary and base64Binary - I still find these confusing. * array prefix and suffix - just review resolution of this issue - leaving out for now just to be conservative. Could put back in fairly easily. * choiceType and length properties on xs:choice My latest musings on hexBinary and base64Binary .... E.g., XSD allows pattern and enumeration facets on these <element name="aThing" type="base64Binary" length="3" pattern="AAAA|////" /> I think that pattern is a regexp for a base64 string matching 3 bytes of zeros or 3 bytes of all ones, but my regexp syntax is no doubt incorrectly escapified. ("A" is 6 bits of zero, "/" is 6 bits of 1 in base64). To me this is marvelously confusing. It feels downright silly to allow pattern and enumeration on these things. I'd like to adopt the strictest possible sensible thing. - hexBinary only, binary representation only, no pattern or enumeration facets supported or allowed for this type. There is still the issue of default/fixed. E.g., <element name="unknownStuff" type="hexBinary" fixed="F41306C0" dfdl:lengthKind="implicit" /> The user specifies what the bytes are as a hex string, and since they said it's hexBinary, they use hex to express the literal content. This is a way to say "right here in the data there's this blob of stuff I don't understand, but it always contains these data bytes". I've certainly seen the need for this sort of thing. It's being ignored on input, but on output it generates the fixed bytes of data so as to create valid output even when you don't understand this part of the data format. In fact, the cases I've seen are using this to skip over decimal numbers the wierd format of which isn't understood. The above pattern could be one or more decimal numbers in strange formats. However, I can see the slippery slope from here to allowing pattern and enumeration facets, so I'd be happy ruling out use of default and fixed on hexBinary type also, just to make it even simpler. Also, this element achieves the same end using our "%" escapes in strings. <element name="unknownStuff" type="string" dfdl:encoding="ascii" fixed="%F4%13%06%C0" dfdl:lengthKind="implicit"/> This works for any single-byte-wide character-set encoding. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

Re: [DFDL-WG] Grammar as adapted to infoset
by Steve Hanson 21 Nov '07

21 Nov '07

Allowing dfdl:lengthKind on choices has other benefits. It means that elements, sequences and choices are consistent in how they are capable of being extracted (literal length, XPath, model determined, prefixed, null terminated). I don't think unresolvableWhenParsing implies fixed length storage. I could have an unresolvable choice being two variable length strings, terminated by the terminator of the choice. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle/Worcester/IBM@IBMUS 20/11/2007 21:23 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset Rather than overloading "implicit" length to mean this, I think we need an enum-valued property for choices. Today there is this choiceUnresolvableWhenParsing boolean. I think this shoudl become choiceType="unresolvableWhenParsing". Then we can have an enum value for the case where all arms of the choice are assumed to be padded to max length of the alternative. So I think we have 3 enum values: variableLength fixedLength unresolvableWhenParsing variableLength means the storage taken depends on the alternative selected, Fixed length means shorter alternatives are padded to the length of the longest. unresolvable means what it means now, which is that you can't tell which variant it is (which also implies the storage required is fixed length). ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Steve Hanson/UK/IBM@IBMGB 11/19/2007 07:13 AM To Mike Beckerle/Worcester/IBM@IBMUS cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset Or allow xs:choice to carry dfdl:length & dfdl:lengthKind. Then lengthKind="implicit" could mean the length is the longest of the modelled alternatives. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle/Worcester/IBM@IBMUS 16/11/2007 19:29 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset There are of course alternatives. E.g., a hidden or non-hidden field which is a byte array or hexBinary to absorb the padding bytes at the ends of the shorter variants. Or one could put trailing skip bytes on the last element or group of each of the shorter variants. One can also encapsulate the whole choice in a sequence whose length is the fixed maximum length. This is my personal favorite, since it uses the FinalUnused that is already in the grammar. E.g., suppose the length of the longest variant is 258 bytes. <sequence dfdl:lengthKind="explicit" dfdl:length="258" dfdl:lengthUnits="bytes" dfdl:applies="hereOnly"> <choice> ..... </choice> </sequence> The only thing I don't like about this is I don't have a way to express the length other than to hard-code the constant 258, but since tooling would typically put this in based on Cobol descriptors or other information I don't mind so much. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Steve Hanson/UK/IBM@IBMGB 11/16/2007 11:24 AM To Mike Beckerle/Worcester/IBM cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset Hi Mike One extra thought post-call, caused by thinking about the FinalUnused part of a sequence. Some fixed length choices have a rule that each branch of the choice must be of equal length. This results in unused bytes for branches shorter than the maximum. In a bitstream instance, a user will see these bytes. Perhaps the grammar should acknowledge their existence, even if there are no DFDL properties to express the bytes (like FinalUnused)? Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 10/11/2007 00:16 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Grammar as adapted to infoset I took a crack at this. I ripped it out of the main document for faster review/turnaround. I think this matches the latest infoset email diagram. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Grammar as adapted to infoset
by Mike Beckerle 20 Nov '07

20 Nov '07

Rather than overloading "implicit" length to mean this, I think we need an enum-valued property for choices. Today there is this choiceUnresolvableWhenParsing boolean. I think this shoudl become choiceType="unresolvableWhenParsing". Then we can have an enum value for the case where all arms of the choice are assumed to be padded to max length of the alternative. So I think we have 3 enum values: variableLength fixedLength unresolvableWhenParsing variableLength means the storage taken depends on the alternative selected, Fixed length means shorter alternatives are padded to the length of the longest. unresolvable means what it means now, which is that you can't tell which variant it is (which also implies the storage required is fixed length). ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Steve Hanson/UK/IBM@IBMGB 11/19/2007 07:13 AM To Mike Beckerle/Worcester/IBM@IBMUS cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset Or allow xs:choice to carry dfdl:length & dfdl:lengthKind. Then lengthKind="implicit" could mean the length is the longest of the modelled alternatives. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle/Worcester/IBM@IBMUS 16/11/2007 19:29 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset There are of course alternatives. E.g., a hidden or non-hidden field which is a byte array or hexBinary to absorb the padding bytes at the ends of the shorter variants. Or one could put trailing skip bytes on the last element or group of each of the shorter variants. One can also encapsulate the whole choice in a sequence whose length is the fixed maximum length. This is my personal favorite, since it uses the FinalUnused that is already in the grammar. E.g., suppose the length of the longest variant is 258 bytes. <sequence dfdl:lengthKind="explicit" dfdl:length="258" dfdl:lengthUnits="bytes" dfdl:applies="hereOnly"> <choice> ..... </choice> </sequence> The only thing I don't like about this is I don't have a way to express the length other than to hard-code the constant 258, but since tooling would typically put this in based on Cobol descriptors or other information I don't mind so much. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Steve Hanson/UK/IBM@IBMGB 11/16/2007 11:24 AM To Mike Beckerle/Worcester/IBM cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Grammar as adapted to infoset Hi Mike One extra thought post-call, caused by thinking about the FinalUnused part of a sequence. Some fixed length choices have a rule that each branch of the choice must be of equal length. This results in unused bytes for branches shorter than the maximum. In a bitstream instance, a user will see these bytes. Perhaps the grammar should acknowledge their existence, even if there are no DFDL properties to express the bytes (like FinalUnused)? Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 10/11/2007 00:16 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Grammar as adapted to infoset I took a crack at this. I ripped it out of the main document for faster review/turnaround. I think this matches the latest infoset email diagram. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

DFDL infoset and hexBinary
by Mike Beckerle 20 Nov '07

20 Nov '07

What is the DFDL infoset value for a hexBinary type element? Our doc says an element of type hexBinary has a value member in the infoset whose type is a hexBinary, but what is a hexBinary anyway? Consider this 12 bytes of data, dumped here as hex: 003100320033004100420043 If that's a string in UTF-16BE encoding, it looks like "123ABC" It is 12 bytes long. Suppose we describe this using this DFDL: <element name="a" type="hexBinary" length='12' /> As I understand it, this is correct XSD in that the length of a hexBinary is always specified as the number of 'octets' of the binary data, not characters. Now, the contents of the basic XML infoset for an element matching this schema are these 24 characters: "003100320033004100420043". This is because the XML infoset has no idea what this string is because it is not schema aware. An XML document corresponding to this might look like: <! xml version='1.0' encoding="UTF-16BE" !> <a>003100320033004100420043</a> So, the question is...... What are the contents of the DFDL Infoset for a hexBinary like this? Same 24 characters? or 12 'octets' of data? This is essentially the answer to the question "what IS a hexBinary?" or maybe "what is a hexBinary in the PSVI other than just a string?" ...mikeb

2 2

DFDL hexBinary and base64Binary
by Mike Beckerle 20 Nov '07

20 Nov '07

I'm trying to wrap up the opaque/hexBinary/base64Binary topic. I need opinions on this discussion. Currently we have a property, dfdl:binaryType : Properties Specific to Binary Types (hexBinary, base64Binary) Property Name Description binaryType Enum This specifies the encoding method for the binary. Valid values are ‘unspecified’, ‘hexBinary’, ‘base64Binary’, ‘uuencoded’ Annotation: dfdl:element (simple type ‘binary’, ‘opaque’) This property speaks to what kinds of representations can we interpret and construct logical hexbinary values from? (similarly base64Binary) I believe the above is not clear, and causes issues with the xs:length facet of XSD. I propose the 4 tables below which describe the 4 cases: hexbinary - binary hexbinary - text base64binary - binary base64binary - text I have specified these so that the meaning of the xs:length facet is always interpreted exactly as in XSD. It always refers to the number of bytes of the unencoded binary data, and never to the number of characters in the encoded form. type representation lengthKind resulting length (in bytes) other xs:hexBinary binary implicit xs:length facet explicit dfdl:length Validation: xs:length facet must be equal to resulting length in bytes (TBD: similar range checks on xs:minLength, xs:maxLength) endOfData or delimited or nullTerminated variable type representation lengthKind resulting length (in characters) other xs:hexBinary text implicit 2 * xs:length facet explicit dfdl:length Validation: xs:length facet * 2 must be equal to resulting character length (after removing all non-hex characters) (TBD: similar range checks on xs:minLength, xs:maxLength) endOfData, delimited, nullTerminated Variable type representation dfdl:lengthKind resulting length (in bytes) other xs:base64Binary binary implicit xs:length facet explicit dfdl:length Validation: xs:length facet must be equal to resulting length in bytes (TBD: similar range checks on xs:minLength, xs:maxLength) endOfData or delimited or nullTerminated variable type representation lengthKind resulting length (in characters) other xs:base64Binary text implicit 8/6 * xs:length facet explicit dfdl:length Validation: xs:length facet * 8/6 must be equal to resulting character length (after removing all non-base64-encoding characters) (TBD: similar range checks on xs:minLength, xs:maxLength) endOfData, delimited, nullTerminated Variable Looking at the above, one way to simplify things quite a bit is to disallow the xs:length and xs:minLength and xs:maxLength facet on hexBinary and base64Binary types in DFDL schemas. Then the implicit lengthKind goes away, and the complex validation check for the xs:length facet goes away. I recommend this. Another simplification alternative is to disallow representation text altogether, but I am concerned that peopel with data that does contain hex or base64 data will naturally want to use these types to model it. I don't recommend this. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

4 7

Fw: DFDL array prefix
by Mike Beckerle 20 Nov '07

20 Nov '07

This is where Suman and I left it on the array prefix issue. The problem is where do you stop with creating special "occurs" versions. You can in principle have an "occurs" variant of almost every property. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 ----- Forwarded by Mike Beckerle/Worcester/IBM on 11/20/2007 10:02 AM ----- Suman Kalia/Toronto/IBM@IBMCA 11/16/2007 11:53 AM To Mike Beckerle/Worcester/IBM@IBMUS cc Subject Fw: DFDL array prefix Mike - The proposed grammar productions seem correct to me.. I see your view point and your concern for bifurcation of properties and I do agree that it introduces level of redundancy. I guess the choice boils down to having extra level of redundancy (dummy sequence) in the logical model or having redundancy in DFDL properties and each alternative can be justified and supported with good arguments. From consumability perspective having clean and less/no extra level of redundancy in the logical model is preferred because it would be at the user face all the time while the DFDL properties pertaining to occurrences will be rare and only for specific use cases. From pure DFDL spec perspective, having no redundancy in DFDL properties is desirable. May be we can go with the conservative approach as you mentioned ie not add the extra production in the grammar for Array for now but add an appendix in the document listing the alternative for overcoming redundancy in the logical model and how the revised grammar would look like. We can get more feedback when the document is reviewed by IBMers (architects, implementers, UCD) and folks in the standards community. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 11/16/2007 10:34 AM ----- Mike Beckerle/Worcester/IBM@IBMUS 11/16/2007 10:17 AM To Suman Kalia/Toronto/IBM@IBMCA cc Subject DFDL array prefix Here's the proposal I think: Array = ArrayPrefix ArrayContent ArrayContent = [ Element [ OccursSeparator Element ]* [ OccursSeparator StopValue] ] StopValue = SimpleElement I'd suggest the keyword occursLeadingSkipBytes is the property which controls the ArrayPrefix content. My argument against this is a slippery-slope argument. This starts us down the path of having a bifurcation of many properties into an occurs and non-occurs flavor. E.g., once you add the above, then by analogy why not have these additional keywords which apply to the array as a whole, not the elements of it: occursAlignment occursAlignmentUnits occursInitiator occursTerminator occursLength occursLengthKind occursLengthUnits All of this is avoided by simply requiring an extra sequence to provide these same format characteristics using the ordinary non-occurs keywords, and that's very preferable to me to this proliferation of occurs and non-occurs variants. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

DFDL - debate around array specific properties
by Steve Hanson 20 Nov '07

20 Nov '07

On Friday's call Mike & Suman were tasked with creating a proposal for the number of array specific properties that we expose in 1.0. Spec v028 lists a core set: occursKind occurs occursSeparator occursStopvalue occursStopValueKind The debate was whether array properties are needed for handling prefix/suffix region, the motivating example being leading skip count. Here's a format I came across recently, JSON, meaning JavaScriptObjectModel (http://json.org/) used in conjunction with REST style web services. For a model that logically looks like... String name; String[] address; String zip; ...an example JSON instance could look like... {"name":"Steve Hanson", "address":["1 Woodlands","Awbridge","Romsey"], "zip":"SO51 0GP" } ... and a DFDL model (ignoring white space) would look like. <xs:sequence dfdl:separator="," dfdl:initiator="{" dfdl:terminator="}" dfdl:lengthKind="implicit"> <xs:element name="name" type="xs:string" dfdl:initiator=""name":" dfdl:terminator="" dfdl:lengthKind="delimited"/> <xs:sequence dfdl:separator="" dfdl:initiator=""address":[" dfdl:terminator="]" dfdl:lengthKind="implicit"> <xs:element name="address" type="xs:string" dfdl:initiator="" dfdl:terminator="" dfdl:lengthKind="delimited" maxOccurs="unbounded" dfdl:occursKind="implicit" dfdl:occursSeparator=","/> </xs:sequence> <xs:element name="zip" type="xs:string" dfdl:initiator=""name":" dfdl:terminator=""/> </xs:sequence> Note the need to wrap the address element in a sequence to consume the array markup **. I could have used a complex element, but that would have added an extra infoset level and I wanted to conform as closely as possible to the logical view of the data above. To do this with explicit array properties the DFDL model would look like: <xs:element name="address" type="xs:string" dfdl:initiator="" dfdl:terminator="" dfdl:lengthKind="delimited" maxOccurs="unbounded" dfdl:occursKind="implicit" dfdl:occursSeparator="," dfdl:occursInitiator=""address":[" dfdl:occursTerminator="]" /> Note the need for dfdl:occursInitiator, dfdl:occursTerminator. Personally, I don't think the extra properties increase the useability by much. ** The one thing that feels a bit odd is having dfdl:lengthKind="implicit" on the wrapping xs:sequence. I feel like I want to use dfdl:lengthKind="delimited", because the sequence is followed by the separator of the outer sequence. And in this instance I could, as there are no 'final unused' bytes in the data. I'm still getting used to length on sequences, I guess. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0