- dfdl-wg - lists.ogf.org

Dfdl information set 7
by Simon Parker 12 Nov '07

12 Nov '07

Good morning. Version 7 of the proposal incorporates the outcome of last week's conference and subsequent email discussion. If I've overlooked or misunderstood any amendments, please let me know. Regards, Simon Simon Parker Software Consultant PolarLake 80 Harcourt Street Dublin 2, Ireland Phone +353 1 449-1075 Fax +353 1 449-1011 Web http://www.polarlake.com/ The information transmitted in this email is intended for the addressee only and may contain confidential and/or privileged material. Any review, retransmission, dissemination, reliance upon or other use of this information by persons or entities other than the addressee is prohibited. If you think, for any reason, that this message may have been addressed to you in error, we would ask you to notify the sender immediately by return email and delete the material. PolarLake Limited | Registered in Dublin, Ireland | Number 357324 | Registered office as above.

1 0

DFDL: *CORRECTED* Minutes from OGF WG call, 7 Nov 2007
by Ian W Parkinson 09 Nov '07

09 Nov '07

One correction in the revised minutes below: some duplicate text removed from para 2.1. With thanks to Simon for spotting the error. Ian Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 7 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Suman Kalia (IBM) Simon Parker (PolarLake) Ian Parkinson (IBM) 1. Introduction The previous meeting left two areas for consideration: the Information Set (InfoSet) proposal, and the work regarding defaulting and nulls. This meeting would cover a review of a revised InfoSet proposal. The defaults/nulls issue has not been progressed. 2. Review of Revised Information Set Proposal Steve and Simon have collaborated on a revision, v6, of the InfoSet proposal. This has been circulated via the DFDL-WG mailing list. The meeting reviewed the v6 proposal, highlighting the following points: 1. There remain two types of information item: Element Information Items and Document Information Items. There are now two categories of Element Information Item: simple, and complex. 2. The description of Document Information Items incorrectly lists a "root" member. This should be replaced by a "children" member. Geoff pointed out that the associated UML diagram suggests that a Document Information Item may contain multiple children; this will be restricted in the text. 3. The "dfdl version" member is now used to describe the version of DFDL to which the InfoSet conforms; it does not describe the version of the parser generating the InfoSet. In particular, for an InfoSet to inlcude a dfdl version identifier, there is now no requirement for that InfoSet to have been generated by a DFDL parser. 4. Mike queried the meaning of "absolute Schema Component Descriptor"; it should identify a schema as a whole. This is a term defined by the SCD specification, and an appropriate citation should be added to the proposal. 5. The meeting discussed, at length, the "data value" member of Element Information Item, which the proposal lists as being of type String. Mike pointed out that this is insufficient to distinguish between some binary floating-point values, and suggested that it should be of a type defined by the "data type" member. Suman suggested this would also make for a more natural API. Simon remarked on the precedent set by XML Schema where all values are represented as strings. Mike closed the discussion advocating dropping the word "String" from the definition of the "data value" member. 6. Suman queried the purpose of the document information item. This is an in-memory artifact only, which acts as a container of the InfoSet's data, and has no representation in a serialization of the InfoSet. This should be clarified in the proposal. 7. The proposal now contains a UML diagram illustrating the object model. The relationship between Parent and Element requires some clarifying: it should be modelled as a composite containment relationship. 8. Simon highlighted that the "schema" member of Element Information Item may be either a relative or an absolute SCD; while the "schema" member of Document Information Item must be an absolute SCD. 9. The current version makes no mention of arrays, preferring to treat arrays as repeated children with the same name. The meeting consdered how to denote arrays in the InfoSet, but opted instead to add a note regarding arrays to the specification. Mike will circulate his notes regarding the agreed changes to the spec. Simon will circulate a new version, v7, incorporating further comments. 3. Other Business There was no other business. Meeting closed Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, 7 Nov 2007
by Ian W Parkinson 08 Nov '07

08 Nov '07

Hi all, Steve and Mike have asked me to record minutes at the regular DFDL-WG conference calls. Here are the minutes from yesterday's meeting - I'd obviously welcome any suggestions you may have on the format or style of these minutes. Thanks, Ian Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 17:00 GMT, 7 Nov 2007 Attendees Mike Beckerle (IBM) Geoff Judd (IBM) Steve Hanson (IBM) Suman Kalia (IBM) Simon Parker (PolarLake) Ian Parkinson (IBM) 1. Introduction The previous meeting left two areas for consideration: the Information Set (InfoSet) proposal, and the work regarding defaulting and nulls. This meeting would cover a review of a revised InfoSet proposal. The defaults/nulls issue has not been progressed. 2. Review of Revised Information Set Proposal Steve and Simon have collaborated on a revision, v6, of the InfoSet proposal. This has been circulated via the DFDL-WG mailing list. The meeting reviewed the v6 proposal, highlighting the following points: There remain two types of information item: Element Information Items and Document Information Items. Element Information Items. There are now two categories of Element Information Item: simple, and complex. The description of Document Information Items incorrectly lists a "root" member. This should be replaced by a "children" member. Geoff pointed out that the associated UML diagram suggests that a Document Information Item may contain multiple children; this will be restricted in the text. The "dfdl version" member is now used to describe the version of DFDL to which the InfoSet conforms; it does not describe the version of the parser generating the InfoSet. In particular, for an InfoSet to inlcude a dfdl version identifier, there is now no requirement for that InfoSet to have been generated by a DFDL parser. Mike queried the meaning of "absolute Schema Component Descriptor"; it should identify a schema as a whole. This is a term defined by the SCD specification, and an appropriate citation should be added to the proposal. The meeting discussed, at length, the "data value" member of Element Information Item, which the proposal lists as being of type String. Mike pointed out that this is insufficient to distinguish between some binary floating-point values, and suggested that it should be of a type defined by the "data type" member. Suman suggested this would also make for a more natural API. Simon remarked on the precedent set by XML Schema where all values are represented as strings. Mike closed the discussion advocating dropping the word "String" from the definition of the "data value" member. Suman queried the purpose of the document information item. This is an in-memory artifact only, which acts as a container of the InfoSet's data, and has no representation in a serialization of the InfoSet. This should be clarified in the proposal. The proposal now contains a UML diagram illustrating the object model. The relationship between Parent and Element requires some clarifying: it should be modelled as a composite containment relationship. Simon highlighted that the "schema" member of Element Information Item may be either a relative or an absolute SCD; while the "schema" member of Document Information Item must be an absolute SCD. The current version makes no mention of arrays, preferring to treat arrays as repeated children with the same name. The meeting consdered how to denote arrays in the InfoSet, but opted instead to add a note regarding arrays to the specification. Mike will circulate his notes regarding the agreed changes to the spec. Simon will circulate a new version, v7, incorporating further comments. 3. Other Business There was no other business. Meeting closed Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL Information Set proposal 6
by Simon Parker 07 Nov '07

07 Nov '07

Good afternoon. Steve and I have collaborated to compose this revised proposal (dfdl-infoset-proposal.6.doc) for the DFDL information set. I attached the diagram source representation (dfdl_infoset.3.uxf) so anyone can produce fresh versions. I used a simple, free Eclipse plugin called 'Umlet <http://www.umlet.com/> '. Comments and questions are welcome, as ever. Simon Simon Parker Software Consultant PolarLake 80 Harcourt Street Dublin 2, Ireland Phone +353 1 449-1075 Fax +353 1 449-1011 Web http://www.polarlake.com/ The information transmitted in this email is intended for the addressee only and may contain confidential and/or privileged material. Any review, retransmission, dissemination, reliance upon or other use of this information by persons or entities other than the addressee is prohibited. If you think, for any reason, that this message may have been addressed to you in error, we would ask you to notify the sender immediately by return email and delete the material. PolarLake Limited | Registered in Dublin, Ireland | Number 357324 | Registered office as above.

1 0

DFDL: Minutes from OGF WG call 2007-10-31
by Mike Beckerle 01 Nov '07

01 Nov '07

OGF DFDL WG Call 2007-10-31 Discussed: Separator semantics via rewriting - Alan Powell (prior to call) raised objections to MikeB's treatment of separator properties semantics in draft of nulls/optionals/defaults memo. Mike agrees. This stuff is a mess. Needs to be reworked. Action item: Mike to rework. Discussed: Boxed data, i.e., sequences with dfdl:length specifications as means of specifying box size surrounding dfdl-described content. Proposed to drop this from DFDL V1.0. Action: examine important formats to see if this feature is needed. We know there are formats that use it (IMS blocks), but must they be supported in single schemas or is a two-layer via two schemas approach acceptable? Resolved: not dropping this. New simpler examples are needed which make this clearer than the example in the email/memo by Mike. (Note: this is the whole section on valueCalc properties. This has already been identified as an area needing a write up.) Discussed: does infoset include things like the representation length of items? This information is needed to express output for formats where one element stores the rep-length of another. Suman pointed out that this information doesn't need to be available in the infoset. The relationships between elements of many kinds cannot be expressed directly in terms of the infoset, so why require this representation-level relationship to be so expressed? Resolved: infoset does not contain represnetation artifacts like representation length. Mike mentioned that it does contain number of occurrences for arrays. However, later discussion of infoset brings into doubt whether the infoset even knows about arrays. Discussed: infoset Metadata link? Suman has advocated for this. It's an optional thing. SCD paths are acceptable to most. Simon maintained that this shouldn't be part of the infoset but really is an implementation consideration. UML model: Discussed that it should match XML infoset terminology and also the grammar terminology used in the spec. (note: or the grammar terminology be made to conform to the infoset - actually this is more likely) Simon's model has arrays as distinct from sequence groups. We discussed eliminating arrays as a concept and going to a neutral term that captures both sequence groups and arrays (things with children). Action: Steve H. and Simon P. to work out remaining infoset issues. Provide UML diagram. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

Pseudocode in null-default-optional proposal
by Simon Parker 31 Oct '07

31 Oct '07

I said I would try to simplify this pseudocode in section 6.2 of the proposal: for $child in original sequence children declarations for $i = 1 to length of array if array[$i].name = $child.name // including namespace comparison then add array[$i] to $matches if array[$i].name != $child.name // including namespace comparison then break; // stop at first non-match assert( $child.minOccurs <= $matches.size ) // processing error otherwise assert( $child.maxOccurs >= $matches.size ) // ditto append $matches to $result return $result I came up with this: int $index = 1 for each $child in schema int $count = 0 while $index <= array.size and array[$index].name = $child.name increment $index increment $count end assert( $child.minOccurs <= $count ) // processing error otherwise assert( $child.maxOccurs >= $count ) // ditto end return array ... but I'm not convinced this programming exercise added to the readability of the specification. I'll keep trying to find a better style, but I'm not sure I understand it well enough yet. Simon Simon Parker Software Consultant PolarLake 80 Harcourt Street Dublin 2, Ireland Phone +353 1 449-1075 Fax +353 1 449-1011 Web http://www.polarlake.com/ The information transmitted in this email is intended for the addressee only and may contain confidential and/or privileged material. Any review, retransmission, dissemination, reliance upon or other use of this information by persons or entities other than the addressee is prohibited. If you think, for any reason, that this message may have been addressed to you in error, we would ask you to notify the sender immediately by return email and delete the material. PolarLake Limited | Registered in Dublin, Ireland | Number 357324 | Registered office as above.

1 0

DFDL infoset
by Simon Parker 31 Oct '07

31 Oct '07

Good afternoon. Steve sent me the infoset draft for review. I considered it along with the embedded comments, and felt a slightly different approach might make our intention clearer and allow more rigour with fewer words. I offer this alternative starting point for further discussion and refinement; if it doesn't find favour I'll go back and refine the original. Hope this helps, Simon ________________________________ From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: 24 October 2007 18:52 To: Simon Parker Subject: Fw: DFDL infoset - please review Hi Simon - draft DFDL infoset attached - doesn't cover the repeating groups 'wrinkle' discussed on the call. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 24/10/2007 18:06 ----- Mike Beckerle/Worcester/IBM@IBMUS 23/10/2007 15:18 To Steve Hanson/UK/IBM@IBMGB cc Subject Re: DFDL infoset - please reviewLink <Notes://D06ML070/802563ED00496501/38D46BF5E8F08834852564B500129B2C/96EA A4070FF5C1598025737D0030858E> It's ok for now. We either need to explain an "array info item" or be comfortanble that array vs. scalar is explained elsewhere. "belongs to an array" should be "belongs to an array of information items" I think so as to make it clearer that there isn't a special "array", but rather these are ... just arrays of information items. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Steve Hanson/UK/IBM@IBMGB 10/23/2007 04:52 AM To Mike Beckerle/Worcester/IBM@IBMUS cc Subject Re: DFDL infoset - please reviewLink <Notes://D01ML259/85256FDB00077D54/38D46BF5E8F08834852564B500129B2C/4028 822A95E0C7838525737C0062C1B2> Thanks Mike. Regarding your array comment, you look to be saying that while it begs the question about an array info item, we are probably ok without one. Correct? I used Sandy's phrase "belongs to an array" rather than "is an array" - perhaps the latter is better? Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle/Worcester/IBM@IBMUS 22/10/2007 18:59 To Steve Hanson/UK/IBM@IBMGB cc Subject Re: DFDL infoset - please reviewLink <Notes://D06ML070/802563ED00496501/38D46BF5E8F08834852564B500129B2C/24EC 934A59422E8C8025737C005D4D16> 3 additional comments and 2 typographicals. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Steve Hanson/UK/IBM@IBMGB 10/22/2007 01:07 PM To Suman Kalia/Toronto/IBM@IBMCA, Mike Beckerle/Worcester/IBM, Sandy Gao/Toronto/IBM, Alan Powell/UK/IBM, Peter Lambros, Ashutosh Arora/Burlingame/IBM, Catalin Bucsa/Boca Raton/IBM, Geoff Judd/UK/IBM, Ian W Parkinson/UK/IBM cc Subject DFDL infoset - please review Comments from Mike, Sandy and Alan incorporated. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ________________________________ Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

DFDL OGF WG Call 2007-10-31 - Agenda
by Mike Beckerle 31 Oct '07

31 Oct '07

Remember our call today is 1pm Eastern time (us and canada) because of the daylight savings time change in the UK. ---------------------------- We have two topics suggested for discusssion today. 1) separator treatment in latest circulated proposal-to-simplify-nulls.... doc/memo - specifically, it is both buggy and unclear. How to fix. Who to fix? 2) boxed data. I.e., sequences with length as a way to describe a box of some size that is a container for data held in the box. This is in response to a note circulated to this WG before. Excerpts below. Specifically: should we punt this for v1.0. It is after all, a kind of layering, and we punted on layering largely for v1.0. -------------------------- Conclusion: It does appear that we need outputLengthCalc, which is tantamount to Steve's concerns that we need input and output variants of many properties. We need to distinguish input length and output length. In the above example, dfdl:length is input length, and dfdl:outputLengthCalc is the property name I'm using for an output length. Perhaps better naming conventions would be Use dfdl:length when it's symmetric, dfdl:inputLength and dfdl:outputLength when it's asymetric. Logical value comes from the representation when parsing unless dfdl:inputValue (formerly dfdl:inputValueCalc) in which case the logical value comes from that expression. Representation comes from the logical value when unparsing unless dfdl:outputValue is provided (formerly dfdl:outputValueCalc), in which case representation comes from that computed value instead. --------------------------- Also, please could you illustrate the issue using a much simpler example than the box array, eg, a variable length string where its length is given by a preceding integer field. Length on input is given by the integer, length on output is given by the actual data. Value of integer is as supplied on input, value on output is the length of the string. I want to see whether we really need input & output length properties for a more typical scenario. Personally I think we should drop the use of dfdl:length on a sequence for 1.0 period. That precludes support of box arrays, but I don't have a problem with that as I don't have a real-life use case. (I would say however that if the only way to model a box array is as below, then we are asking an awful lot from our audience to be able to create such a model.) Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 20/09/2007 15:02 To Alan Powell/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Subject [DFDL-WG] output value and length (was Re: Fw: Notes from 2007-09-12 call) We have many use cases to work out for the output direction. E.g., consider a string in utf-8 characters, stored in a box which must be of N "words" long, i.e., length will be a multiple of 4 bytes long. Now suppose we have to store the length of the box measured in number of words, in a field L1. The String is S1. Some of this stuff might want to be hidden in a real schema, but let's ignore that for now. So, one might model this without DFDL as: <sequence> <element name="L1" type="int" /> <element name="box"> <complexType> <sequence id="box"> <element name="S1" type="string" /> </sequence> </complexType> </element> </sequence> So we have the length, a box surrounding the string, and the string S1 itself. Now we want to annotate this for input parsing. I'm going to leave off all the dfdl:applies properties to save space: <sequence> <element name="L1" type="int" dfdl:length="4" dfdl:lengthUnits="bytes" dfdl:byteOrder="bigEndian" dfdl:representation="binaryInteger" /> <element name="box"> <complexType> <sequence dfdl:length="{ ../L1 * 4 }" dfdl:lengthUnits="bytes"> <element name="S1" type="string" dfdl:encoding="utf-8" dfdl:length="fillAvailableSpace" /> </sequence> </complexType> </element> </sequence> So far so good. The sequence's length is L1 * 4, and the string fills the space in that sequence. Now we want to annotate it for output/unparse. First we put in outputValueCalc on L1. This seems ok. <sequence> <element name="L1" type="int" dfdl:length="4" dfdl:lengthUnits="bytes" dfdl:byteOrder="bigEndian" dfdl:representation="binaryInteger" dfdl:outputValueCalc="{ cieling(../box.length(), 4) }" /> <element name="box"> <complexType> <sequence dfdl:length="{ ../L1 * 4 }" dfdl:lengthUnits="bytes"> <element name="S1" type="string" dfdl:encoding="utf-8" dfdl:length="fillAvailableSpace" /> </sequence> </complexType> </element> </sequence> The above however appears to be circularly defined. The length of the sequence inside the box element is defined in terms of the value of L1, and the output value of L1 is defined in terms of the length of element box. So really we need to distinguish input length and output length calculations. So it seems we need dfdl:outputLengthCalc="{ cieling(S1.length('bytes'), 4) * 4 }" as an additional rep prop on the box sequence. Notice how we've had to ask for the length to be presented in a particular kind of units, and the cieling and multiply trick rounds up to a multiple of 4 in size. <sequence> <element name="L1" type="int" dfdl:length="4" dfdl:lengthUnits="bytes" dfdl:byteOrder="bigEndian" dfdl:representation="binaryInteger" dfdl:outputValueCalc="{ cieling(../box.length(), 4) }" /> <element name="box"> <complexType> <sequence dfdl:length="{ ../L1 * 4 }" dfdl:lengthUnits="bytes" dfdl:outputLengthCalc="{ cieling(S1.length('bytes'), 4) * 4 }"> <element name="S1" type="string" dfdl:encoding="utf-8" dfdl:length="fillAvailableSpace" /> </sequence> </complexType> </element> </sequence> But now we still have an issue, which is that the length of S1 on output might need to be enlarged with padding characters because the output length of the box is being rounded up to a multiple of 4 bytes. One idea for how to solve this is to use layers. I.e, we need another string S2 because we can't get all the description we need onto just the string S1. <sequence> <element name="L1" type="int" dfdl:length="4" dfdl:lengthUnits="bytes" dfdl:byteOrder="bigEndian" dfdl:representation="binaryInteger" dfdl:outputValueCalc="{ cieling(../box.length(), 4) }" /> <element name="box"> <complexType> <sequence dfdl:length="{ ../L1 * 4 }" dfdl:lengthUnits="bytes" dfdl:outputLengthCalc="{ cieling(S1.length('bytes'), 4) * 4 }"> <element name="S2" type="string" dfdl:encoding="utf-8" dfdl:length="fillAvailableSpace" dfdl:outputValueCalc="{ ../../S1 }" dfdl:padCharacter=" " /> </sequence> </complexType> </element> <element name="S1" type="string" dfdl:inputValueCalc="{ ../box/S2 }" /> </sequence> The above we have S2, which is the string that really lives in the representation. Now hiding the rep stuff and making it into a reusable type definition: <complexType name="wordLengthStringType"> <sequence> <annotation><appinfo><dfdl:hidden> <element name="rep"> <complexType> <sequence> <element name="L1" type="int" dfdl:length="4" dfdl:lengthUnits="bytes" dfdl:byteOrder="bigEndian" dfdl:representation="binaryInteger" dfdl:outputValueCalc="{ cieling(../box.length(), 4) }" /> <element name="box"> <complexType> <sequence dfdl:length="{ ../L1 * 4 }" dfdl:lengthUnits="bytes" dfdl:outputLengthCalc="{ cieling(../../../S1.length('bytes'), 4) * 4 }"> <element name="S2" type="string" dfdl:encoding="utf-8" dfdl:length="fillAvailableSpace" dfdl:outputValueCalc="{ ../../../S1 }" dfdl:padCharacter=" " /> </sequence> </complexType> </element> </sequence> </complexType> </element> </dfdl:hidden></appinfo></annotation> <element name="S1" type="string" dfdl:inputValueCalc="{ ../rep/box/S2 }" /> </sequence> </complexType> Now to use it: <element name="myString" type="wordLengthStringType"/> Logical expression myString/S1 is the string's value. (Probably should rename the element "S1" to "value" so this would be myString/value) In DFDL v1.0 as currently defined, we do not have any way to make this into a "real string type", because we don't provide a way to define a complex type as the representation of a simple type. That's ok. We can consider that later. Conclusion: It does appear that we need outputLengthCalc, which is tantamount to Steve's concerns that we need input and output variants of many properties. We need to distinguish input length and output length. In the above example, dfdl:length is input length, and dfdl:outputLengthCalc is the property name I'm using for an output length. Perhaps better naming conventions would be Use dfdl:length when it's symmetric, dfdl:inputLength and dfdl:outputLength when it's asymetric. Logical value comes from the representation when parsing unless dfdl:inputValue (formerly dfdl:inputValueCalc) in which case the logical value comes from that expression. Representation comes from the logical value when unparsing unless dfdl:outputValue is provided (formerly dfdl:outputValueCalc), in which case representation comes from that computed value instead. We also need the expression language to be able to ask what the length of the representation of an element is, measured in whatever units we need. We may need to be able to ask for the inputLength and the outputLength separately. -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Comments on lengths <awp> below </awp> Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Steve Hanson/UK/IBM@IBMGB Sent by: dfdl-wg-bounces(a)ogf.org 19/09/2007 14:30 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Fw: Notes from 2007-09-12 call More on expressions, <smh>below</smh> Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/09/2007 14:14 ----- Mike Beckerle <beckerle(a)us.ibm.com> 19/09/2007 13:43 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Subject Re: [DFDL-WG] Notes from 2007-09-12 call Comments below in BLUE Steve Hanson <smh(a)uk.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 09/19/2007 06:04 AM To dfdl-wg(a)ogf.org cc Subject Re: [DFDL-WG] Notes from 2007-09-12 call Some thoughts since last week's call: 1) Expression language We've not thought much about how expressions will work on output. It's fine to say something like dfdl:length="..\count+1" when parsing, but what happens on output. I think we should not try to reverse engineer expressions, and rely on the user to set output fields correctly. So, taking my example, on output we would assume count had been set by the user, apply the expression to calculate the intended length of data, then apply padding etc rules as needed. Can we generalise that philosophy across all our uses of expressions? If we can't then perhaps that places a bound on the actual uses of expressions that we permit. Inverting will generally not be possible. Just make the example dfdl:length="{ ../count * ../scale + 1 }" How do you split up the length into count and scale? <smh>Agree</smh> In your example, I would expect the count field to have an outputValueCalc="{ ../x.length() - 1 }" (I'm assuming the field with the length calculation formula is named "x".) <smh>Doesn't outputValueCalc mean that we are deriving the count from the length of the data supplied for x? That forces the user to pad x to the correct value, in order to derive count. Which is not how we want things to work. We want count to define the length of x, so the DFDL serialiser can pad x according to other DFDL properties. Maybe I'm missing something about input/outputValueCalc?</smh> <awp> There are multiple cases to consider. There are some formats that require the length field to be the physical length of a structure, ie including padding, code page considerations, etc that it is impossible for the user to know. For example IMS transaction header has LLbbHeaderData. DFDL should fill in these lengths. I would assume that in most cases a field with it's length in other field is variable length but again it may need to be the physical length. I tend to agree with Mike that outvaluecalc can be used to set the length field but how do we get to ignore the dfdl:length specification on the data field? I hope this doesn't mean that we need to distinguish between logical and physical lengths in the expression language. </awp> In general, when something uses something else in it's calculation (length or just the value - inputValueCalc), then the inverse is outputValueCalc on the contributing parts. 2) dfdl:length for sequences We have three cases here: a) Empty sequence - we agreed to disallow this b) Non-empty normal sequence - what does the length mean here? It means the box is potentially larger than the contents. If it isn't at least as big it's an error. If these lengths are data dependent it could be a processing error. Otherwise a schema definition error. Draft 025 discusses this in the part on sequences with length. We also could disallow this case if we want for now, knowing we could add it back if we want. One can always convert this case into the one below by wrapping the sequence's child elements in an array element, with array occurrences determined by "fillAvailableSpace" policy. If we allow this at all, I think this should be the way we explain the semantics of it. (Though with the inserted array the paths would all change which is undesirable. - so we would say it works like this, but without the paths being changed...) c) Non-empty sequence used as box array - the motivating scenario I think we should also disallow b). If we are disallowing a) on the grounds of not using sequence with a length to model opaque data then we should also disallow b). Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

Fw: Notes from 2007-09-12 call
by Steve Hanson 31 Oct '07

31 Oct '07

More on expressions, <smh>below</smh> Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/09/2007 14:14 ----- Mike Beckerle <beckerle(a)us.ibm.com> 19/09/2007 13:43 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Subject Re: [DFDL-WG] Notes from 2007-09-12 call Comments below in BLUE Steve Hanson <smh(a)uk.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 09/19/2007 06:04 AM To dfdl-wg(a)ogf.org cc Subject Re: [DFDL-WG] Notes from 2007-09-12 call Some thoughts since last week's call: 1) Expression language We've not thought much about how expressions will work on output. It's fine to say something like dfdl:length="..\count+1" when parsing, but what happens on output. I think we should not try to reverse engineer expressions, and rely on the user to set output fields correctly. So, taking my example, on output we would assume count had been set by the user, apply the expression to calculate the intended length of data, then apply padding etc rules as needed. Can we generalise that philosophy across all our uses of expressions? If we can't then perhaps that places a bound on the actual uses of expressions that we permit. Inverting will generally not be possible. Just make the example dfdl:length="{ ../count * ../scale + 1 }" How do you split up the length into count and scale? <smh>Agree</smh> In your example, I would expect the count field to have an outputValueCalc="{ ../x.length() - 1 }" (I'm assuming the field with the length calculation formula is named "x".) <smh>Doesn't outputValueCalc mean that we are deriving the count from the length of the data supplied for x? That forces the user to pad x to the correct value, in order to derive count. Which is not how we want things to work. We want count to define the length of x, so the DFDL serialiser can pad x according to other DFDL properties. Maybe I'm missing something about input/outputValueCalc?</smh> In general, when something uses something else in it's calculation (length or just the value - inputValueCalc), then the inverse is outputValueCalc on the contributing parts. 2) dfdl:length for sequences We have three cases here: a) Empty sequence - we agreed to disallow this b) Non-empty normal sequence - what does the length mean here? It means the box is potentially larger than the contents. If it isn't at least as big it's an error. If these lengths are data dependent it could be a processing error. Otherwise a schema definition error. Draft 025 discusses this in the part on sequences with length. We also could disallow this case if we want for now, knowing we could add it back if we want. One can always convert this case into the one below by wrapping the sequence's child elements in an array element, with array occurrences determined by "fillAvailableSpace" policy. If we allow this at all, I think this should be the way we explain the semantics of it. (Though with the inserted array the paths would all change which is undesirable. - so we would say it works like this, but without the paths being changed...) c) Non-empty sequence used as box array - the motivating scenario I think we should also disallow b). If we are disallowing a) on the grounds of not using sequence with a length to model opaque data then we should also disallow b). 3) Handling of comments in data Mentioning this as it was discussed on the call but not minuted. Example: A record is allowed to be followed by multiple free form text lines where the first such line contained //ADDINFOSTART, and the last such line contained //ADDINFOEND. This was put forward as a use case for regular expressions but it was noted that an explicit dfdl:commentScheme based on, or maybe an extension of, the existing dfdl:escapeSchema property would be a more natural solution for users Whooops. Forgot this. We need a concrete proposal here. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 12/09/2007 21:20 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Notes from 2007-09-12 call Mike Beckerle, Alan Powell, Steve Hanson, Suman Kalia attended. Discussed these questions from Alan about expression language. 1. Accessing hidden values - it seems inconsistent to allow access to hidden values when xpath is used within the DFDL domain but not when used outside. 2. Where xpath is allowed in the schema - It is currently allowed in an arbitrary set of properties (initiator, terminator, separator, occurseparator, null, etc ). Why not allow it everywhere? Wr.t. (1) we decided this is correct. path expressions for dfdl properties can see hidden elements, path expressions in other places (e.g., schematron assertions) cannot. Wr.t (2) we decided that expressions should be allowed in principle everywhere for the value of any property; however, there may be exceptions for certain properties. Particularly, it seems some enum-valued properties are unlikely to ever want to be expressions. Example: dfdl:representation. However, it was also pointed out that once we put selectors back into the language you can interleave multiple formats in the same schema, and for any enumerated property you could just have one selector-chosen format for each possible value of the enumerated property. The reason we don't want a blanket statement that you can have expressions anywhere you need a property value is that there is some potential that this makes implementations unnecessarily complex due to the excess flexibility. Digression: (This added by MikeB - was not part of the call today.) Consider dfdl:byteOrder=" if (../../x = 'B') then 'bigEndian' else if (../../x='L') then 'littleEndian' else 'I don't know' }" DFDL implementations must be prepared to cope with recieving "I don't know" as the proposed value for the byteOrder. This is a schema definition error, but it is happening at run time so becomes a processing error. The only way to rule this out is to treat enumerated property values not as strings but as an enum type and force the expressions that compute them to return an enum type, not a string. This is a kind of type inference I had hoped implementations would not need. Selectors have the advantage of being statically verifiable. i.e., each selected format is known to use a value of the enum that is valid or a diagnostic could be issued by the DFDL processor. If we allow an arbitrary expression to return the value of an enumerated property then it presumably could also return a nonsense value: We discussed proposals circulated by MikeB: Here's an update to the first one. We decided sequences shouldn't be another way to carry opaque data. Easy and conservative way to fix this is to require the length of an empty sequence to be zero. Second proposal to eliminate hexBinary and base64Binary was discussed lightly. It was suggested that one could have both, and that would make it easy to explain what the hexBinary type is, because it is a shorthand for a string with encoding="hex", and similarly for base64Binary. We did not resolve this issue on the call. Finally, we discussed regular expression features for DFDL. There does appear to be need for regexp features to support parsing data which is delimited by changing data content. E.g. consider "12345Mike Beckerle". and a two-element sequence. One is a number which continues until the first non-digit character. The other is a string which begins with a non-digit character. Regexp length appears to be a good way to handle this kind of thing. Alan Powell has the action item to talk with the IBM internal TX product group. They have a speculative parser and so have fewer regular-expression features in their language. We want to understand how they deal with the header, body[], trailer use case. This case is where the data is lines of text, the header is the first line, the trailer is the last line, the body records are everything in between and there's no content that can be used to distinguish the record types. This is handled in some format-description systems with regexp features. In TX this is handled by speculative parsing and we want to understand how this comes out and if it is preferable to adding regexp features. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 4

Material for Call on 2007-10-24
by Mike Beckerle 31 Oct '07

31 Oct '07

We had been reviewing the facets and default/null/optional a few weeks back. I said I would make revisions as a number of people had objections. Here are documents based on these comments. Main things I did: (1) strict adherence to minOccurs and maxOccurs. No more notion that these are only for validity checking. (2) simple rule for default values: if it's required and there's a default value, then use the default value. This is for both input and output. I put a bunch of work into illustrating the semantics of things like dfdl:separatorPolicy="suppress" and so forth. I had been trying to simplify this, but gave up. We're back to 3 policies here: keep, suppress, suppressAtEnd. I did figure out a way to express these via DFDL rewriting. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

3 2