I agree that "end" or whatever
we decide to call it should be reserved for the last object in a sequence.
I prefer "endOfParent".
I have a general unease around the lengthKind
enum "implicit". It originally meant something quite specific,
the length was derived from the underlying xsd. That's now been extended
for text decimals to mean derived from the textNumberPattern pattern length.
And for a sequence to mean derived from the length of its children. I think
we are overloading it. I think that "implicit" should be reserved
for simple elements only, with its current semantic. And we should come
up with a new enum, reserved for complex elements or sequences only, suggest
"children" (given I have also suggested "endOfParent")
or maybe "content".
Regards
Steve Hanson
Programming Model Architect
WebSphere Message Brokers
Hursley, UK
Internet: smh@uk.ibm.com
Phone (+44)/(0) 1962-815848
| "Mike Beckerle"
<mbeckerle.dfdl@gmail.com>
28/10/2008 00:48
|
Please respond to
<mbeckerle.dfdl@gmail.com> |
|
|
To
| Steve Hanson/UK/IBM@IBMGB
|
|
cc
| Alan Powell/UK/IBM@IBMGB
|
|
Subject
| RE: endOfData - was: RE: FW: MIke's
notes from call on 2008-08-13 |
|
Not sure where this leaves us.
It is ok to reserve lengthKind="end"
or "parent" or whatever for the last element of a sequence.
...mike
Mike Beckerle | OGF DFDL
WG Co-Chair | CTO | Oco, Inc.
Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451
| mbeckerle.dfdl@gmail.com
From: Steve Hanson [mailto:smh@uk.ibm.com]
Sent: Wednesday, October 22, 2008 12:56 PM
To: mbeckerle.dfdl@gmail.com
Cc: Alan Powell
Subject: Re: endOfData - was: RE: FW: MIke's notes from call on 2008-08-13
Mike - sorry but I think users will find this baffling. lengthKind="implicit"
was intended to mean that the logical xsd provided the length. lengthKind="delimited"
means that markup provides the length. We are overloading the word "implicit"
and we are wrong to do so. Trying to wrap these together, and include "endOfData"
(as "parent") as well, is taking the abstraction too far. It
is not how people view their data.
Regards
Steve Hanson
Programming Model Architect
WebSphere Message Brokers
Hursley, UK
Internet: smh@uk.ibm.com
Phone (+44)/(0) 1962-815848
| "Mike Beckerle"
<mbeckerle.dfdl@gmail.com>
22/10/2008 15:29
|
Please respond to
<mbeckerle.dfdl@gmail.com> |
|
|
To
| Alan Powell/UK/IBM@IBMGB,
Steve Hanson/UK/IBM@IBMGB
|
|
cc
|
|
|
Subject
| endOfData - was: RE: FW: MIke's notes
from call on 2008-08-13 |
|
First: apologies for missing the call today without notice. I've been solid
on a rather urgent customer-related matter since before our meeting time
and unable to break away.
Now: w.r.t. end of data email from Steve.
In the example you highlight, the reason both children of the sequence
have lengthKind="endOfData" is that the parent is providing the
way of determining the length, in this case using delimiters. Conceptually,
the parser can carve out the box of data bytes for the first child by scanning
for the separator, and the box for the 2nd child by scanning for the terminator.
Then it can present those finite size boxes to the parser to parse each
child, and each child consumes the entire box, i.e., to the end of (its
box of) data.
However, I agree the notion of "endOfData" is confusing as I
have just explained it above.
Perhaps the right lengthKind for a child to have when the enclosing
parent has a terminator or separator is
lengthKind="parent"
which you can read conceptually as:
"length kind for this child is determined by something
specified in the parent. So you'll find nothing here about length."
We could then drop the whole "endOfData" concept entirely.
So in the example, both children would still have lengthKind="parent".
The implied "parent" of the top level is the real true "end
of the data", so a top-level element could have lengthKind="parent"
also. This is an important composition property. It allows you to take
a well specified format and drop it in as the description of an MQ message
payload, for example.
Now, lengthKind="parent" is kind of the opposite of lengthKind="implicit".
"parent" is top down, i.e., from the enclosing structure. "implicit"
is bottom up, i.e., length implied by the contents of the element.
Here's a trick that can make this all more palatable. For certain kinds
of child elements, lengthKind="implicit" will behave as lengthKind="parent".
This would happen for variable length children without any way of determining
the variable length "bottom up". Examples of this are: variable
length text strings, variable occurrances of anything (with no way to determine
how many occurrances), or ordered sequences whose final element is a variable
length child without any way of determining the variable length. (This
definition is recursive intentionally.)
Given this, I think the DFDL fragment could be:
<complexType dfdl:lengthKind="implicit"
dfdl:representation="text"
> // these are in the scope
....
<sequence dfdl:separator=”,” dfdl:terminator=”;”
dfdl:lengthKind="delimited">
<element name=”f1” type=”string” />
<element name=”f2” type=”string” />
</sequence>
....
</complexType>
Which I claim is what we want to have to write to capture the simple thing
this is trying to express, which is the format of "string1,string2;"
after all.
Comments?
BTW: notice my use of an enclosing complexType and ellipsis in order to
achieve the notion that certain property bindings surround the example.
This is one of the reasons I think we don't need a full up 2-level semantic
model as Sandy suggested. I think examples like the above are sufficiently
clear, particularly given the simplfied scoping.
Mike Beckerle | OGF DFDL
WG Co-Chair | CTO | Oco, Inc.
Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451
| mbeckerle.dfdl@gmail.com
From: Steve Hanson [mailto:smh@uk.ibm.com]
Sent: Wednesday, October 22, 2008 9:09 AM
To: Mike Beckerle
Cc: Alan Powell
Subject: Re: FW: MIke's notes from call on 2008-08-13
Hi Mike
I owe a review of the "EndOfData
Semantics" discussion below.
The only thing that looks slightly odd in the examples below is this:
<sequence dfdl:separator=”,” dfdl:terminator=”;”>
<element name=”f1” type=”string” dfdl:lengthKind=”endOfData”/>
<element name=”f2” type=”string” dfdl:lengthKind=”endOfData”/>
</sequence>
It doesn't seem right for f1 to have "endOfData". Should we have
a rule that says "endOfData" is only allowed on the last object
in a sequence? After all, that was its original - a way of the last thing
saying it is bounded by the end of its parent.
Would "endOfParent" be better than "endOfData" ?
Regards
Steve Hanson
Programming Model Architect
WebSphere Message Brokers
Hursley, UK
Internet: smh@uk.ibm.com
Phone (+44)/(0) 1962-815848
| "Mike Beckerle"
<mbeckerle.dfdl@gmail.com>
10/09/2008 14:10
|
Please respond to
<mbeckerle.dfdl@gmail.com> |
|
|
To
| Steve Hanson/UK/IBM@IBMGB
|
|
cc
|
|
|
Subject
| FW: MIke's notes from call on 2008-08-13 |
|
Mike Beckerle | OGF DFDL
WG Co-Chair | CTO | Oco, Inc.
Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451
| mbeckerle.dfdl@gmail.com
From: Mike Beckerle [mailto:mbeckerle.dfdl@gmail.com]
Sent: Friday, August 15, 2008 11:53 AM
To: dfdl-wg@ogf.com
Subject: MIke's notes from call on 2008-08-13
Only Alan Powell and myself were on the call.
These are my notes.
TOPIC: Decimal Calendar – idea: should behave as if decimal to text
then text to date/time. I.e., use same date/time pattern language, but
a subset of it since decimal can express nothing but digits.
TOPIC: Notes to authors (at start of spec) add that we don’t do scalar
type coersions/conversions generally. I.e., if the representation is a
floating point, then the logical must be a floating point. If the representation
is decimal, the logical must be decimal. We don’t allow you to have a
logical int whose rep is decimal or vice versa. Rationale: it adds complexity
that we an avoid. Doesn’t provide anything you can’t easily do another
way (layering), etc.
TOPIC: EndOfData Semantics:
We discussed that currently we were overloading the delimited concept to
include the end-of-data concept, and that was unsatisfactory and was resulting
in attempts to reinject end-of-data as “end-of-bitstream” and the like.
Points -
- Distinguish delimited to mean
we positively ARE scanning for a text pattern delimiter, and not confusing
this with the end-of-data case which is fundamentally different.
- Avoid special-case keyword only
for the “top level” end of the data stream. This has really bad composition
properties.
- lengthKind=’endOfData’ applies
to both binary and text representations. For text it means there is no
terminator for this element. The enclosing construct’s length, however
determined (separator, terminator, fixed, prefix, etc.) will bound length
of this contained element.
- Case: <sequence dfdl:separator=”,”
dfdl:terminator=”;”>
-
<element name=”f1” type=”string” dfdl:lengthKind=”endOfData”/>
-
<element name=”f2” type=”string” dfdl:lengthKind=”endOfData”/>
-
</sequence>
- The above seems ok to me.
- Case: <sequence dfdl:lengthKind=”prefixed”
dfdl:representation=”binary”>
-
<element name=”f1” type=”int” dfdl:length=”4”
dfdl:lengthKind=”explicit”>
-
<element name=”f2” type=”hexBinary”
dfdl:lengthKind=”endOfData”>
-
</sequence>
- The above seems ok to me.
Important
use cases:
Case 1: binary element at the end of a top-level sequence.
<schema …>
<element name=”theTop”>
<complexType>
<sequence dfdl:lengthKind=”implicit”>
<element name=”f1” type=”int” dfdl:length=”4” dfdl:lengthKind=”explicit”/>
<element
name=”f2” type=”hexBinary” dfdl:lengthKind=”endOfData”/>
</sequence>
</complexType>
</element>
</schema>
In the above, the top level sequence has implicit length kind. This is
ok, because the top level is assumed to be in an “end of data” context.
Case 2: deeper nesting, same implicit-length sequence.
<schema …>
<element name=”NestedInside”>
<complexType>
<sequence dfdl:lengthKind=”implicit”>
<element name=”f1” type=”int” dfdl:length=”4” dfdl:lengthKind=”explicit”/>
<element
name=”f2” type=”hexBinary” dfdl:lengthKind=”endOfData”/>
</sequence>
</complexType>
</element>
<element name=”stillNotTheTop”>
<complexType>
<sequence dfdl:lengthKind=”implicit”>
…
<element
ref=”NestedInside”/>
</sequence>
</complexType>
</element>
<element name=”hasFixedLength”>
<complexType>
<sequence dfdl:lengthKind=”explicit”
dfdl:length=”100”>
…
<element
ref=”stillNotTheTop”/>
</sequence>
</complexType>
</element>
….
</schema>
This case illustrates how the composition properties work for explicit/implicit
lengths.
The definition of how this works goes something like this.
When the last element of a sequence is binary with lengthKind=”endOfData”
this implies that the enclosing sequence is:
(a) length
kind explicit or prefixed or endOfdata
(b) length
kind implicit – in this case recursively this enclosing sequence must
itself be enclosed in a sequence similarly constrained on length kind (cases
a, b, c here)
(c) the
top-level sequence
Note: We need to revisit whether the name ‘endOfData’ is desirable or
not. There’s a list of alternatives from the F2F meeting. Problem is that
a naïve user will be thinking “top level” but the concept actually needs
to be compositional/nestable.
TOPIC: float/double – we concluded that until XML has floating point
types that can handle extended precisions that DFDL can’t handle extended
precisions in any reasonable way, so we should simply say DFDL v1.0 supports
only 64-bit floating point precision and 32 bit floating point precision.
This narrows down float types to IEEE (single and double), and IBM390 (single
and double), and maybe AS400 if that’s different and still within 64 bits
precision.
Mike Beckerle | OGF DFDL
WG Co-Chair | CTO | Oco, Inc.
Tel: 781-810-2100 | 504 Totten Pond Road, Waltham MA 02451
| mbeckerle.dfdl@gmail.com
Unless stated otherwise
above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU