Fw: Action : 088 define semantics of choiceKind 'fixedLength'

6 Jul 2010

      Forwarding to DFDL WG mailing list in anticipation of discussion on this 
week's call.

Regards

Steve Hanson
Strategy, Common Transformation & DFDL
Co-Chair, OGF DFDL WG
IBM SWG, Hursley, UK,
smh@uk.ibm.com,
tel +44-(0)1962-815848
----- Forwarded by Steve Hanson/UK/IBM on 06/07/2010 10:46 -----

From:
Steve Hanson/UK/IBM
To:
Suman Kalia/Toronto/IBM@IBMCA
Cc:
Alan Powell/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB
Date:
05/07/2010 16:06
Subject:
Re: Action : 088 define semantics of choiceKind 'fixedLength' 

We discussed this briefly on the last DFDL OGF WG call.  The use case we 
are trying to support is where a programming language has reserved memory 
for the longest branch of the choice.

I would love to be able to drop the dfdl:choiceKind property and instead 
use the dfdl:length property of the parent element of the choice, set from 
the TD size attribute.  In other words, we do nothing special for choices; 

- on input we expect dfdl:length bytes or characters, we parse the chosen 
branch, if it is short we then skip to the end
- on output we serialise the chosen branch, if it is short we then pad out 
to dfdl:length bytes or characters using dfdl:fillByte. 

But this has some limitations...
a) It only works with choices that are the content model of a complex type
b) It forces the DFDL modeller to calculate the length of the choice

While b) is not a problem in practice because the importer sets it and not 
the user, unfortunately a) is, because the importers today generate a 
xs:group ref to a global choice group, so there is nowhere to carry the 
dfdl:length property.  I think the alternatives are:

i) Change the importer to add a parent element for each group. Con: This 
changes the logical model by inserting an extra level. 
ii) Allow dfdl:length to be carried on embedded xs:choice (along with 
dfdl:lengthUnits & dfdl:lengthKind). 
iii)  New dfdl:choiceLength property to be carried on embedded xs:choice 
(always bytes). 
iv) dfdl:choiceKind as today. Con: length from TD model is lost and must 
be recalculated.
v) dfdl:choiceKind but with limitations to make the calculation easier. 
Con: length from TD model is lost and must be recalculated.

The DFDL bottom-up 'model the data' principle would point at i) but there 
are compatibility issues with existing uses of the importer. That leaves 
iii) and v) as main contenders. It seems a real shame to throw away the 
length that the TD model has calculated, so I think we should discuss 
iii).

Regards

Steve Hanson
Strategy, Common Transformation & DFDL
Co-Chair, OGF DFDL WG
IBM SWG, Hursley, UK,
smh@uk.ibm.com,
tel +44-(0)1962-815848

From:
Suman Kalia/Toronto/IBM@IBMCA
To:
Alan Powell/UK/IBM@IBMGB, Steve Hanson/UK/IBM@IBMGB, Tim 
Kimber/UK/IBM@IBMGB
Date:
12/05/2010 16:25
Subject:
Re: Action : 088 define semantics of choiceKind 'fixedLength' 

See the updated contentSize() function in the attached mxsd file. It takes 
the largest size value for the choice branch; in this case element x of 
size 20. 

      01   A.
           05  b  pic 9(3).
           05  c  occurs 1 to 10 times depending on b.
               10  x    pic x(20).
               10  y redefines x.
                   20  d    pic x(5).
                   20  e    pic 9(3). 

<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:xsd="
http://www.w3.org/2001/XMLSchema">
  <xsd:complexType name="A">
    <xsd:sequence>
      <xsd:element name="b">
        <xsd:annotation>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfSimpleRep accessor="readWrite" addrUnit="byte" 
alignment="byte" attributeInBit="false" contentSize="3" offset="0" 
signFormat="trailing" signed="false" size="3" 
typeName="MRCWFExternalDecimalRep" width="3"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
        <xsd:simpleType>
          <xsd:restriction base="xsd:short">
            <xsd:minInclusive value="0"/>
            <xsd:maxInclusive value="999"/>
          </xsd:restriction>
        </xsd:simpleType>
      </xsd:element>
      <xsd:element maxOccurs="10" minOccurs="1" name="c" type="A_c">
        <xsd:annotation>
          <xsd:appinfo source="http://www.wsadie.com/appinfo">

<dependingOn>/0/A;XSDComplexTypeDefinition/XSDParticle/XSDModelGroup/XSDParticle/b;XSDElementDeclaration</dependingOn>
          </xsd:appinfo>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1" 
repeatRef="/0/A;XSDComplexTypeDefinition/XSDParticle/XSDModelGroup/XSDParticle/b;XSDElementDeclaration">
              <cwfAggrRep accessor="readWrite" attributeInBit="false" 
contentSize="mpy(20,val(1,1))" offset="3" size="200">
                <ArrayTD_arrayDescr alignmentKind="byte" lowerBound="1" 
stride="20" strideInBit="false" upperBound="val(1,1)"/>
              </cwfAggrRep>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
      </xsd:element>
    </xsd:sequence>
  </xsd:complexType>
  <xsd:complexType name="A_c">
        <xsd:group ref="a_c"/>
    </xsd:complexType>
  <xsd:group name="a_c">
    <xsd:sequence>
      <xsd:group ref="RedefinedElement_a_c_x_y">
        <xsd:annotation>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfAggrRep accessor="readWrite" attributeInBit="false" 
contentSize="20" offset="3" size="20"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
      </xsd:group>
    </xsd:sequence>
  </xsd:group>
  <xsd:group name="RedefinedElement_a_c_x_y">
    <xsd:choice>
      <xsd:element name="x">
        <xsd:annotation>
          <xsd:appinfo source="http://www.wsadie.com/appinfo">
            <initialValue kind="SPACE"/>
          </xsd:appinfo>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfSimpleRep accessor="readWrite" addrUnit="byte" 
alignment="byte" attributeInBit="false" characterSize="1" contentSize="20" 
lengthEncoding="fixedLength" offset="3" paddingCharacter="SPACE" 
prefixLength="0" size="20" typeName="MRCWFStringRep" width="20"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
        <xsd:simpleType>
          <xsd:restriction base="xsd:string">
            <xsd:maxLength value="20"/>
          </xsd:restriction>
        </xsd:simpleType>
      </xsd:element>
      <xsd:element name="y" type="a_c_y">
        <xsd:annotation>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfAggrRep accessor="readWrite" attributeInBit="false" 
contentSize="8" offset="3" size="8"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
      </xsd:element>
    </xsd:choice>
  </xsd:group>
  <xsd:group name="a_c_y">
    <xsd:sequence>
      <xsd:element name="d">
        <xsd:annotation>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfSimpleRep accessor="readWrite" addrUnit="byte" 
alignment="byte" attributeInBit="false" characterSize="1" contentSize="5" 
lengthEncoding="fixedLength" offset="3" paddingCharacter="SPACE" 
prefixLength="0" size="5" typeName="MRCWFStringRep" width="5"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
        <xsd:simpleType>
          <xsd:restriction base="xsd:string">
            <xsd:maxLength value="5"/>
          </xsd:restriction>
        </xsd:simpleType>
      </xsd:element>
      <xsd:element name="e">
        <xsd:annotation>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfSimpleRep accessor="readWrite" addrUnit="byte" 
alignment="byte" attributeInBit="false" contentSize="3" offset="8" 
signFormat="trailing" signed="false" size="3" 
typeName="MRCWFExternalDecimalRep" width="3"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
        <xsd:simpleType>
          <xsd:restriction base="xsd:short">
            <xsd:minInclusive value="0"/>
            <xsd:maxInclusive value="999"/>
          </xsd:restriction>
        </xsd:simpleType>
      </xsd:element>
    </xsd:sequence>
  </xsd:group>
  <xsd:complexType name="a_c_y">
    <xsd:group ref="a_c_y"/>
  </xsd:complexType>
    <xsd:element name="msg_A" type="A">
    <xsd:annotation>
      <xsd:appinfo source="WMQI_APPINFO">
        <MRMessage 
messageDefinition="/0/msg_A;XSDElementDeclaration$MRObject"/>
      </xsd:appinfo>
    </xsd:annotation>
  </xsd:element>
</xsd:schema>

Suman Kalia
IBM Toronto Lab
WebSphere Message Broker Toolkit Architect and Development Lead
WebSphere Business Integration Application Connectivity Tools 

http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.ht...

Tel : 905-413-3923  T/L  969-3923
Fax : 905-413-4850 T/L  969-4850
Internet ID : kalia@ca.ibm.com

From:   Suman Kalia/Toronto/IBM
To:     Alan Powell/UK/IBM@IBMGB, Steve Hanson/UK/IBM@IBMGB, Tim 
Kimber/UK/IBM@IBMGB
Date:   05/12/2010 10:22 AM
Subject:        Action : 088 define semantics of choiceKind 'fixedLength' 

Attached is the sample cobol copy book and the mxsd generated from it.. It 
contains the TD model annotations that we don't use in WMB. The TD model 
annotations that are of particular interest for this action is the 
contentSize which is an expression that tells the compiler runtime how 
much memory is actually allocated for the repeating element. The size 
attribute tells what is the maximum it can have based on the definition in 
the copy book.
...
From DFDL perspective, the calculated length (reference Tim's note) would 
be similar to the contentSize() attribute..
01   A.
           05  b  pic 9(3).
           05  c  occurs 1 to 10 times depending on b.
               10  d    pic x(5).
               10  e    pic 9(3). 

<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:xsd="
http://www.w3.org/2001/XMLSchema">
  <xsd:complexType name="A">
    <xsd:sequence>
      <xsd:element name="b">
        <xsd:annotation>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfSimpleRep accessor="readWrite" addrUnit="byte" 
alignment="byte" attributeInBit="false" contentSize="3" offset="0" 
signFormat="trailing" signed="false" size="3" 
typeName="MRCWFExternalDecimalRep" width="3"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
        <xsd:simpleType>
          <xsd:restriction base="xsd:short">
            <xsd:minInclusive value="0"/>
            <xsd:maxInclusive value="999"/>
          </xsd:restriction>
        </xsd:simpleType>
      </xsd:element>
      <xsd:element maxOccurs="10" minOccurs="1" name="c" type="A_c">
        <xsd:annotation>
          <xsd:appinfo source="http://www.wsadie.com/appinfo">

<dependingOn>/0/A;XSDComplexTypeDefinition/XSDParticle/XSDModelGroup/XSDParticle/b;XSDElementDeclaration</dependingOn>
          </xsd:appinfo>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1" 
repeatRef="/0/A;XSDComplexTypeDefinition/XSDParticle/XSDModelGroup/XSDParticle/b;XSDElementDeclaration">
              <cwfAggrRep accessor="readWrite" attributeInBit="false" 
contentSize="mpy(8,val(1,1))" offset="3" size="80">
                <ArrayTD_arrayDescr alignmentKind="byte" lowerBound="1" 
stride="8" strideInBit="false" upperBound="val(1,1)"/>
              </cwfAggrRep>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
      </xsd:element>
    </xsd:sequence>
  </xsd:complexType>
  <xsd:complexType name="A_c">
        <xsd:group ref="a_c"/>
    </xsd:complexType>
  <xsd:group name="a_c">
    <xsd:sequence>
      <xsd:element name="d">
        <xsd:annotation>
          <xsd:appinfo source="http://www.wsadie.com/appinfo">
            <initialValue kind="SPACE"/>
          </xsd:appinfo>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfSimpleRep accessor="readWrite" addrUnit="byte" 
alignment="byte" attributeInBit="false" characterSize="1" contentSize="5" 
lengthEncoding="fixedLength" offset="3" paddingCharacter="SPACE" 
prefixLength="0" size="5" typeName="MRCWFStringRep" width="5"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
        <xsd:simpleType>
          <xsd:restriction base="xsd:string">
            <xsd:maxLength value="5"/>
          </xsd:restriction>
        </xsd:simpleType>
      </xsd:element>
      <xsd:element name="e">
        <xsd:annotation>
          <xsd:appinfo source="WMQI_APPINFO">
            <cwfInclRep messageSetDefaultRep="Binary1">
              <cwfSimpleRep accessor="readWrite" addrUnit="byte" 
alignment="byte" attributeInBit="false" contentSize="3" offset="8" 
signFormat="trailing" signed="false" size="3" 
typeName="MRCWFExternalDecimalRep" width="3"/>
            </cwfInclRep>
          </xsd:appinfo>
        </xsd:annotation>
        <xsd:simpleType>
          <xsd:restriction base="xsd:short">
            <xsd:minInclusive value="0"/>
            <xsd:maxInclusive value="999"/>
          </xsd:restriction>
        </xsd:simpleType>
      </xsd:element>
    </xsd:sequence>
  </xsd:group>
    <xsd:element name="msg_A" type="A">
    <xsd:annotation>
      <xsd:appinfo source="WMQI_APPINFO">
        <MRMessage 
messageDefinition="/0/msg_A;XSDElementDeclaration$MRObject"/>
      </xsd:appinfo>
    </xsd:annotation>
  </xsd:element>
</xsd:schema>

Suman Kalia
IBM Toronto Lab
WebSphere Message Broker Toolkit Architect and Development Lead
WebSphere Business Integration Application Connectivity Tools 

http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.ht...

Tel : 905-413-3923  T/L  969-3923
Fax : 905-413-4850 T/L  969-4850
Internet ID : kalia@ca.ibm.com

More work to do on this, but no time before the meeting. Let's discuss the 
rules below, and refine afterwards: 

If dfdl:choiceKind='fixedLength' then there must be at least one branch of 
the choice for which all of the content has a calculable length. 

Definition : A DFDL region is a region of the data stream described by the 
DFDL grammar. 
Note : we need to define this term - we use the term 'region' without 
definition throughout the specification. Sometimes we call it a 'grammar 
region'. 

Definition: A DFDL region has a calculable length if the length of its 
representation in units of bytes can be calculated from the DFDL 
description alone. 

The leading alignment region has a calculable length if the alignment of 
the component is the same as the alignment of the parent. 
The trailing alignment region has a calculable length if the alignment of 
the component is the same as the alignment of the parent. 
Q: Is this rule necessary? 

The initiator region has a calculable length if all of the possible 
initiator values have a calculable length. 
An initiator value is a DFDL string literal. It has a calculable length if 
the DFDL string literal does not contain any variable-length parts. 

The terminator region has a calculable length if all of the possible 
terminator values have a calculable length. 
A terminator value is a DFDL string literal. It has a calculable length if 
the DFDL string literal does not contain any variable-length parts. 

A variable-length part  is any of the following 
- a character string in a variable-width encoding 
- a DFLD entity in a variable-width encoding 
- a DFDL generic entity which can match a variable number of characters ( 
NL, WSP+, WSP* ) 

The simple content region has a calculable length if both of the following 
are true: 
- either the encoding is a fixed-width encoding or the representation does 
not contain characters. 
- either lengthKind is 'explicit' and the length is not a DFDL expression 
or lengthKind is 'implicit' and there is an implicit length for the simple 
type and its dfdl:representation 

The complex content region has a calculable length if all of the following 
are true: 
- either the encoding is a fixed-width encoding or the representation ( 
recursively ) does not contain characters. 
- the element has lengthKind='explicit' and dfdl:length is not a DFDL 
expression 
- all group members, recursively, are required ( minOccurs=maxOccurs ) 

regards,

Tim Kimber, Common Transformation Team,
Hursley, UK
Internet:  kimbert@uk.ibm.com
Tel. 01962-816742 
Internal tel. 246742

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Steve Hanson

tags

participants (1)