Forwarding to DFDL WG mailing list in anticipation of discussion on this week's call. Regards Steve Hanson Strategy, Common Transformation & DFDL Co-Chair, OGF DFDL WG IBM SWG, Hursley, UK, smh@uk.ibm.com, tel +44-(0)1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/07/2010 10:46 ----- From: Steve Hanson/UK/IBM To: Suman Kalia/Toronto/IBM@IBMCA Cc: Alan Powell/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 05/07/2010 16:06 Subject: Re: Action : 088 define semantics of choiceKind 'fixedLength' We discussed this briefly on the last DFDL OGF WG call. The use case we are trying to support is where a programming language has reserved memory for the longest branch of the choice. I would love to be able to drop the dfdl:choiceKind property and instead use the dfdl:length property of the parent element of the choice, set from the TD size attribute. In other words, we do nothing special for choices; - on input we expect dfdl:length bytes or characters, we parse the chosen branch, if it is short we then skip to the end - on output we serialise the chosen branch, if it is short we then pad out to dfdl:length bytes or characters using dfdl:fillByte. But this has some limitations... a) It only works with choices that are the content model of a complex type b) It forces the DFDL modeller to calculate the length of the choice While b) is not a problem in practice because the importer sets it and not the user, unfortunately a) is, because the importers today generate a xs:group ref to a global choice group, so there is nowhere to carry the dfdl:length property. I think the alternatives are: i) Change the importer to add a parent element for each group. Con: This changes the logical model by inserting an extra level. ii) Allow dfdl:length to be carried on embedded xs:choice (along with dfdl:lengthUnits & dfdl:lengthKind). iii) New dfdl:choiceLength property to be carried on embedded xs:choice (always bytes). iv) dfdl:choiceKind as today. Con: length from TD model is lost and must be recalculated. v) dfdl:choiceKind but with limitations to make the calculation easier. Con: length from TD model is lost and must be recalculated. The DFDL bottom-up 'model the data' principle would point at i) but there are compatibility issues with existing uses of the importer. That leaves iii) and v) as main contenders. It seems a real shame to throw away the length that the TD model has calculated, so I think we should discuss iii). Regards Steve Hanson Strategy, Common Transformation & DFDL Co-Chair, OGF DFDL WG IBM SWG, Hursley, UK, smh@uk.ibm.com, tel +44-(0)1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Alan Powell/UK/IBM@IBMGB, Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 12/05/2010 16:25 Subject: Re: Action : 088 define semantics of choiceKind 'fixedLength' See the updated contentSize() function in the attached mxsd file. It takes the largest size value for the choice branch; in this case element x of size 20. 01 A. 05 b pic 9(3). 05 c occurs 1 to 10 times depending on b. 10 x pic x(20). 10 y redefines x. 20 d pic x(5). 20 e pic 9(3). <?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:xsd=" http://www.w3.org/2001/XMLSchema"> <xsd:complexType name="A"> <xsd:sequence> <xsd:element name="b"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfSimpleRep accessor="readWrite" addrUnit="byte" alignment="byte" attributeInBit="false" contentSize="3" offset="0" signFormat="trailing" signed="false" size="3" typeName="MRCWFExternalDecimalRep" width="3"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:short"> <xsd:minInclusive value="0"/> <xsd:maxInclusive value="999"/> </xsd:restriction> </xsd:simpleType> </xsd:element> <xsd:element maxOccurs="10" minOccurs="1" name="c" type="A_c"> <xsd:annotation> <xsd:appinfo source="http://www.wsadie.com/appinfo"> <dependingOn>/0/A;XSDComplexTypeDefinition/XSDParticle/XSDModelGroup/XSDParticle/b;XSDElementDeclaration</dependingOn> </xsd:appinfo> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1" repeatRef="/0/A;XSDComplexTypeDefinition/XSDParticle/XSDModelGroup/XSDParticle/b;XSDElementDeclaration"> <cwfAggrRep accessor="readWrite" attributeInBit="false" contentSize="mpy(20,val(1,1))" offset="3" size="200"> <ArrayTD_arrayDescr alignmentKind="byte" lowerBound="1" stride="20" strideInBit="false" upperBound="val(1,1)"/> </cwfAggrRep> </cwfInclRep> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> </xsd:complexType> <xsd:complexType name="A_c"> <xsd:group ref="a_c"/> </xsd:complexType> <xsd:group name="a_c"> <xsd:sequence> <xsd:group ref="RedefinedElement_a_c_x_y"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfAggrRep accessor="readWrite" attributeInBit="false" contentSize="20" offset="3" size="20"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> </xsd:group> </xsd:sequence> </xsd:group> <xsd:group name="RedefinedElement_a_c_x_y"> <xsd:choice> <xsd:element name="x"> <xsd:annotation> <xsd:appinfo source="http://www.wsadie.com/appinfo"> <initialValue kind="SPACE"/> </xsd:appinfo> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfSimpleRep accessor="readWrite" addrUnit="byte" alignment="byte" attributeInBit="false" characterSize="1" contentSize="20" lengthEncoding="fixedLength" offset="3" paddingCharacter="SPACE" prefixLength="0" size="20" typeName="MRCWFStringRep" width="20"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:maxLength value="20"/> </xsd:restriction> </xsd:simpleType> </xsd:element> <xsd:element name="y" type="a_c_y"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfAggrRep accessor="readWrite" attributeInBit="false" contentSize="8" offset="3" size="8"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:choice> </xsd:group> <xsd:group name="a_c_y"> <xsd:sequence> <xsd:element name="d"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfSimpleRep accessor="readWrite" addrUnit="byte" alignment="byte" attributeInBit="false" characterSize="1" contentSize="5" lengthEncoding="fixedLength" offset="3" paddingCharacter="SPACE" prefixLength="0" size="5" typeName="MRCWFStringRep" width="5"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:maxLength value="5"/> </xsd:restriction> </xsd:simpleType> </xsd:element> <xsd:element name="e"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfSimpleRep accessor="readWrite" addrUnit="byte" alignment="byte" attributeInBit="false" contentSize="3" offset="8" signFormat="trailing" signed="false" size="3" typeName="MRCWFExternalDecimalRep" width="3"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:short"> <xsd:minInclusive value="0"/> <xsd:maxInclusive value="999"/> </xsd:restriction> </xsd:simpleType> </xsd:element> </xsd:sequence> </xsd:group> <xsd:complexType name="a_c_y"> <xsd:group ref="a_c_y"/> </xsd:complexType> <xsd:element name="msg_A" type="A"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <MRMessage messageDefinition="/0/msg_A;XSDElementDeclaration$MRObject"/> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:schema> Suman Kalia IBM Toronto Lab WebSphere Message Broker Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.ht... Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia@ca.ibm.com From: Suman Kalia/Toronto/IBM To: Alan Powell/UK/IBM@IBMGB, Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 05/12/2010 10:22 AM Subject: Action : 088 define semantics of choiceKind 'fixedLength' Attached is the sample cobol copy book and the mxsd generated from it.. It contains the TD model annotations that we don't use in WMB. The TD model annotations that are of particular interest for this action is the contentSize which is an expression that tells the compiler runtime how much memory is actually allocated for the repeating element. The size attribute tells what is the maximum it can have based on the definition in the copy book.
From DFDL perspective, the calculated length (reference Tim's note) would be similar to the contentSize() attribute..
01 A. 05 b pic 9(3). 05 c occurs 1 to 10 times depending on b. 10 d pic x(5). 10 e pic 9(3). <?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:xsd=" http://www.w3.org/2001/XMLSchema"> <xsd:complexType name="A"> <xsd:sequence> <xsd:element name="b"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfSimpleRep accessor="readWrite" addrUnit="byte" alignment="byte" attributeInBit="false" contentSize="3" offset="0" signFormat="trailing" signed="false" size="3" typeName="MRCWFExternalDecimalRep" width="3"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:short"> <xsd:minInclusive value="0"/> <xsd:maxInclusive value="999"/> </xsd:restriction> </xsd:simpleType> </xsd:element> <xsd:element maxOccurs="10" minOccurs="1" name="c" type="A_c"> <xsd:annotation> <xsd:appinfo source="http://www.wsadie.com/appinfo"> <dependingOn>/0/A;XSDComplexTypeDefinition/XSDParticle/XSDModelGroup/XSDParticle/b;XSDElementDeclaration</dependingOn> </xsd:appinfo> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1" repeatRef="/0/A;XSDComplexTypeDefinition/XSDParticle/XSDModelGroup/XSDParticle/b;XSDElementDeclaration"> <cwfAggrRep accessor="readWrite" attributeInBit="false" contentSize="mpy(8,val(1,1))" offset="3" size="80"> <ArrayTD_arrayDescr alignmentKind="byte" lowerBound="1" stride="8" strideInBit="false" upperBound="val(1,1)"/> </cwfAggrRep> </cwfInclRep> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> </xsd:complexType> <xsd:complexType name="A_c"> <xsd:group ref="a_c"/> </xsd:complexType> <xsd:group name="a_c"> <xsd:sequence> <xsd:element name="d"> <xsd:annotation> <xsd:appinfo source="http://www.wsadie.com/appinfo"> <initialValue kind="SPACE"/> </xsd:appinfo> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfSimpleRep accessor="readWrite" addrUnit="byte" alignment="byte" attributeInBit="false" characterSize="1" contentSize="5" lengthEncoding="fixedLength" offset="3" paddingCharacter="SPACE" prefixLength="0" size="5" typeName="MRCWFStringRep" width="5"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:maxLength value="5"/> </xsd:restriction> </xsd:simpleType> </xsd:element> <xsd:element name="e"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <cwfInclRep messageSetDefaultRep="Binary1"> <cwfSimpleRep accessor="readWrite" addrUnit="byte" alignment="byte" attributeInBit="false" contentSize="3" offset="8" signFormat="trailing" signed="false" size="3" typeName="MRCWFExternalDecimalRep" width="3"/> </cwfInclRep> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base="xsd:short"> <xsd:minInclusive value="0"/> <xsd:maxInclusive value="999"/> </xsd:restriction> </xsd:simpleType> </xsd:element> </xsd:sequence> </xsd:group> <xsd:element name="msg_A" type="A"> <xsd:annotation> <xsd:appinfo source="WMQI_APPINFO"> <MRMessage messageDefinition="/0/msg_A;XSDElementDeclaration$MRObject"/> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:schema> Suman Kalia IBM Toronto Lab WebSphere Message Broker Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.ht... Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia@ca.ibm.com More work to do on this, but no time before the meeting. Let's discuss the rules below, and refine afterwards: If dfdl:choiceKind='fixedLength' then there must be at least one branch of the choice for which all of the content has a calculable length. Definition : A DFDL region is a region of the data stream described by the DFDL grammar. Note : we need to define this term - we use the term 'region' without definition throughout the specification. Sometimes we call it a 'grammar region'. Definition: A DFDL region has a calculable length if the length of its representation in units of bytes can be calculated from the DFDL description alone. The leading alignment region has a calculable length if the alignment of the component is the same as the alignment of the parent. The trailing alignment region has a calculable length if the alignment of the component is the same as the alignment of the parent. Q: Is this rule necessary? The initiator region has a calculable length if all of the possible initiator values have a calculable length. An initiator value is a DFDL string literal. It has a calculable length if the DFDL string literal does not contain any variable-length parts. The terminator region has a calculable length if all of the possible terminator values have a calculable length. A terminator value is a DFDL string literal. It has a calculable length if the DFDL string literal does not contain any variable-length parts. A variable-length part is any of the following - a character string in a variable-width encoding - a DFLD entity in a variable-width encoding - a DFDL generic entity which can match a variable number of characters ( NL, WSP+, WSP* ) The simple content region has a calculable length if both of the following are true: - either the encoding is a fixed-width encoding or the representation does not contain characters. - either lengthKind is 'explicit' and the length is not a DFDL expression or lengthKind is 'implicit' and there is an implicit length for the simple type and its dfdl:representation The complex content region has a calculable length if all of the following are true: - either the encoding is a fixed-width encoding or the representation ( recursively ) does not contain characters. - the element has lengthKind='explicit' and dfdl:length is not a DFDL expression - all group members, recursively, are required ( minOccurs=maxOccurs ) regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU