Two corrections: a) Use correct prefix for XPath integer() function b) Can't use dfdl:outputValueCalc (or any other property) with dfdl:inputValueCalc Regards Steve Hanson Strategy, Common Transformation & DFDL Co-Chair, OGF DFDL WG IBM SWG, Hursley, UK, smh@uk.ibm.com, tel +44-(0)1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 20/09/2010 17:29 ----- From: Alan Powell/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Date: 20/09/2010 15:57 Subject: Re: Fw: DFDL functions testOn/Off and setOn/Off - action 107 Steve The spec doesn't allow inputValueCalc and outputValueCalc on the same element. Regards Alan Powell Development - MQSeries, Message Broker, ESB IBM Software Group, Application and Integration Middleware Software ------------------------------------------------------------------------------------------------------------------------------------------- IBM MP211, Hursley Park Hursley, SO21 2JN United Kingdom Phone: +44-1962-815073 e-mail: alan_powell@uk.ibm.com ----- Forwarded by Steve Hanson/UK/IBM on 20/09/2010 11:19 ----- From: Steve Hanson/UK/IBM To: dfdl-wg@ogf.org Date: 20/09/2010 11:18 Subject: DFDL functions testOn/Off and setOn/Off - action 107 The expression language section of the spec (21.5.3). defines some functions that make it easier to test/set individual bits in a byte. Action 107 was raised to question whether they were needed and if so to complete the definitions. In particular the spec for setOn/setOff seems odd. Here's an example of where they are useful, although upon investigation the functions need changing a bit. A financial standard exists called ISO8583. It is widely used with debit/credit card transactions. An ISO8583 data stream consists of 128 optional fixed length fields with no tags or delimiters, preceded by a mandatory bitmap that indicates the presence of all the other fields. In DFDL a nice approach is to model each optional field as having dfdl:occursCountKind="expression" with dfdl:occursCount referencing the corresponding bit in the bitmap and casting its value to an integer to give 0 or 1. Each bit in the bitmap would have its output value set using dfdl:outputValueCalc, true if the corresponding field existed in the infoset. The representation of the bitmap is the interesting part. It can either be packed or unpacked. Packed In packed format the bitmap is a real array of bits (8 bytes worth) which must be modelled as individual booleans (ie, not an array) because each needs to carry a distinct dfdl:outputValueCalc expression, as shown below. For example: <complexType name="ISO8583_Packed"> <sequence> <element name="BitMap"> <complexType> <sequence> <!-- each bit in the bitmap modelled individually --> <element name="Bit001" type="boolean" dfdl:representation="binary" dfdl:binaryBooleanTrueRep="1" dfdl:binaryBooleanFalseRep="0" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:lengthUnits="bits" dfdl:outputValueCalc="{if fn:exists(../Field001) then true else false}" /> ... <element name="Bit064" type="boolean" dfdl:representation="binary" dfdl:binaryBooleanTrueRep="1" dfdl:binaryBooleanFalseRep="0" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:lengthUnits="bits" dfdl:outputValueCalc="{if fn:exists(../Field064) then true else false}" /> </sequence> </complexType> </element> <!-- each field below determines its optional occurrence based on a bit in the bitmap field --> <element name="Field001" type="string" minOccurs="0" dfdl:occursKind="expression" dfdl:occurs="{xs:integer(../BitMap/Bit001)}" /> ... <element name="Field064" type="string" minOccurs="0" dfdl:occursKind="expression" dfdl:occurs="{xs:integer(../BitMap/Bit064)}" /> </sequence> </complexType> Unpacked However, in the unpacked format the value of each nibble of the bitmap is taken and unpacked as text (ASCII or EBCDIC). Eg, bitmap '0000000100100011010001010110011110001001101010111100110111101111' (64 bits) = x01x23x45x67x89xABxCDxEF (8 bytes) = x30x31x32x33x34x35x36x37x38x39x41x42x43x44x45x46 (16 bytes) in ASCII. This can be modelled as an unsigned long with representation as a base 16 standard text number. The problem is then how to test and set the individual bits in the unsigned long. That's where the DFDL functions come in useful, because there are no XPath functions for bitwise operations. If the functions are defined as follows: xs:boolean dfdl:testBit(xs:unsignedLong, xs:unsignedShort) Returns Boolean true if the bit number given by arg #2 is set on in the integer given by arg #1, otherwise returns Boolean false. xs:unsignedLong dfdl:setBits(xs:boolean+) Returns an integer being the value of the bit positions provided by the Boolean arguments, where true=1, false=0. The # of args must be a multiple of 8. Note that the bit numbering goes from left to right, in accordance with section 12.3.7.2 of the spec. Applying this to the example above gives: <complexType name="ISO8583_Unpacked"> <sequence> <element name="BitMapInt" type="unsignedLong" dfdl:representation="text" dfdl:textNumberRep="standard" dfdl:textNumberBase="16" dfdl:encoding="ascii" dfdl:lengthKind="explicit" dfdl:length="16" dfdl:lengthUnits="characters" dfdl:outputValueCalc="{dfdl:setBits(fn:exists(../Field001), ... fn:exists(../Field064)}" /> <!-- each field below determines its optional occurrence based on a bit in the bitmap field --> <element name="Field001" type="string" minOccurs="0" dfdl:occursKind="expression" dfdl:occurs="{xs:integer(dfdl:testBit(../BitMapInt, 1)}" /> ... <element name="Field064" type="string" minOccurs="0" dfdl:occursKind="expression" dfdl:occurs="{xs:integer(dfdl:testBit(../BitMapInt, 64)}" /> </sequence> </complexType> To keep the unpacked model as close to the packed model as possible, we can incorporate the logical BitMap from the packed model and hide BitMapInt. This means the Fieldxxx elements are the same for each. <complexType name="ISO8583_Unpacked"> <sequence> <sequence dfdl:hiddenGroupRef="HiddenBitMap" /> <element name="BitMap"> <complexType> <sequence> <!-- each bit in the bitmap modelled individually but derived --> <element name="Bit001" type="boolean" dfdl:inputValueCalc="{dfdl:testBit(../BitMapInt, 1)}" /> ... <element name="Bit064" type="boolean" dfdl:inputValueCalc="{dfdl:testBit(../BitMapInt, 64)}" /> </sequence> </complexType> </element> <!-- each field below determines its optional occurrence based on a bit in the bitmap field --> <element name="Field001" type="string" minOccurs="0" dfdl:occursKind="expression" dfdl:occurs="{xs:integer(../BitMap/Bit001)}" /> ... <element name="Field064" type="string" minOccurs="0" dfdl:occursKind="expression" dfdl:occurs="{xs:integer(../BitMap/Bit064)}" /> </sequence> </complexType> <group name="HiddenBitMap"> <sequence> <element name="BitMapInt" type="unsignedLong" dfdl:representation="text" dfdl:textNumberRep="standard" dfdl:textNumberBase="16" dfdl:encoding="ascii" dfdl:lengthKind="explicit" dfdl:length="16" dfdl:lengthUnits="characters" dfdl:outputValueCalc="{dfdl:setBits(fn:exists(../Field001), ... fn:exists(../Field064)}" /> </sequence> </group> For both packed and unpacked, the BitMap element could be hidden as well, as there is no need for it in the infoset. Regards Steve Hanson Strategy, Common Transformation & DFDL Co-Chair, OGF DFDL WG IBM SWG, Hursley, UK, smh@uk.ibm.com, tel +44-(0)1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU