September 2012 - dfdl-wg

OGF DFDL WG Call Agenda 2012-09-18 - follow on call to be held
by Steve Hanson 18 Sep '12

18 Sep '12

We didn't get through all the agenda items so another call will take place on Thursday at 13:30 UK. Same agenda on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (continued) (20 Sep 13:30 GDT in IBM Hursley DE2K09)
by Steve Hanson 18 Sep '12

18 Sep '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Had to reschedule

1 0

Fw: Transaction log data types from IBM 4690 point-of-sale terminals - add to DFDL spec?
by Steve Hanson 18 Sep '12

18 Sep '12

DFDL WG agreed to add the new enum to binaryNumberRep and binaryCalendarRep. Errata will be taken. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/09/2012 16:43 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 17/09/2012 15:23 Subject: Transaction log data types from IBM 4690 point-of-sale terminals - add to DFDL spec? IBM 4690 data format IBM 4690 is a widely used retail Point-of-Sale system. The transactions logs emitted by 4690 are in a compact, ASCII delimited binary format, known as TLOG. Each TLOG consists of a number of records, called 'strings'. Each 'string' consists of fields delimited by a colon (x2C), although some strings use double quote/colon/double quote instead (x22x2Cx22). The fields themselves can be fixed length or variable length but the delimiter is always present. The fields can have a number of physical reps but the most frequent rep is a packed number rep that is not BCD or S/390 packed decimal, and so DFDL can't model it. It is used to represent the vast majority of decimal and integer numbers in TLOG data. This packed decimal format is best described by the operations that act on the data: Pack: Take an ASCII string of characters, drop the first nibble of each byte, shuffle the remaining nibbles to the right, and if there is an odd number of nibbles pad with an xF nibble on the left. Unpack: Discard any leading xF nibble, and expand each nibble into a byte by adding a x3 nibble in front, giving an ASCII string. It can be seen that while the pack operation may be applied to any ASCII string, the unpack operation will only round trip if the original characters are in the range x30 to x3E, hence the operations were only intended to be used for packing and unpacking unsigned numbers. A convention has arisen however whereby the minus sign x2D may be used to indicate negative values, even though xD unpacks as x3D which is the equals sign. In summary: Nibbles represent digits 0 - 9 in the usual BCD manner A positive value is simply indicated by digits A negative number is indicated by digits with the leading nibble being xD If a positive or negative value packs to an odd number of nibbles, pad on the left with xF Some examples help: 12 x12 123 xF1x23 -12 xFDx12 -123 xD1x23 Note that x00xD1x23 is not valid as this would unpack to 0=123 (ie, 0-123). The only nibble that can appear to the left of an xD nibble is an xF pad nibble. Or put another way, you can't blindly byte pad with x00 to make up a fixed length. If such byte padding is required, replace any xF and xD nibbles with x0 nibble, then byte pad with x00, then reinstate any xD nibble in the new leftmost nibble. Proposal to add support to DFDL: Existing dfdl:binaryNumberRep property gets a new enum 'ibm4690Packed'. It applies to all signed and unsigned simple types derived from xsd:decimal (both built-in and user-defined). The following DFDL properties are used in conjunction: dfdl:byteOrder (although in practice IBM 4690 is always big-endian) dfdl:binaryDecimalVirtualPoint Note that dfdl:binaryPackedSignCodes does not apply - the only negative nibble is xD - and dfdl:binaryNumberCheckPolicy has no effect. Where the DFDL specification provides for general behaviours for 'packed' and 'bcd', those behaviours apply also to 'ibm4690Packed'. Specifically: The same dfdl:lengthKind enums and rules apply. No rounding when serializing, a value that can't be accommodated is a processing error. If the logical type is unsigned and a negative value is received, it is a processing error. The same should be done for dfdl:binaryCalendarRep to allow TLOG packed decimal representation for calendars. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Broadcast: OGF DFDL Working Group weekly call (continued) (20 Sep 16:00 GDT in IBM Hursley DE2K09)
by Steve Hanson 18 Sep '12

18 Sep '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0

OGF DFDL WG Call Agenda 2012-09-18
by Steve Hanson 18 Sep '12

18 Sep '12

Please find agenda for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Specify field based on the beggining of the binary file
by Miguel Vieira 18 Sep '12

18 Sep '12

Hi! I have a binary file specification that specifies some fields referenced by the beginning of the binary info. For instance imagine that there is a field with 1024Bytes and the next one starts at byte 512 counting from the beginning of the info. Is it possible to do that using dfdl? I've seen dfdl:floating, but I can't find much info on that. Thanks. ________________________________ [Description: cid:image001.jpg@01CB4ADC.488557F0] Miguel Vieira Software Engineer carlos.vieira(a)gmv.com<mailto:carlos.vieira@gmv.com> GMV-SKYSOFT Torre Fernão de Magalhães Av. D. João II Lote 1.17.02, 7º Andar 1998 - 025 Lisboa Portugal Tlm. +351 962 060 326 Tel. +351 21 382 93 66 Fax +351 21 386 64 93 www.gmv.com.pt<http://www.gmv.com.pt> P Please consider the environment before printing this e-mail. ______________________ This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. ______________________ Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacion clasificada por su emisor como confidencial en el marco de su Sistema de Gestion de Seguridad de la Informacion siendo para uso exclusivo del destinatario, quedando prohibida su divulgacion copia o distribucion a terceros sin la autorizacion expresa del remitente. Si Vd. ha recibido este mensaje erroneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracion. ______________________

3 6

Transaction log data types from IBM 4690 point-of-sale terminals - add to DFDL spec?
by Steve Hanson 17 Sep '12

17 Sep '12

IBM 4690 data format IBM 4690 is a widely used retail Point-of-Sale system. The transactions logs emitted by 4690 are in a compact, ASCII delimited binary format, known as TLOG. Each TLOG consists of a number of records, called 'strings'. Each 'string' consists of fields delimited by a colon (x2C), although some strings use double quote/colon/double quote instead (x22x2Cx22). The fields themselves can be fixed length or variable length but the delimiter is always present. The fields can have a number of physical reps but the most frequent rep is a packed number rep that is not BCD or S/390 packed decimal, and so DFDL can't model it. It is used to represent the vast majority of decimal and integer numbers in TLOG data. This packed decimal format is best described by the operations that act on the data: Pack: Take an ASCII string of characters, drop the first nibble of each byte, shuffle the remaining nibbles to the right, and if there is an odd number of nibbles pad with an xF nibble on the left. Unpack: Discard any leading xF nibble, and expand each nibble into a byte by adding a x3 nibble in front, giving an ASCII string. It can be seen that while the pack operation may be applied to any ASCII string, the unpack operation will only round trip if the original characters are in the range x30 to x3E, hence the operations were only intended to be used for packing and unpacking unsigned numbers. A convention has arisen however whereby the minus sign x2D may be used to indicate negative values, even though xD unpacks as x3D which is the equals sign. In summary: Nibbles represent digits 0 - 9 in the usual BCD manner A positive value is simply indicated by digits A negative number is indicated by digits with the leading nibble being xD If a positive or negative value packs to an odd number of nibbles, pad on the left with xF Some examples help: 12 x12 123 xF1x23 -12 xFDx12 -123 xD1x23 Note that x00xD1x23 is not valid as this would unpack to 0=123 (ie, 0-123). The only nibble that can appear to the left of an xD nibble is an xF pad nibble. Or put another way, you can't blindly byte pad with x00 to make up a fixed length. If such byte padding is required, replace any xF and xD nibbles with x0 nibble, then byte pad with x00, then reinstate any xD nibble in the new leftmost nibble. Proposal to add support to DFDL: Existing dfdl:binaryNumberRep property gets a new enum 'ibm4690Packed'. It applies to all signed and unsigned simple types derived from xsd:decimal (both built-in and user-defined). The following DFDL properties are used in conjunction: dfdl:byteOrder (although in practice IBM 4690 is always big-endian) dfdl:binaryDecimalVirtualPoint Note that dfdl:binaryPackedSignCodes does not apply - the only negative nibble is xD - and dfdl:binaryNumberCheckPolicy has no effect. Where the DFDL specification provides for general behaviours for 'packed' and 'bcd', those behaviours apply also to 'ibm4690Packed'. Specifically: The same dfdl:lengthKind enums and rules apply. No rounding when serializing, a value that can't be accommodated is a processing error. If the logical type is unsigned and a negative value is received, it is a processing error. The same should be done for dfdl:binaryCalendarRep to allow TLOG packed decimal representation for calendars. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

assert and discriminator - no more before/after
by Mike Beckerle 17 Sep '12

17 Sep '12

I am looking in the spec for guidance about the evaluation order of assert statements. We used to have before/after control properties, but eliminated them. If I annotate a simpleType'd element with an assert that says { . eq 'x' }, that of necessity references the current value, so must execute after the value has been computed. If on the other hand I annotate a complexType element with a discriminator that says { ../flag eq 'C1' } then this of necessity must execute before I go after the contents because the whole point is to evalutate the discriminator first. Did we ever articulate exactly what the rules are here about order of evaluation? Thanks for reminders ..mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 4

spec bug
by Mike Beckerle 13 Sep '12

13 Sep '12

section 7.4.1 in the example, the element named 'branchSimple' is not well formed (lacks type, for example). Just reporting this as a bug to fix next spin. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

DFDL variables as path steps and with predicates
by Mike Beckerle 12 Sep '12

12 Sep '12

Currently the DFDL spec's grammar productions are quite liberal about where a VarRef can appear: PathExpr ::= ("/" RelativePathExpr?) | RelativePathExpr RelativePathExpr ::= StepExpr (("/") StepExpr)* StepExpr ::= FilterExpr | AxisStep AxisStep ::= (ReverseStep | ForwardStep) Predicate FilterExpr ::= PrimaryExpr Predicate Predicate ::= "[" Expr "]" PrimaryExpr ::= Literal | *VarRef* | ParenthesizedExp in terms of XPath 2.0 syntax, you could write $x/a/b/$y/../$z. However, the spec also says the type of a variable can only be one of the simple types allowed by DFDL only. So no path steps in the sense of children are meaningful after a DFDL variable. Furthermore, variables are all declared at top level. There is no notion of parent nodes for variable values; hence, a/$x is meaningless (or means the same as $x by itself), and $x/.. is similarly meaningless. But, a variable reference can be followed by a predicate. The resulting node set, would either be one node, containing the value of the variable, or zero nodes. For example is { fn:exists($x[. eq 3] } is presumably a boolean valued expression true if variable x's value is 3. Are there any issues here with predicates?? Should we update the expression language productions to enable only sensible use of DFDL variables in expressions or leave it to match XPath 2.0's more general syntax. If we update the productions should we disallow predicates after variable references also? This loses no expressive power, you can still write { if ($x eq 3) then true else false }, which is to say I think there is no inherent capability lost if we require variable references to be atomic expressions that produce exactly a single node value. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

4 5