- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2012-09-18 & follow-up
by Steve Hanson 20 Sep '12

20 Sep '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is now at v010 here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Specify field based on the beggining of the
by Miguel Vieira 19 Sep '12

19 Sep '12

Well yes this structure is ok. The problem is that my next fields will continue after field 4. For instance field5 would start right after the end of field4. Going back to my data description: Header: 512Bytes; (bytes 1-512) Field1: 21504 bytes (21KB); (bytes 513-21016) Field2: 5Bytes; (bytes 21017-21021) Field3: 1Byte; (bytes 21021-21022) Field4: 3 Bytes - Stating at byte 316 counting from the beginning of the file; (bytes 316-318) Field5:10Bytes; (bytes 319-318) ...; bytes (319-...) Basically I would need to do something like fseek function in C language :) (http://www.cplusplus.com/reference/clibrary/cstdio/fseek/) Message: 1 Date: Tue, 18 Sep 2012 09:20:26 +0100 From: Tim Kimber <KIMBERT(a)uk.ibm.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Message-ID: <OF826A1ADE.83551CD7-ON80257A7D.002D3AE6-80257A7D.002DD59A(a)uk.ibm.com> Content-Type: text/plain; charset="utf-8" Hi Miguel, I work with Steve, and we were discussing this a moment ago. Would it be acceptable to get this structure in the info set: Document ( complex element ) |---Header ( complex element ) |---|----Field4 ( simple element ) |---Body |---|---Field1 ( simple element ) |---|---Field2 ( simple element ) |---|---Field3 ( simple element ) If you really need to get the entire Header field as a BLOB *and* get Field4 as a separate field then you could achieve that using inputValueCalc. The value of Field4 would be calculated from the BLOB or string value of Header using XPath string operations. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: Miguel Vieira <carlos.vieira(a)gmv.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/09/2012 18:29 Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces(a)ogf.org It sounds like you need two DFDL schemas and two parse operations. The first schema models the whole message. It takes the whole data stream as input and creates an infoset for Header, Field1, Field2, Field3. The second schema models just Field 4 using leadingSkip and trailingSkip to position correctly. It takes the Header from the first infoset as input, and creates an infoset for Field4. <xs:element name="Field4" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="3" dfdl:lengthUnits="bytes" dfdl:leadingSkip="316" dfdl:trailingSkip="193" dfdl:alignmentUnits="bytes" /> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira(a)gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/09/2012 18:04 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file Hi Steve! I need them both. From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 18:00 To: Miguel Vieira Cc: dfdl-wg(a)ogf.org Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file Is Field4 the only part of the Header that is of interest? Do you want the full Header (as a 512 byte BLOB) and Field4 in the infoset? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira(a)gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/09/2012 16:06 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file First of all: Thanks! J My binary data specification is: Header: 512Bytes; Field1: 21504 bytes (21KB); Field2: 5Bytes; Field3: 1Byte; Field4: 3 Bytes ?Stating at byte 316 counting from the beginning of the file. This means that Field4 is placed inside the Header field. ?leadingSkipBytes? or ?alignment? would do, but can you specify the place you start counting? Thanks. Miguel From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 15:41 To: Miguel Vieira Cc: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Miguel, please can you send us a graphic of the data in question, I want to make sure I have understood your data correctly. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira(a)gmv.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/09/2012 15:03 Subject: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces(a)ogf.org Hi! I have a binary file specification that specifies some fields referenced by the beginning of the binary info. For instance imagine that there is a field with 1024Bytes and the next one starts at byte 512 counting from the beginning of the info. Is it possible to do that using dfdl? I?ve seen dfdl:floating, but I can?t find much info on that. Thanks. P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informaci?n clasificada por su emisor como confidencial en el marco de su Sistema de Gesti?n de Seguridad de la Informaci?n siendo para uso exclusivo del destinatario, quedando prohibida su divulgaci?n copia o distribuci?n a terceros sin la autorizaci?n expresa del remitente. Si Vd. ha recibido este mensaje err?neamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboraci?n. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informa??o confidencial, de acordo com nosso Sistema de Gest?o de Seguran?a da Informa??o, sendo para uso exclusivo do destinat?rio e estando proibida a sua divulga??o, c?pia ou distribui??o a terceiros sem autoriza??o expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colabora??o. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 3

OGF DFDL WG Call Agenda 2012-09-18 - follow on call to be held
by Steve Hanson 18 Sep '12

18 Sep '12

We didn't get through all the agenda items so another call will take place on Thursday at 13:30 UK. Same agenda on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (continued) (20 Sep 13:30 GDT in IBM Hursley DE2K09)
by Steve Hanson 18 Sep '12

18 Sep '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Had to reschedule

1 0

Fw: Transaction log data types from IBM 4690 point-of-sale terminals - add to DFDL spec?
by Steve Hanson 18 Sep '12

18 Sep '12

DFDL WG agreed to add the new enum to binaryNumberRep and binaryCalendarRep. Errata will be taken. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/09/2012 16:43 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 17/09/2012 15:23 Subject: Transaction log data types from IBM 4690 point-of-sale terminals - add to DFDL spec? IBM 4690 data format IBM 4690 is a widely used retail Point-of-Sale system. The transactions logs emitted by 4690 are in a compact, ASCII delimited binary format, known as TLOG. Each TLOG consists of a number of records, called 'strings'. Each 'string' consists of fields delimited by a colon (x2C), although some strings use double quote/colon/double quote instead (x22x2Cx22). The fields themselves can be fixed length or variable length but the delimiter is always present. The fields can have a number of physical reps but the most frequent rep is a packed number rep that is not BCD or S/390 packed decimal, and so DFDL can't model it. It is used to represent the vast majority of decimal and integer numbers in TLOG data. This packed decimal format is best described by the operations that act on the data: Pack: Take an ASCII string of characters, drop the first nibble of each byte, shuffle the remaining nibbles to the right, and if there is an odd number of nibbles pad with an xF nibble on the left. Unpack: Discard any leading xF nibble, and expand each nibble into a byte by adding a x3 nibble in front, giving an ASCII string. It can be seen that while the pack operation may be applied to any ASCII string, the unpack operation will only round trip if the original characters are in the range x30 to x3E, hence the operations were only intended to be used for packing and unpacking unsigned numbers. A convention has arisen however whereby the minus sign x2D may be used to indicate negative values, even though xD unpacks as x3D which is the equals sign. In summary: Nibbles represent digits 0 - 9 in the usual BCD manner A positive value is simply indicated by digits A negative number is indicated by digits with the leading nibble being xD If a positive or negative value packs to an odd number of nibbles, pad on the left with xF Some examples help: 12 x12 123 xF1x23 -12 xFDx12 -123 xD1x23 Note that x00xD1x23 is not valid as this would unpack to 0=123 (ie, 0-123). The only nibble that can appear to the left of an xD nibble is an xF pad nibble. Or put another way, you can't blindly byte pad with x00 to make up a fixed length. If such byte padding is required, replace any xF and xD nibbles with x0 nibble, then byte pad with x00, then reinstate any xD nibble in the new leftmost nibble. Proposal to add support to DFDL: Existing dfdl:binaryNumberRep property gets a new enum 'ibm4690Packed'. It applies to all signed and unsigned simple types derived from xsd:decimal (both built-in and user-defined). The following DFDL properties are used in conjunction: dfdl:byteOrder (although in practice IBM 4690 is always big-endian) dfdl:binaryDecimalVirtualPoint Note that dfdl:binaryPackedSignCodes does not apply - the only negative nibble is xD - and dfdl:binaryNumberCheckPolicy has no effect. Where the DFDL specification provides for general behaviours for 'packed' and 'bcd', those behaviours apply also to 'ibm4690Packed'. Specifically: The same dfdl:lengthKind enums and rules apply. No rounding when serializing, a value that can't be accommodated is a processing error. If the logical type is unsigned and a negative value is received, it is a processing error. The same should be done for dfdl:binaryCalendarRep to allow TLOG packed decimal representation for calendars. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Broadcast: OGF DFDL Working Group weekly call (continued) (20 Sep 16:00 GDT in IBM Hursley DE2K09)
by Steve Hanson 18 Sep '12

18 Sep '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0

OGF DFDL WG Call Agenda 2012-09-18
by Steve Hanson 18 Sep '12

18 Sep '12

Please find agenda for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Specify field based on the beggining of the binary file
by Miguel Vieira 18 Sep '12

18 Sep '12

Hi! I have a binary file specification that specifies some fields referenced by the beginning of the binary info. For instance imagine that there is a field with 1024Bytes and the next one starts at byte 512 counting from the beginning of the info. Is it possible to do that using dfdl? I've seen dfdl:floating, but I can't find much info on that. Thanks. ________________________________ [Description: cid:image001.jpg@01CB4ADC.488557F0] Miguel Vieira Software Engineer carlos.vieira(a)gmv.com<mailto:carlos.vieira@gmv.com> GMV-SKYSOFT Torre Fernão de Magalhães Av. D. João II Lote 1.17.02, 7º Andar 1998 - 025 Lisboa Portugal Tlm. +351 962 060 326 Tel. +351 21 382 93 66 Fax +351 21 386 64 93 www.gmv.com.pt<http://www.gmv.com.pt> P Please consider the environment before printing this e-mail. ______________________ This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. ______________________ Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacion clasificada por su emisor como confidencial en el marco de su Sistema de Gestion de Seguridad de la Informacion siendo para uso exclusivo del destinatario, quedando prohibida su divulgacion copia o distribucion a terceros sin la autorizacion expresa del remitente. Si Vd. ha recibido este mensaje erroneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracion. ______________________

3 6

Transaction log data types from IBM 4690 point-of-sale terminals - add to DFDL spec?
by Steve Hanson 17 Sep '12

17 Sep '12

IBM 4690 data format IBM 4690 is a widely used retail Point-of-Sale system. The transactions logs emitted by 4690 are in a compact, ASCII delimited binary format, known as TLOG. Each TLOG consists of a number of records, called 'strings'. Each 'string' consists of fields delimited by a colon (x2C), although some strings use double quote/colon/double quote instead (x22x2Cx22). The fields themselves can be fixed length or variable length but the delimiter is always present. The fields can have a number of physical reps but the most frequent rep is a packed number rep that is not BCD or S/390 packed decimal, and so DFDL can't model it. It is used to represent the vast majority of decimal and integer numbers in TLOG data. This packed decimal format is best described by the operations that act on the data: Pack: Take an ASCII string of characters, drop the first nibble of each byte, shuffle the remaining nibbles to the right, and if there is an odd number of nibbles pad with an xF nibble on the left. Unpack: Discard any leading xF nibble, and expand each nibble into a byte by adding a x3 nibble in front, giving an ASCII string. It can be seen that while the pack operation may be applied to any ASCII string, the unpack operation will only round trip if the original characters are in the range x30 to x3E, hence the operations were only intended to be used for packing and unpacking unsigned numbers. A convention has arisen however whereby the minus sign x2D may be used to indicate negative values, even though xD unpacks as x3D which is the equals sign. In summary: Nibbles represent digits 0 - 9 in the usual BCD manner A positive value is simply indicated by digits A negative number is indicated by digits with the leading nibble being xD If a positive or negative value packs to an odd number of nibbles, pad on the left with xF Some examples help: 12 x12 123 xF1x23 -12 xFDx12 -123 xD1x23 Note that x00xD1x23 is not valid as this would unpack to 0=123 (ie, 0-123). The only nibble that can appear to the left of an xD nibble is an xF pad nibble. Or put another way, you can't blindly byte pad with x00 to make up a fixed length. If such byte padding is required, replace any xF and xD nibbles with x0 nibble, then byte pad with x00, then reinstate any xD nibble in the new leftmost nibble. Proposal to add support to DFDL: Existing dfdl:binaryNumberRep property gets a new enum 'ibm4690Packed'. It applies to all signed and unsigned simple types derived from xsd:decimal (both built-in and user-defined). The following DFDL properties are used in conjunction: dfdl:byteOrder (although in practice IBM 4690 is always big-endian) dfdl:binaryDecimalVirtualPoint Note that dfdl:binaryPackedSignCodes does not apply - the only negative nibble is xD - and dfdl:binaryNumberCheckPolicy has no effect. Where the DFDL specification provides for general behaviours for 'packed' and 'bcd', those behaviours apply also to 'ibm4690Packed'. Specifically: The same dfdl:lengthKind enums and rules apply. No rounding when serializing, a value that can't be accommodated is a processing error. If the logical type is unsigned and a negative value is received, it is a processing error. The same should be done for dfdl:binaryCalendarRep to allow TLOG packed decimal representation for calendars. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

assert and discriminator - no more before/after
by Mike Beckerle 17 Sep '12

17 Sep '12

I am looking in the spec for guidance about the evaluation order of assert statements. We used to have before/after control properties, but eliminated them. If I annotate a simpleType'd element with an assert that says { . eq 'x' }, that of necessity references the current value, so must execute after the value has been computed. If on the other hand I annotate a complexType element with a discriminator that says { ../flag eq 'C1' } then this of necessity must execute before I go after the contents because the whole point is to evalutate the discriminator first. Did we ever articulate exactly what the rules are here about order of evaluation? Thanks for reminders ..mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 4