Re: [DFDL-WG] Specify field based on the beggining of the
Well yes this structure is ok. The problem is that my next fields will continue after field 4. For instance field5 would start right after the end of field4. Going back to my data description: Header: 512Bytes; (bytes 1-512) Field1: 21504 bytes (21KB); (bytes 513-21016) Field2: 5Bytes; (bytes 21017-21021) Field3: 1Byte; (bytes 21021-21022) Field4: 3 Bytes - Stating at byte 316 counting from the beginning of the file; (bytes 316-318) Field5:10Bytes; (bytes 319-318) ...; bytes (319-...) Basically I would need to do something like fseek function in C language :) (http://www.cplusplus.com/reference/clibrary/cstdio/fseek/). Message: 1 Date: Tue, 18 Sep 2012 09:20:26 +0100 From: Tim Kimber <KIMBERT@uk.ibm.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Message-ID: <OF826A1ADE.83551CD7-ON80257A7D.002D3AE6-80257A7D.002DD59A@uk.ibm.com> Content-Type: text/plain; charset="utf-8" Hi Miguel, I work with Steve, and we were discussing this a moment ago. Would it be acceptable to get this structure in the info set: Document ( complex element ) |---Header ( complex element ) |---|----Field4 ( simple element ) |---Body |---|---Field1 ( simple element ) |---|---Field2 ( simple element ) |---|---Field3 ( simple element ) If you really need to get the entire Header field as a BLOB *and* get Field4 as a separate field then you could achieve that using inputValueCalc. The value of Field4 would be calculated from the BLOB or string value of Header using XPath string operations. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: Miguel Vieira <carlos.vieira@gmv.com>, Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 18:29 Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces@ogf.org It sounds like you need two DFDL schemas and two parse operations. The first schema models the whole message. It takes the whole data stream as input and creates an infoset for Header, Field1, Field2, Field3. The second schema models just Field 4 using leadingSkip and trailingSkip to position correctly. It takes the Header from the first infoset as input, and creates an infoset for Field4. <xs:element name="Field4" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="3" dfdl:lengthUnits="bytes" dfdl:leadingSkip="316" dfdl:trailingSkip="193" dfdl:alignmentUnits="bytes" /> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 18:04 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file Hi Steve! I need them both. From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 18:00 To: Miguel Vieira Cc: dfdl-wg@ogf.org Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file Is Field4 the only part of the Header that is of interest? Do you want the full Header (as a 512 byte BLOB) and Field4 in the infoset? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 16:06 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file First of all: Thanks! J My binary data specification is: Header: 512Bytes; Field1: 21504 bytes (21KB); Field2: 5Bytes; Field3: 1Byte; Field4: 3 Bytes ?Stating at byte 316 counting from the beginning of the file. This means that Field4 is placed inside the Header field. ?leadingSkipBytes? or ?alignment? would do, but can you specify the place you start counting? Thanks. Miguel From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 15:41 To: Miguel Vieira Cc: dfdl-wg@ogf.org Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Miguel, please can you send us a graphic of the data in question, I want to make sure I have understood your data correctly. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 15:03 Subject: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces@ogf.org Hi! I have a binary file specification that specifies some fields referenced by the beginning of the binary info. For instance imagine that there is a field with 1024Bytes and the next one starts at byte 512 counting from the beginning of the info. Is it possible to do that using dfdl? I?ve seen dfdl:floating, but I can?t find much info on that. Thanks. P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informaci?n clasificada por su emisor como confidencial en el marco de su Sistema de Gesti?n de Seguridad de la Informaci?n siendo para uso exclusivo del destinatario, quedando prohibida su divulgaci?n copia o distribuci?n a terceros sin la autorizaci?n expresa del remitente. Si Vd. ha recibido este mensaje err?neamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboraci?n. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informa??o confidencial, de acordo com nosso Sistema de Gest?o de Seguran?a da Informa??o, sendo para uso exclusivo do destinat?rio e estando proibida a sua divulga??o, c?pia ou distribui??o a terceiros sem autoriza??o expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colabora??o. -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
A DFDL model describes a hierarchical data format, just like XML schema, so you just do this: Document ( complex element ) |---Header ( complex element ) |---|----Field4 ( simple element ) |---|----Field5 ( simple element ) |---Field1 ( simple element ) |---Field2 ( simple element ) |---Field3 ( simple element ) Header is just an example of a complex element, ie, a structure, which contains child elements. To complete the picture some of the DFDL properties help: Document dfdl:lengthKind="implicit" |---Header dfdl:lengthKind="explicit" dfdl:length="512" |---|----Field4 dfdl:lengthKind="explicit" dfdl:length="3" dfdl:leadingSkip="316" |---|----Field5 dfdl:lengthKind="explicit" dfdl:length="10" |---|----Field6 dfdl:lengthKind="explicit" dfdl:length="..." |---|----etc |---Field1 dfdl:lengthKind="explicit" dfdl:length="21504" |---Field2 dfdl:lengthKind="explicit" dfdl:length="5" |---Field3 dfdl:lengthKind="explicit" dfdl:length="1" After the last child of Header, the DFDL parser will automatically skip to the beginning of Field1. If you need to access the data in Header before Field4, change your model to look like: Document dfdl:lengthKind="implicit" |---Header dfdl:lengthKind="explicit" dfdl:length="512" |---|----Unused dfdl:lengthKind="explicit" dfdl:length="316" |---|----Field4 dfdl:lengthKind="explicit" dfdl:length="3" |---|----Field5 dfdl:lengthKind="explicit" dfdl:length="10" |---|----Field6 dfdl:lengthKind="explicit" dfdl:length="..." |---|----etc |---Field1 dfdl:lengthKind="explicit" dfdl:length="21504" |---Field2 dfdl:lengthKind="explicit" dfdl:length="5" |---Field3 dfdl:lengthKind="explicit" dfdl:length="1" Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 19/09/2012 09:38 Subject: Re: [DFDL-WG] Specify field based on the beggining of the Sent by: dfdl-wg-bounces@ogf.org Well yes this structure is ok. The problem is that my next fields will continue after field 4. For instance field5 would start right after the end of field4. Going back to my data description: Header: 512Bytes; (bytes 1-512) Field1: 21504 bytes (21KB); (bytes 513-21016) Field2: 5Bytes; (bytes 21017-21021) Field3: 1Byte; (bytes 21021-21022) Field4: 3 Bytes - Stating at byte 316 counting from the beginning of the file; (bytes 316-318) Field5:10Bytes; (bytes 319-318) ...; bytes (319-...) Basically I would need to do something like fseek function in C language :) (http://www.cplusplus.com/reference/clibrary/cstdio/fseek/). Message: 1 Date: Tue, 18 Sep 2012 09:20:26 +0100 From: Tim Kimber <KIMBERT@uk.ibm.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Message-ID: <OF826A1ADE.83551CD7-ON80257A7D.002D3AE6-80257A7D.002DD59A@uk.ibm.com> Content-Type: text/plain; charset="utf-8" Hi Miguel, I work with Steve, and we were discussing this a moment ago. Would it be acceptable to get this structure in the info set: Document ( complex element ) |---Header ( complex element ) |---|----Field4 ( simple element ) |---Body |---|---Field1 ( simple element ) |---|---Field2 ( simple element ) |---|---Field3 ( simple element ) If you really need to get the entire Header field as a BLOB *and* get Field4 as a separate field then you could achieve that using inputValueCalc. The value of Field4 would be calculated from the BLOB or string value of Header using XPath string operations. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: Miguel Vieira <carlos.vieira@gmv.com>, Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 18:29 Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces@ogf.org It sounds like you need two DFDL schemas and two parse operations. The first schema models the whole message. It takes the whole data stream as input and creates an infoset for Header, Field1, Field2, Field3. The second schema models just Field 4 using leadingSkip and trailingSkip to position correctly. It takes the Header from the first infoset as input, and creates an infoset for Field4. <xs:element name="Field4" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="3" dfdl:lengthUnits="bytes" dfdl:leadingSkip="316" dfdl:trailingSkip="193" dfdl:alignmentUnits="bytes" /> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 18:04 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file Hi Steve! I need them both. From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 18:00 To: Miguel Vieira Cc: dfdl-wg@ogf.org Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file Is Field4 the only part of the Header that is of interest? Do you want the full Header (as a 512 byte BLOB) and Field4 in the infoset? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 16:06 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file First of all: Thanks! J My binary data specification is: Header: 512Bytes; Field1: 21504 bytes (21KB); Field2: 5Bytes; Field3: 1Byte; Field4: 3 Bytes ?Stating at byte 316 counting from the beginning of the file. This means that Field4 is placed inside the Header field. ?leadingSkipBytes? or ?alignment? would do, but can you specify the place you start counting? Thanks. Miguel From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 15:41 To: Miguel Vieira Cc: dfdl-wg@ogf.org Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Miguel, please can you send us a graphic of the data in question, I want to make sure I have understood your data correctly. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 15:03 Subject: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces@ogf.org Hi! I have a binary file specification that specifies some fields referenced by the beginning of the binary info. For instance imagine that there is a field with 1024Bytes and the next one starts at byte 512 counting from the beginning of the info. Is it possible to do that using dfdl? I?ve seen dfdl:floating, but I can?t find much info on that. Thanks. P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informaci?n clasificada por su emisor como confidencial en el marco de su Sistema de Gesti?n de Seguridad de la Informaci?n siendo para uso exclusivo del destinatario, quedando prohibida su divulgaci?n copia o distribuci?n a terceros sin la autorizaci?n expresa del remitente. Si Vd. ha recibido este mensaje err?neamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboraci?n. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informa??o confidencial, de acordo com nosso Sistema de Gest?o de Seguran?a da Informa??o, sendo para uso exclusivo do destinat?rio e estando proibida a sua divulga??o, c?pia ou distribui??o a terceiros sem autoriza??o expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colabora??o. -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
A DFDL parser walks the schema starting at the specified global element, and moving depth-first, left to right. DFDL does not allow you to specify absolute offsets from the start of the stream, so your DFDL schema must declare the elements in the same order in which they occur in the data stream. In your example Field4 and Field5 are located within the header. In your DFDL xsd you should declare Field4 and Field5 as child elements within Header's complex type. It sounds as if you are trying to combine the parsing of the data with some transformation. The consumer wants the data presented as Field1, Field2, Field3, Field4, Field5 but in the data Field4 and Field5 appear first. In general, DFDL does not attempt to solve that problem.because there are some very powerful, standardized transformation languages that already exist ( XSLT, XPath, XQuery ). regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Miguel Vieira <carlos.vieira@gmv.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org>, Date: 19/09/2012 09:38 Subject: Re: [DFDL-WG] Specify field based on the beggining of the Sent by: dfdl-wg-bounces@ogf.org Well yes this structure is ok. The problem is that my next fields will continue after field 4. For instance field5 would start right after the end of field4. Going back to my data description: Header: 512Bytes; (bytes 1-512) Field1: 21504 bytes (21KB); (bytes 513-21016) Field2: 5Bytes; (bytes 21017-21021) Field3: 1Byte; (bytes 21021-21022) Field4: 3 Bytes - Stating at byte 316 counting from the beginning of the file; (bytes 316-318) Field5:10Bytes; (bytes 319-318) ...; bytes (319-...) Basically I would need to do something like fseek function in C language :) (http://www.cplusplus.com/reference/clibrary/cstdio/fseek/). Message: 1 Date: Tue, 18 Sep 2012 09:20:26 +0100 From: Tim Kimber <KIMBERT@uk.ibm.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Message-ID: <OF826A1ADE.83551CD7-ON80257A7D.002D3AE6-80257A7D.002DD59A@uk.ibm.com> Content-Type: text/plain; charset="utf-8" Hi Miguel, I work with Steve, and we were discussing this a moment ago. Would it be acceptable to get this structure in the info set: Document ( complex element ) |---Header ( complex element ) |---|----Field4 ( simple element ) |---Body |---|---Field1 ( simple element ) |---|---Field2 ( simple element ) |---|---Field3 ( simple element ) If you really need to get the entire Header field as a BLOB *and* get Field4 as a separate field then you could achieve that using inputValueCalc. The value of Field4 would be calculated from the BLOB or string value of Header using XPath string operations. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: Miguel Vieira <carlos.vieira@gmv.com>, Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 18:29 Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces@ogf.org It sounds like you need two DFDL schemas and two parse operations. The first schema models the whole message. It takes the whole data stream as input and creates an infoset for Header, Field1, Field2, Field3. The second schema models just Field 4 using leadingSkip and trailingSkip to position correctly. It takes the Header from the first infoset as input, and creates an infoset for Field4. <xs:element name="Field4" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="3" dfdl:lengthUnits="bytes" dfdl:leadingSkip="316" dfdl:trailingSkip="193" dfdl:alignmentUnits="bytes" /> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 18:04 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file Hi Steve! I need them both. From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 18:00 To: Miguel Vieira Cc: dfdl-wg@ogf.org Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file Is Field4 the only part of the Header that is of interest? Do you want the full Header (as a 512 byte BLOB) and Field4 in the infoset? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 16:06 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file First of all: Thanks! J My binary data specification is: Header: 512Bytes; Field1: 21504 bytes (21KB); Field2: 5Bytes; Field3: 1Byte; Field4: 3 Bytes ?Stating at byte 316 counting from the beginning of the file. This means that Field4 is placed inside the Header field. ?leadingSkipBytes? or ?alignment? would do, but can you specify the place you start counting? Thanks. Miguel From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 15:41 To: Miguel Vieira Cc: dfdl-wg@ogf.org Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Miguel, please can you send us a graphic of the data in question, I want to make sure I have understood your data correctly. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Miguel Vieira <carlos.vieira@gmv.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 15:03 Subject: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces@ogf.org Hi! I have a binary file specification that specifies some fields referenced by the beginning of the binary info. For instance imagine that there is a field with 1024Bytes and the next one starts at byte 512 counting from the beginning of the info. Is it possible to do that using dfdl? I?ve seen dfdl:floating, but I can?t find much info on that. Thanks. P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informaci?n clasificada por su emisor como confidencial en el marco de su Sistema de Gesti?n de Seguridad de la Informaci?n siendo para uso exclusivo del destinatario, quedando prohibida su divulgaci?n copia o distribuci?n a terceros sin la autorizaci?n expresa del remitente. Si Vd. ha recibido este mensaje err?neamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboraci?n. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informa??o confidencial, de acordo com nosso Sistema de Gest?o de Seguran?a da Informa??o, sendo para uso exclusivo do destinat?rio e estando proibida a sua divulga??o, c?pia ou distribui??o a terceiros sem autoriza??o expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colabora??o. -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Ok, so I believe I understand the problem. You have a specification in terms of either absolute offsets, or relative offsets from absolute anchor fields. While you could construct a DFDL schema by simply figuring out the length of each field, and putting suitable leading/trailing skip in to take up any unused space, the specification you have does not match that style. So the correspondence between your specification and the DFDL schema would not be obvious. So, while it is possible to create a schema this way, a schema created by translating your specification that way would be fragile. One error in the middle and all subsequent fields would be wrong. I have seen this problem in the past also, as well as mixes of things that are 'back to back' fields, where each starts immediately after the previous, with things that have absolute or relative offsets specified. Basically, if you create a big enough data dictionary, you want to use both styles to avoid this "error in the middle throws everything off" problem. It would be greatly preferable have the option to simply put a DFDL annotation on each element giving its start position as an offset (measured in lengthUnits) relative to the start of some other element. Then an error in the middle would throw off the value of that one field only. At one time we did have a dfdl feature for such a capability. It was either dfdl:offset and dfdl:offsetFrom, or dfdl:startPosition or something like that. I believe we dropped it because we were desperate to just reduce the feature count on DFDL because the specification was so large already. On Wed, Sep 19, 2012 at 4:24 AM, Miguel Vieira <carlos.vieira@gmv.com>wrote:
Well yes this structure is ok.
The problem is that my next fields will continue after field 4. For instance field5 would start right after the end of field4. Going back to my data description:
Header: 512Bytes; (bytes 1-512) Field1: 21504 bytes (21KB); (bytes 513-21016) Field2: 5Bytes; (bytes 21017-21021) Field3: 1Byte; (bytes 21021-21022) Field4: 3 Bytes - Stating at byte 316 counting from the beginning of the file; (bytes 316-318) Field5:10Bytes; (bytes 319-318) ...; bytes (319-...)
Basically I would need to do something like fseek function in C language :) (http://www.cplusplus.com/reference/clibrary/cstdio/fseek/).
Message: 1 Date: Tue, 18 Sep 2012 09:20:26 +0100 From: Tim Kimber <KIMBERT@uk.ibm.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Message-ID: < OF826A1ADE.83551CD7-ON80257A7D.002D3AE6-80257A7D.002DD59A@uk.ibm.com> Content-Type: text/plain; charset="utf-8"
Hi Miguel,
I work with Steve, and we were discussing this a moment ago.
Would it be acceptable to get this structure in the info set:
Document ( complex element ) |---Header ( complex element ) |---|----Field4 ( simple element ) |---Body |---|---Field1 ( simple element ) |---|---Field2 ( simple element ) |---|---Field3 ( simple element )
If you really need to get the entire Header field as a BLOB *and* get Field4 as a separate field then you could achieve that using inputValueCalc. The value of Field4 would be calculated from the BLOB or string value of Header using XPath string operations.
regards,
Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 37246742
From: Steve Hanson/UK/IBM@IBMGB To: Miguel Vieira <carlos.vieira@gmv.com>, Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 18:29 Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces@ogf.org
It sounds like you need two DFDL schemas and two parse operations. The first schema models the whole message. It takes the whole data stream as input and creates an infoset for Header, Field1, Field2, Field3. The second schema models just Field 4 using leadingSkip and trailingSkip to position correctly. It takes the Header from the first infoset as input, and creates an infoset for Field4.
<xs:element name="Field4" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="3" dfdl:lengthUnits="bytes" dfdl:leadingSkip="316" dfdl:trailingSkip="193" dfdl:alignmentUnits="bytes" />
Regards
Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848
From: Miguel Vieira <carlos.vieira@gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 18:04 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file
Hi Steve!
I need them both.
From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 18:00 To: Miguel Vieira Cc: dfdl-wg@ogf.org Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file
Is Field4 the only part of the Header that is of interest? Do you want the full Header (as a 512 byte BLOB) and Field4 in the infoset?
Regards
Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848
From: Miguel Vieira <carlos.vieira@gmv.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 16:06 Subject: RE: [DFDL-WG] Specify field based on the beggining of the binary file
First of all: Thanks! J
My binary data specification is: Header: 512Bytes; Field1: 21504 bytes (21KB); Field2: 5Bytes; Field3: 1Byte; Field4: 3 Bytes ?Stating at byte 316 counting from the beginning of the file.
This means that Field4 is placed inside the Header field.
?leadingSkipBytes? or ?alignment? would do, but can you specify the place you start counting?
Thanks. Miguel
From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: segunda-feira, 17 de Setembro de 2012 15:41 To: Miguel Vieira Cc: dfdl-wg@ogf.org Subject: Re: [DFDL-WG] Specify field based on the beggining of the binary file
Miguel, please can you send us a graphic of the data in question, I want to make sure I have understood your data correctly.
Regards
Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848
From: Miguel Vieira <carlos.vieira@gmv.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 17/09/2012 15:03 Subject: [DFDL-WG] Specify field based on the beggining of the binary file Sent by: dfdl-wg-bounces@ogf.org
Hi!
I have a binary file specification that specifies some fields referenced by the beginning of the binary info. For instance imagine that there is a field with 1024Bytes and the next one starts at byte 512 counting from the beginning of the info.
Is it possible to do that using dfdl? I?ve seen dfdl:floating, but I can?t find much info on that.
Thanks.
P Please consider the environment before printing this e-mail.
This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it.
Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informaci?n clasificada por su emisor como confidencial en el marco de su Sistema de Gesti?n de Seguridad de la Informaci?n siendo para uso exclusivo del destinatario, quedando prohibida su divulgaci?n copia o distribuci?n a terceros sin la autorizaci?n expresa del remitente. Si Vd. ha recibido este mensaje err?neamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboraci?n.
Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informa??o confidencial, de acordo com nosso Sistema de Gest?o de Seguran?a da Informa??o, sendo para uso exclusivo do destinat?rio e estando proibida a sua divulga??o, c?pia ou distribui??o a terceiros sem autoriza??o expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colabora??o.
-- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
P Please consider the environment before printing this e-mail.
This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it.
Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin.
Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao.
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
P Please consider the environment before printing this e-mail. This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacin clasificada por su emisor como confidencial en el marco de su Sistema de Gestin de Seguridad de la Informacin siendo para uso exclusivo del destinatario, quedando prohibida su divulgacin copia o distribucin a terceros sin la autorizacin expresa del remitente. Si Vd. ha recibido este mensaje errneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracin. Esta mensagem, incluindo qualquer ficheiro anexo, pode conter informao confidencial, de acordo com nosso Sistema de Gesto de Segurana da Informao, sendo para uso exclusivo do destinatrio e estando proibida a sua divulgao, cpia ou distribuio a terceiros sem autorizao expressa do remetente da mesma. Se recebeu esta mensagem por engano, por favor avise de imediato o remetente e apague-a. Obrigado pela sua colaborao.
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
participants (4)
-
Miguel Vieira -
Mike Beckerle -
Steve Hanson -
Tim Kimber