- dfdl-wg - lists.ogf.org

Question re: dfdl:property element and element form
by Alonza Mumford 09 Aug '12

09 Aug '12

Ref. DFDL v1.0 (Jan 31, 11'), section 7.1.3.2 (Property Binding Syntax: Element Form): I interpret from the reading that dfdl:property is child of dfdl:format, and implicitly that this form is necessary for element form binding of dfdl attributes (properties). Can I then infer that dfdl:format is the only dfld element that can use element form to specify properties (attributes) for itself? Thanks in advance.

2 1

OGF DFDL WG Call Agenda 2012-08-07
by Steve Hanson 06 Aug '12

06 Aug '12

Please find agenda for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

bits and alignment and delimiters
by Mike Beckerle 03 Aug '12

03 Aug '12

Are delimiters implicitly text in the sense of being aligned to an 8-bit boundary? E.g., if I have <element name="threeBits" type="xs:byte" dfdl:alignmentUnits="bits" dfdl:alignment="1" dfdl:lengthUnits="bits" dfdl:length="{3}" dfdl:terminator="%#x55;"/> Does that terminator byte of 0x55 begin at bit 4, immediately after the last bit of the content region, or does it implicitly move up to bit 8? I know the answer if instead of a terminator, the threeBits had no terminator but was followed by an element of type xs:string. Text always has byte alignment (at least). But what about initiators/terminators/separators which are allowed in binary data. I know I really want the answer here to be that we treat delimiters as text always, so byte alignment (minimum). But I don't think the spec discusses the possibility of non-byte alignment of delimiters. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Repeating groups in DFDL with continuation indicators - action 146 - updated
by Steve Hanson 01 Aug '12

01 Aug '12

Here's the proposal for discussion on today's call, updated to fix some issues: - Moved the dfdl:assert onto element A1 where it more naturally belongs - Added a variable $pos to save the dfdl:position() of A1 as this is needed by A1's child repeatBit element - Corrected repeatBit dfdl:outputValueCalc expression to use $pos - Corrected function prefixes Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 30/07/2012 18:30 ----- From: Steve Hanson/UK/IBM To: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, "'Ryan Farrell'" <ryan.farrell.ctr(a)nrl.navy.mil> Date: 27/07/2011 09:29 Subject: RE: [DFDL-WG] [wg-all] Repeating groups in DFDL Hi Mike That looks pretty good. A couple of minor changes in bold red (I don't think there's a length() function). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, "'Ryan Farrell'" <ryan.farrell.ctr(a)nrl.navy.mil> Cc: <dfdl-wg(a)ogf.org> Date: 26/07/2011 18:04 Subject: RE: [DFDL-WG] [wg-all] Repeating groups in DFDL From our call today, I suggest this schema, based on the assertion suggestion from steve/tim, and adding in the unparser/output direction calculations, and putting this together to match Ryan’s example schema. I’m not certain on the xpath expressions syntax. I may have number of “..” moves wrong (at least). I added outputValueCalc properties to the flag bits, I added occursCountKind=’parsed’ to the array, and the assertion to the repeatBit element. I put in the check so that a zero length array is possible based on the presentBit as well. <xs:element name="example"> <xs:annotation><xs:appinfo source="..."> <dfdl:defineVariable name="$pos" type="int"/> </xs:appinfo></xs:annotation> <xs:complexType> <xs:sequence> <xs:element name="presentBit" type="xs:unsignedByte" dfdl:representation="binary" dfdl:alignment="1" dfdl:alignmentUnits="bits" dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:outputValueCalc='{ if dfdl:count(../A1) eq 0 then 0 else 1 }'/> <xs:element name="A1" minOccurs="0" maxOccurs="10" dfdl:occursCountKind='parsed'> <xs:annotation><xs:appinfo source="..."> <dfdl:newVariableInstance ref="$pos" defaultValue="{ dfdl:position() }"/> <dfdl:assert> {if ../presentBit eq 0 then false else if $pos eq 1 then true else if .[$pos-1]/repeatBit eq 0 then false else true} </dfdl:assert> </xs:appinfo></xs:annotation> <xs:complexType> <xs:sequence> <xs:element name="repeatBit" type="xs:unsignedByte" dfdl:representation="binary" dfdl:alignment="1" dfdl:alignmentUnits="bits" dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:outputValueCalc='{ if $pos eq dfdl:count(..) then 0 else 1 }'/> </xs:element> <xs:element name="someField" type="xs:unsignedByte" dfdl:representation="binary" dfdl:alignment="1" dfdl:alignmentUnits="bits" dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" dfdl:length="2"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="aFieldAfterA1" type="xs:unsignedByte" dfdl:representation="binary" dfdl:alignment="1" dfdl:alignmentUnits="bits" dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" dfdl:length="2"/> </xs:sequence> </xs:complexType> </xs:element> From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Tuesday, July 26, 2011 6:40 AM To: Mike Beckerle Cc: dfdl-wg(a)ogf.org; 'Ryan Farrell' Subject: Re: [DFDL-WG] [wg-all] Repeating groups in DFDL Yes we need a friendly way of saying 'I am the last item' when using occursCountKind is 'parsed'. But with what we have today, could we use a discriminator on the repeating element that fails when the indicator of the previous repeat was 0 ? So speculation starts to parse the thing after the last repeat on the assumption it is another repeat and then fails and backtracks. <dfdl:assert>{if dfdl:position() eq 1 then true else if './present[dfdl:position()-1]' eq 0 then false else true}</dfdl:assert> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: "'Ryan Farrell'" <ryan.farrell.ctr(a)nrl.navy.mil>, <dfdl-wg(a)ogf.org> Date: 22/07/2011 23:44 Subject: Re: [DFDL-WG] [wg-all] Repeating groups in DFDL Sent by: dfdl-wg-bounces(a)ogf.org Great example Ryan, We've spent quite a bit of time over the last several months on textual formats with these kinds of characteristics, which I would describe as "occurrances determined by parsing" - that is, you look at the data to determine "is there another element or not" one by one as they are parsed. We need to make sure the properties are there to make this work for the binary case as well. I actually believe we cannot handle this format right now. We don't have a property which says to look at the current element to determine whether to expect a subsequent element, as a way of determining occurrence counts. Next week we'll add resolving this example to the workgroup actions list. ...mikeb Mike Beckerle Senior Technology Leader/Manager Deloitte Managed Analytics 100 Fifth Avenue, Waltham, MA 02451 Tel/Direct: +1 781 330 0412 | Fax: +1 866 253 3006 www.deloitte.com -----Original Message----- From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Ryan Farrell Sent: Tuesday, July 19, 2011 2:12 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] [wg-all] Repeating groups in DFDL For anyone that was at the 19/07 phone meeting, this is what Adam Fox and I were trying to find a solution to. If anyone knows of a solution, please contact me or adam.fox(a)nrl.navy.mil. ---------------------------------------------------------------------------- ------------------------ Please see the attached xml schema while reading this message. The problem we want to solve is how to represent a structure that can have a variable amount of occurences (even zero), and the amount of repeats is controlled by a one bit field, which we will call"repeatBit". The repeating structure in this case is "A1". The element before it, "presentBit", determines if there is at least one occurence of A1. If presentBit is zero, then A1 has zero occurences. If presentBit is one, then A1 has AT LEAST one occurence. The first element in A1 is "repeatBit". What repeatBit does is tells us if there will be another occurence of A1 after the current occurence. When repeatBit is zero, then that means we will read through the rest of A1, then that is the last occurence of A1. As long as repeatBit continues to be one however, we will read through the rest of A1, then start a new occurence. I have included all the known required DFDL notation. Please let me know what is missing. EXAMPLE INPUT (results are in base 10): NOTE: Not all bits are used in these examples. Unused bits should be ignored for the purpose of the examples. 1) 0111 0101 presentBit = 0 aFieldAfterA1 = 3 2) 1011 1010 presentBit = 1 A1 -repeatBit = 0 -someField = 3 aFieldAfterA1 = 2 3) 1101 0110 0000 0000 presentBit = 1 A1 -repeatBit = 1 -someField = 1 A1 -repeatBit = 0 -someField = 3 aFieldAfterA1 = 0 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 3

OGF DFDL WG Call Minutes 2012-07-31
by Steve Hanson 31 Jul '12

31 Jul '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is now at v010 here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Allowable usage of entity class %WSP*;
by Steve Hanson 31 Jul '12

31 Jul '12

We decided in the end to continue to allow %WSP; to appear on its own in the places it is currently permitted, as there are some scenarios where it could prove useful. If scanning, then when no white space is present in the data, a zero length match is returned.. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 31/07/2012 16:17 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 31/07/2012 10:32 Subject: Allowable usage of entity class %WSP*; The DFDL spec is very restrictive about the use of %ES; - it is only permitted as a nil literal value. But the spec does not say anything about the use of %WSP*; when used on its own - which may equivalently be written as "%ES; %WSP+;". I don't think that %WSP*; should be allowed on its own other than for nil literal value. It may of course be used in combination with other DFDL entities. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Allowable usage of entity class %WSP*;
by Steve Hanson 31 Jul '12

31 Jul '12

The DFDL spec is very restrictive about the use of %ES; - it is only permitted as a nil literal value. But the spec does not say anything about the use of %WSP*; when used on its own - which may equivalently be written as "%ES; %WSP+;". I don't think that %WSP*; should be allowed on its own other than for nil literal value. It may of course be used in combination with other DFDL entities. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-07-31
by Steve Hanson 30 Jul '12

30 Jul '12

Please find agenda for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Repeating groups in DFDL with continuation indicators - action 146
by Steve Hanson 30 Jul '12

30 Jul '12

Here's the proposal for discussion on tomorrow's call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 30/07/2012 18:30 ----- From: Steve Hanson/UK/IBM To: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, "'Ryan Farrell'" <ryan.farrell.ctr(a)nrl.navy.mil> Date: 27/07/2011 09:29 Subject: RE: [DFDL-WG] [wg-all] Repeating groups in DFDL Hi Mike That looks pretty good. A couple of minor changes in bold red (I don't think there's a length() function). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, "'Ryan Farrell'" <ryan.farrell.ctr(a)nrl.navy.mil> Cc: <dfdl-wg(a)ogf.org> Date: 26/07/2011 18:04 Subject: RE: [DFDL-WG] [wg-all] Repeating groups in DFDL From our call today, I suggest this schema, based on the assertion suggestion from steve/tim, and adding in the unparser/output direction calculations, and putting this together to match Ryan’s example schema. I’m not certain on the xpath expressions syntax. I may have number of “..” moves wrong (at least). I added outputValueCalc properties to the flag bits, I added occursCountKind=’parsed’ to the array, and the assertion to the repeatBit element. I put in the check so that a zero length array is possible based on the presentBit as well. <xs:element name="example"> <xs:complexType> <xs:sequence> <xs:element name="presentBit" type="xs:unsignedByte" dfdl:representation="binary" dfdl:alignment="1" dfdl:alignmentUnits="bits" dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:outputValueCalc='{ if count(../A1) eq 0 then 0 else 1 }'/> <xs:element name="A1" minOccurs="0" maxOccurs="10" dfdl:occursCountKind='parsed'> <xs:complexType> <xs:sequence> <xs:element name="repeatBit" type="xs:unsignedByte" dfdl:representation="binary" dfdl:alignment="1" dfdl:alignmentUnits="bits" dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:outputValueCalc='{ if exists( ../../A1[position()+1]) then 1 else 0 }'> <xs:annotation><xs:appinfo source="...dfdl url..."> <dfdl:assert>{ if ../../presentBit eq 0 then false else if dfdl:position() eq 1 then true else if ../[dfdl:position()-1]/repeatBit eq 0 then false else true } </dfdl:assert> </xs:appinfo></xs:annotation> </xs:element> <xs:element name="someField" type="xs:unsignedByte" dfdl:representation="binary" dfdl:alignment="1" dfdl:alignmentUnits="bits" dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" dfdl:length="2"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="aFieldAfterA1" type="xs:unsignedByte" dfdl:representation="binary" dfdl:alignment="1" dfdl:alignmentUnits="bits" dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" dfdl:length="2"/> </xs:sequence> </xs:complexType> </xs:element> From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Tuesday, July 26, 2011 6:40 AM To: Mike Beckerle Cc: dfdl-wg(a)ogf.org; 'Ryan Farrell' Subject: Re: [DFDL-WG] [wg-all] Repeating groups in DFDL Yes we need a friendly way of saying 'I am the last item' when using occursCountKind is 'parsed'. But with what we have today, could we use a discriminator on the repeating element that fails when the indicator of the previous repeat was 0 ? So speculation starts to parse the thing after the last repeat on the assumption it is another repeat and then fails and backtracks. <dfdl:assert>{if dfdl:position() eq 1 then true else if './present[dfdl:position()-1]' eq 0 then false else true}</dfdl:assert> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: "'Ryan Farrell'" <ryan.farrell.ctr(a)nrl.navy.mil>, <dfdl-wg(a)ogf.org> Date: 22/07/2011 23:44 Subject: Re: [DFDL-WG] [wg-all] Repeating groups in DFDL Sent by: dfdl-wg-bounces(a)ogf.org Great example Ryan, We've spent quite a bit of time over the last several months on textual formats with these kinds of characteristics, which I would describe as "occurrances determined by parsing" - that is, you look at the data to determine "is there another element or not" one by one as they are parsed. We need to make sure the properties are there to make this work for the binary case as well. I actually believe we cannot handle this format right now. We don't have a property which says to look at the current element to determine whether to expect a subsequent element, as a way of determining occurrence counts. Next week we'll add resolving this example to the workgroup actions list. ...mikeb Mike Beckerle Senior Technology Leader/Manager Deloitte Managed Analytics 100 Fifth Avenue, Waltham, MA 02451 Tel/Direct: +1 781 330 0412 | Fax: +1 866 253 3006 www.deloitte.com -----Original Message----- From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Ryan Farrell Sent: Tuesday, July 19, 2011 2:12 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] [wg-all] Repeating groups in DFDL For anyone that was at the 19/07 phone meeting, this is what Adam Fox and I were trying to find a solution to. If anyone knows of a solution, please contact me or adam.fox(a)nrl.navy.mil. ---------------------------------------------------------------------------- ------------------------ Please see the attached xml schema while reading this message. The problem we want to solve is how to represent a structure that can have a variable amount of occurences (even zero), and the amount of repeats is controlled by a one bit field, which we will call"repeatBit". The repeating structure in this case is "A1". The element before it, "presentBit", determines if there is at least one occurence of A1. If presentBit is zero, then A1 has zero occurences. If presentBit is one, then A1 has AT LEAST one occurence. The first element in A1 is "repeatBit". What repeatBit does is tells us if there will be another occurence of A1 after the current occurence. When repeatBit is zero, then that means we will read through the rest of A1, then that is the last occurence of A1. As long as repeatBit continues to be one however, we will read through the rest of A1, then start a new occurence. I have included all the known required DFDL notation. Please let me know what is missing. EXAMPLE INPUT (results are in base 10): NOTE: Not all bits are used in these examples. Unused bits should be ignored for the purpose of the examples. 1) 0111 0101 presentBit = 0 aFieldAfterA1 = 3 2) 1011 1010 presentBit = 1 A1 -repeatBit = 0 -someField = 3 aFieldAfterA1 = 2 3) 1101 0110 0000 0000 presentBit = 1 A1 -repeatBit = 1 -someField = 1 A1 -repeatBit = 0 -someField = 3 aFieldAfterA1 = 0 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2012-07-25
by Steve Hanson 27 Jul '12

27 Jul '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is now at v010 here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0