Repeating groups in DFDL with continuation indicators - action 146 - updated - dfdl-wg

31 Jul 2012

      Here's the proposal for discussion on today's call, updated to fix some 
issues:

- Moved the dfdl:assert onto element A1 where it more naturally belongs
- Added a variable $pos to save the dfdl:position() of A1 as this is 
needed by A1's child repeatBit element
- Corrected repeatBit dfdl:outputValueCalc expression to use $pos
- Corrected function prefixes 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
----- Forwarded by Steve Hanson/UK/IBM on 30/07/2012 18:30 -----

From:   Steve Hanson/UK/IBM
To:     "Mike Beckerle" <mbeckerle.dfdl@gmail.com>
Cc:     dfdl-wg@ogf.org, "'Ryan Farrell'" <ryan.farrell.ctr@nrl.navy.mil>
Date:   27/07/2011 09:29
Subject:        RE: [DFDL-WG] [wg-all] Repeating groups in DFDL

Hi Mike

That looks pretty good.  A couple of minor changes in bold red (I don't 
think there's a length() function).

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848

From:
"Mike Beckerle" <mbeckerle.dfdl@gmail.com>
To:
Steve Hanson/UK/IBM@IBMGB, "'Ryan Farrell'" 
<ryan.farrell.ctr@nrl.navy.mil>
Cc:
<dfdl-wg@ogf.org>
Date:
26/07/2011 18:04
Subject:
RE: [DFDL-WG] [wg-all] Repeating groups in DFDL

From our call today, I suggest this schema, based on the assertion 
suggestion from steve/tim, and adding in the unparser/output direction 
calculations, and putting this together to match Ryan’s example schema. 
I’m not certain on the xpath expressions syntax. I may have number of “..” 
moves wrong (at least).

I added outputValueCalc properties to the flag bits, I added 
occursCountKind=’parsed’ to the array, and the assertion to the repeatBit 
element. I put in the check so that a zero length array is possible based 
on the presentBit as well. 

  <xs:element name="example">
   <xs:annotation><xs:appinfo source="...">
      <dfdl:defineVariable name="$pos" type="int"/> 
   </xs:appinfo></xs:annotation>
   <xs:complexType>
     <xs:sequence>

       <xs:element name="presentBit" type="xs:unsignedByte" 
                   dfdl:representation="binary" dfdl:alignment="1" 
dfdl:alignmentUnits="bits" 
                   dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" 
dfdl:length="1"
                   dfdl:outputValueCalc='{ if dfdl:count(../A1) eq 0 then 
0 else 1 }'/>

       <xs:element name="A1" minOccurs="0" maxOccurs="10"
                   dfdl:occursCountKind='parsed'> 
         <xs:annotation><xs:appinfo source="...">
           <dfdl:newVariableInstance ref="$pos" defaultValue="{ 
dfdl:position() }"/> 
           <dfdl:assert>
                   {if ../presentBit eq 0 then false
                    else if $pos eq 1 then true 
                    else if .[$pos-1]/repeatBit eq 0 then false 
                    else true}
           </dfdl:assert> 
         </xs:appinfo></xs:annotation>
         <xs:complexType>
           <xs:sequence> 

             <xs:element name="repeatBit" type="xs:unsignedByte" 
                         dfdl:representation="binary" dfdl:alignment="1" 
dfdl:alignmentUnits="bits" 
                         dfdl:byteOrder="bigEndian" 
dfdl:lengthKind="explicit" dfdl:length="1"
                         dfdl:outputValueCalc='{ if $pos eq dfdl:count(..) 
then 0 else 1 }'/>
             </xs:element>

             <xs:element name="someField" type="xs:unsignedByte" 
                         dfdl:representation="binary" dfdl:alignment="1" 
dfdl:alignmentUnits="bits" 
                         dfdl:byteOrder="bigEndian" 
dfdl:lengthKind="explicit" dfdl:length="2"/>

           </xs:sequence>
         </xs:complexType>
       </xs:element>

       <xs:element name="aFieldAfterA1" type="xs:unsignedByte" 
                   dfdl:representation="binary" dfdl:alignment="1" 
dfdl:alignmentUnits="bits" 
                   dfdl:byteOrder="bigEndian" dfdl:lengthKind="explicit" 
dfdl:length="2"/>

     </xs:sequence>
   </xs:complexType>
 </xs:element>

From: Steve Hanson [mailto:smh@uk.ibm.com] 
Sent: Tuesday, July 26, 2011 6:40 AM
To: Mike Beckerle
Cc: dfdl-wg@ogf.org; 'Ryan Farrell'
Subject: Re: [DFDL-WG] [wg-all] Repeating groups in DFDL

Yes we need  a friendly way of saying 'I am the last item' when using 
occursCountKind is 'parsed'.   

But with what we have today, could we use a discriminator on the repeating 
element that fails when the indicator of the previous repeat was 0 ? So 
speculation starts to parse the thing after the last repeat on the 
assumption it is another repeat and then fails and backtracks. 

<dfdl:assert>{if dfdl:position() eq 1 then true else if 
'./present[dfdl:position()-1]' eq 0 then false else true}</dfdl:assert> 

Regards

Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848 

From: 
"Mike Beckerle" <mbeckerle.dfdl@gmail.com> 
To: 
"'Ryan Farrell'" <ryan.farrell.ctr@nrl.navy.mil>, <dfdl-wg@ogf.org> 
Date: 
22/07/2011 23:44 
Subject: 
Re: [DFDL-WG] [wg-all] Repeating groups in DFDL 
Sent by: 
dfdl-wg-bounces@ogf.org

Great example Ryan,

We've spent quite a bit of time over the last several months on textual
formats with these kinds of characteristics, which I would describe as
"occurrances determined by parsing" - that is, you look at the data to
determine "is there another element or not" one by one as they are parsed.

We need to make sure the properties are there to make this work for the
binary case as well. 

I actually believe we cannot handle this format right now. We don't have a
property which says to look at the current element to determine whether to
expect a subsequent element, as a way of determining occurrence counts. 

Next week we'll add resolving this example to the workgroup actions list. 

...mikeb

Mike Beckerle
Senior Technology Leader/Manager
Deloitte Managed Analytics
100 Fifth Avenue, Waltham, MA 02451
Tel/Direct: +1 781 330 0412 | Fax: +1 866 253 3006
www.deloitte.com

-----Original Message-----
From: dfdl-wg-bounces@ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf 
Of
Ryan Farrell
Sent: Tuesday, July 19, 2011 2:12 PM
To: dfdl-wg@ogf.org
Subject: [DFDL-WG] [wg-all] Repeating groups in DFDL

For anyone that was at the 19/07  phone meeting, this is what Adam Fox 
and I were trying to find a solution to. If anyone knows of a solution, 
please contact me or adam.fox@nrl.navy.mil.

----------------------------------------------------------------------------
------------------------

Please see the attached xml schema while reading this message.

The problem we want to solve is how to represent a structure that can 
have a variable amount of occurences (even zero), and the amount of 
repeats is controlled by a one bit field, which we will call"repeatBit". 
The repeating structure in this case is "A1". The element before it, 
"presentBit", determines if there is at least one occurence of A1. If 
presentBit is zero, then A1 has zero occurences. If presentBit is one, 
then A1 has AT LEAST one occurence.

The first element in A1 is "repeatBit". What repeatBit does is tells us 
if there will be another occurence of A1 after the current occurence. 
When repeatBit is zero, then that means we will read through the rest of 
A1, then that is the last occurence of A1. As long as repeatBit 
continues to be one however, we will read through the rest of A1, then 
start a new occurence.

I have included all the known required DFDL notation. Please let me know 
what is missing.

       EXAMPLE INPUT (results are in base 10):
       NOTE: Not all bits are used in these examples. Unused bits 
should be ignored for the purpose of the examples.

       1) 0111 0101
       presentBit = 0
       aFieldAfterA1 = 3

       2) 1011 1010
       presentBit = 1
       A1
       -repeatBit = 0
       -someField = 3
       aFieldAfterA1 = 2

       3) 1101 0110 0000 0000
       presentBit = 1
       A1
       -repeatBit = 1
       -someField = 1
       A1
       -repeatBit = 0
       -someField = 3
       aFieldAfterA1 = 0

--
 dfdl-wg mailing list
 dfdl-wg@ogf.org
 http://www.ogf.org/mailman/listinfo/dfdl-wg

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU 

Repeating groups in DFDL with continuation indicators - action 146 - updated

Steve Hanson

Mike Beckerle

Steve Hanson

Mike Beckerle

tags

participants (2)