Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU

24 Jul 2020

      Action 319 raised to add clarification to 9.3.3.1.

Regards

Steve Hanson
IBM Hybrid Integration, Hursley, UK
Architect, IBM DFDL
Co-Chair, OGF DFDL Working Group
smh@uk.ibm.com
tel:+44-1962-815848
mob:+44-7717-378890
Note: I work Tuesday to Friday 

From:   Mike Beckerle <mbeckerle.dfdl@gmail.com>
To:     Steve Hanson <smh@uk.ibm.com>
Cc:     DFDL-WG <dfdl-wg@ogf.org>
Date:   17/07/2020 13:35
Subject:        [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with 
direct dispatch is/is-not a PoU

Yes I think this clarification in 9.3.3.1 is definitely needed, as we 
learned daffodil doesn't actually implement this right in the case of two 
regular nested choices (where the inner one is NOT choice by dispatch). 

Basically, there's a stack of discriminator booleans, and a discriminator 
evaluating to true sets the state of the top of stack. That's wrong. It 
has to find the first non-true discriminator, searching down the stack, 
and set that one true instead. 

Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | 
www.owlcyberdefense.com
Please note: Contributions to the DFDL Workgroup's email discussions are 
subject to the OGF Intellectual Property Policy

On Thu, Jul 16, 2020 at 3:09 AM Steve Hanson <smh@uk.ibm.com> wrote:
That's good to hear. I didn't think Daffodil had a different behaviour, as 
I think it works with the IBM EDI schemas. 

It is probably worth updating 9.3.3.1 to be explicit on what happens if a 
second discriminator is encountered in a nested choice branch before a 
further PoU is encountered. 

Regards

Steve Hanson 
IBM Hybrid Integration, Hursley, UK
Architect, IBM DFDL
Co-Chair, OGF DFDL Working Group
smh@uk.ibm.com
tel:+44-1962-815848
mob:+44-7717-378890
Note: I work Tuesday to Friday 

From:        Mike Beckerle <mbeckerle.dfdl@gmail.com> 
To:        Steve Hanson <smh@uk.ibm.com> 
Cc:        DFDL-WG <dfdl-wg@ogf.org> 
Date:        15/07/2020 18:34 
Subject:        [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with 
direct dispatch is/is-not a PoU 

So, if the behavior is that after the choice-dispatch is complete, that a 
discriminator on a branch resolves the outer choice, that is the semantics 
I am also counting on working exactly that way. 

If the existing documentation is consistent with that, then no change is 
needed. 

Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | 
www.owlcyberdefense.com 
Please note: Contributions to the DFDL Workgroup's email discussions are 
subject to the OGF Intellectual Property Policy 

On Wed, Jul 15, 2020 at 5:08 AM Steve Hanson <smh@uk.ibm.com> wrote: 
No. A discriminator is only ignored if there is no PoU in scope.  
Otherwise it applies to the nearest in-scope PoU.  This is covered in 
9.3.3.1 which deals with nested PoUs. It talks about the behaviour of a 
processing error after a choice has been resolved. Given that an example 
of a processing error is a discriminator resolving to false, the behaviour 
of a discriminator evaluating to true is implied.
...
From the spec for direct dispatch choice ... "When a match is found, it is 
as if a dfdl:discriminator had evaluated to true on that branch. It is 
selected as resolution of the choice, and there is no backtracking to try 
other alternative selections if a processing error occurs."
So in your inner/outer scenario, if you encounter a further discriminator 
on the resolved branch then that discriminates the OUTER choice. 

The IBM schemas for EDI rely on this nested choice behaviour.  The inner 
choice has a branch per possible transaction type, with a discriminator to 
resolve each one. If a subsequent processing error occurs, it causes the 
first branch of the OUTER choice to fail, which instead drives the 'Bad 
Transaction' branch.  It would make no difference to the behaviour if the 
inner choice was resolved by direct dispatch or initiatedContent. 

  <xsd:complexType name="Transaction"> 
      <xsd:sequence> 
        <xsd:choice> 
          <xsd:sequence> 
            <xsd:choice> 
              <xsd:element ref="v5010:T997"> 
                <xsd:annotation> 
                  <xsd:appinfo source="http://www.ogf.org/dfdl/"> 
                    <dfdl:discriminator> 

{fn:contains(./ST/ST01_TransactionSetIdentifierCode,'997')} 
                    </dfdl:discriminator> 
                  </xsd:appinfo> 
                </xsd:annotation> 
              </xsd:element> 
              <xsd:element ref="v5010:T998"> 
                <xsd:annotation> 
                  <xsd:appinfo source="http://www.ogf.org/dfdl/"> 
                    <dfdl:discriminator> 

{fn:contains(./ST/ST01_TransactionSetIdentifierCode,'998')} 
                    </dfdl:discriminator> 
                  </xsd:appinfo> 
                </xsd:annotation> 
              </xsd:element> 
              <xsd:sequence> 
                <xsd:annotation> 
                         <xsd:appinfo source="http://www.ogf.org/dfdl/"> 
                               <dfdl:assert message="Unsupported message" 
test="{fn:false()}"/> 
                         </xsd:appinfo> 
                </xsd:annotation> 
              </xsd:sequence> 
            </xsd:choice> 
          </xsd:sequence> 
          <xsd:sequence> 
            <xsd:element ref="v5010:BadTransaction"> 
            </xsd:element> 
          </xsd:sequence> 
        </xsd:choice> 
      </xsd:sequence> 
    </xsd:complexType> 

Bottom line is that you need to be careful with your discriminator 
placement. Keep each discriminator as close as possible to the PoU it is 
resolving.  You can always look down into structures. 

Regards

Steve Hanson 
IBM Hybrid Integration, Hursley, UK
Architect, IBM DFDL
Co-Chair, OGF DFDL Working Group
smh@uk.ibm.com
tel:+44-1962-815848
mob:+44-7717-378890
Note: I work Tuesday to Friday 

From:        Mike Beckerle <mbeckerle.dfdl@gmail.com> 
To:        Steve Hanson <smh@uk.ibm.com> 
Cc:        DFDL-WG <dfdl-wg@ogf.org> 
Date:        14/07/2020 20:46 
Subject:        [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with 
direct dispatch is/is-not a PoU 

Ok, then to clarify, if I put a discriminator inside a branch of a choice 
with direct dispatch, that discriminator should simply confirm the direct 
dispatch selection of the choice dispatch key? I.e., it is ignored? 

So if I have two nested choices, the outer backtracks, the inner is choice 
by dispatch, then to discriminate the OUTER choice, I have to issue two 
discriminators in a row. The first is a noop because it applies to the 
inner choice. The second affects the outer choice? 

This would seem to be the implications of having the choice with direct 
dispatch be a PoU still. 

Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | 
www.owlcyberdefense.com 
Please note: Contributions to the DFDL Workgroup's email discussions are 
subject to the OGF Intellectual Property Policy 

On Tue, Jul 14, 2020 at 4:08 AM Steve Hanson <smh@uk.ibm.com> wrote: 
I don't agree, because unlike an array with a fixed number of occurrences, 
it is a processing error if the value of the expression does not match any 
of the dfdl:choiceBranchKey property values for any of the branches. Which 
currently causes backtracking because there is a PoU. 

I consider direct dispatch as more like the use of dfdl:initiatedContent 
when resolving a choice. 

This is not a behaviour that can be changed in DFDL 1.0, it would affect 
too many existing schemas. For example, IBM's DFDL schemas for SWIFT make 
heavy use of direct dispatch. 

Regards

Steve Hanson 
IBM Hybrid Integration, Hursley, UK
Architect, IBM DFDL
Co-Chair, OGF DFDL Working Group
smh@uk.ibm.com
tel:+44-1962-815848
mob:+44-7717-378890
Note: I work Tuesday to Friday 

From:        Mike Beckerle <mbeckerle.dfdl@gmail.com> 
To:        DFDL-WG <dfdl-wg@ogf.org> 
Date:        13/07/2020 14:50 
Subject:        [EXTERNAL] [DFDL-WG] clarification needed: choice with 
direct dispatch        is/is-not a PoU 
Sent by:        "dfdl-wg" <dfdl-wg-bounces@ogf.org> 

Just like an array with a computed number of occurrences, I believe a 
choice with direct dispatch should have no PoU. 

But the spec has this phrase "An xs:choice is always a point of 
uncertainty. It is resolved sequentially, or by direct dispatch." 

Which suggests there is a role for asserts/discriminators in resolving a 
choice by direct dispatch even though there shouldn't be. 

I think we should clarify this to "An xs:choice either is a point of 
uncertainty, or uses direct dispatch." 

Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | 
www.owlcyberdefense.com 
Please note: Contributions to the DFDL Workgroup's email discussions are 
subject to the OGF Intellectual Property Policy 
--
 dfdl-wg mailing list
 dfdl-wg@ogf.org
 https://www.ogf.org/mailman/listinfo/dfdl-wg 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU 