clarification needed: choice with direct dispatch is/is-not a PoU
Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU. But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch." Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be. I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>
I don't agree, because unlike an array with a fixed number of occurrences, it is a processing error if the value of the expression does not match any of the dfdl:choiceBranchKey property values for any of the branches. Which currently causes backtracking because there is a PoU. I consider direct dispatch as more like the use of dfdl:initiatedContent when resolving a choice. This is not a behaviour that can be changed in DFDL 1.0, it would affect too many existing schemas. For example, IBM's DFDL schemas for SWIFT make heavy use of direct dispatch. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: DFDL-WG <dfdl-wg@ogf.org> Date: 13/07/2020 14:50 Subject: [EXTERNAL] [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU. But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch." Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be. I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=EoHX9tRXl8caFGAzeLwtkERmCtDvOji0NO1_XwO6frM&s=A1-K5nQlKOzxx53UBJdXdSdMfC1vsb0XPOHHbVyE2I0&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Ok, then to clarify, if I put a discriminator inside a branch of a choice with direct dispatch, that discriminator should simply confirm the direct dispatch selection of the choice dispatch key? I.e., it is ignored? So if I have two nested choices, the outer backtracks, the inner is choice by dispatch, then to discriminate the OUTER choice, I have to issue two discriminators in a row. The first is a noop because it applies to the inner choice. The second affects the outer choice? This would seem to be the implications of having the choice with direct dispatch be a PoU still. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Jul 14, 2020 at 4:08 AM Steve Hanson <smh@uk.ibm.com> wrote:
I don't agree, because unlike an array with a fixed number of occurrences, it is a *processing error* if the value of the expression does not match any of the dfdl:choiceBranchKey property values for any of the branches. Which currently causes backtracking because there is a PoU.
I consider direct dispatch as more like the use of dfdl:initiatedContent when resolving a choice.
This is not a behaviour that can be changed in DFDL 1.0, it would affect too many existing schemas. For example, IBM's DFDL schemas for SWIFT make heavy use of direct dispatch.
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: DFDL-WG <dfdl-wg@ogf.org> Date: 13/07/2020 14:50 Subject: [EXTERNAL] [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> ------------------------------
Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU.
But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch."
Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be.
I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch."
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | *www.owlcyberdefense.com* <http://www.owlcyberdefense.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
No. A discriminator is only ignored if there is no PoU in scope. Otherwise it applies to the nearest in-scope PoU. This is covered in 9.3.3.1 which deals with nested PoUs. It talks about the behaviour of a processing error after a choice has been resolved. Given that an example of a processing error is a discriminator resolving to false, the behaviour of a discriminator evaluating to true is implied.
From the spec for direct dispatch choice ... "When a match is found, it is as if a dfdl:discriminator had evaluated to true on that branch. It is selected as resolution of the choice, and there is no backtracking to try other alternative selections if a processing error occurs."
So in your inner/outer scenario, if you encounter a further discriminator on the resolved branch then that discriminates the OUTER choice. The IBM schemas for EDI rely on this nested choice behaviour. The inner choice has a branch per possible transaction type, with a discriminator to resolve each one. If a subsequent processing error occurs, it causes the first branch of the OUTER choice to fail, which instead drives the 'Bad Transaction' branch. It would make no difference to the behaviour if the inner choice was resolved by direct dispatch or initiatedContent. <xsd:complexType name="Transaction"> <xsd:sequence> <xsd:choice> <xsd:sequence> <xsd:choice> <xsd:element ref="v5010:T997"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator> {fn:contains(./ST/ST01_TransactionSetIdentifierCode,'997')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element ref="v5010:T998"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator> {fn:contains(./ST/ST01_TransactionSetIdentifierCode,'998')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:sequence> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert message="Unsupported message" test="{fn:false()}"/> </xsd:appinfo> </xsd:annotation> </xsd:sequence> </xsd:choice> </xsd:sequence> <xsd:sequence> <xsd:element ref="v5010:BadTransaction"> </xsd:element> </xsd:sequence> </xsd:choice> </xsd:sequence> </xsd:complexType> Bottom line is that you need to be careful with your discriminator placement. Keep each discriminator as close as possible to the PoU it is resolving. You can always look down into structures. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org> Date: 14/07/2020 20:46 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Ok, then to clarify, if I put a discriminator inside a branch of a choice with direct dispatch, that discriminator should simply confirm the direct dispatch selection of the choice dispatch key? I.e., it is ignored? So if I have two nested choices, the outer backtracks, the inner is choice by dispatch, then to discriminate the OUTER choice, I have to issue two discriminators in a row. The first is a noop because it applies to the inner choice. The second affects the outer choice? This would seem to be the implications of having the choice with direct dispatch be a PoU still. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Jul 14, 2020 at 4:08 AM Steve Hanson <smh@uk.ibm.com> wrote: I don't agree, because unlike an array with a fixed number of occurrences, it is a processing error if the value of the expression does not match any of the dfdl:choiceBranchKey property values for any of the branches. Which currently causes backtracking because there is a PoU. I consider direct dispatch as more like the use of dfdl:initiatedContent when resolving a choice. This is not a behaviour that can be changed in DFDL 1.0, it would affect too many existing schemas. For example, IBM's DFDL schemas for SWIFT make heavy use of direct dispatch. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: DFDL-WG <dfdl-wg@ogf.org> Date: 13/07/2020 14:50 Subject: [EXTERNAL] [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU. But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch." Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be. I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
So, if the behavior is that after the choice-dispatch is complete, that a discriminator on a branch resolves the outer choice, that is the semantics I am also counting on working exactly that way. If the existing documentation is consistent with that, then no change is needed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Wed, Jul 15, 2020 at 5:08 AM Steve Hanson <smh@uk.ibm.com> wrote:
No. A discriminator is only ignored if there is no PoU in scope. Otherwise it applies to the nearest in-scope PoU. This is covered in 9.3.3.1 which deals with nested PoUs. It talks about the behaviour of a processing error after a choice has been resolved. Given that an example of a processing error is a discriminator resolving to false, the behaviour of a discriminator evaluating to true is implied.
From the spec for direct dispatch choice ... *"**When a match is found, it is as if a dfdl:discriminator had evaluated to true on that branch. It is selected as resolution of the choice, and there is no backtracking to try other alternative selections if a processing error occurs.**"*
So in your inner/outer scenario, if you encounter a further discriminator on the resolved branch then that discriminates the OUTER choice.
The IBM schemas for EDI rely on this nested choice behaviour. The inner choice has a branch per possible transaction type, with a discriminator to resolve each one. If a subsequent processing error occurs, it causes the first branch of the OUTER choice to fail, which instead drives the 'Bad Transaction' branch. It would make no difference to the behaviour if the inner choice was resolved by direct dispatch or initiatedContent.
<xsd:complexType name=*"Transaction"*> <xsd:sequence> <xsd:choice> <xsd:sequence> <xsd:choice> <xsd:element ref=*"v5010:T997"*> <xsd:annotation> <xsd:appinfo source=*"**http://www.ogf.org/dfdl/* <http://www.ogf.org/dfdl/>*"*> <dfdl:discriminator>
{fn:contains(./ST/ST01_TransactionSetIdentifierCode,'997')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element ref=*"v5010:T998"*> <xsd:annotation> <xsd:appinfo source=*"**http://www.ogf.org/dfdl/* <http://www.ogf.org/dfdl/>*"*> <dfdl:discriminator>
{fn:contains(./ST/ST01_TransactionSetIdentifierCode,'998')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:sequence> <xsd:annotation> <xsd:appinfo source=*"**http://www.ogf.org/dfdl/* <http://www.ogf.org/dfdl/>*"*> <dfdl:assert message=*"Unsupported message"* test=*"{fn:false()}"*/> </xsd:appinfo> </xsd:annotation> </xsd:sequence> </xsd:choice> </xsd:sequence> <xsd:sequence> <xsd:element ref=*"v5010:BadTransaction"*> </xsd:element> </xsd:sequence> </xsd:choice> </xsd:sequence> </xsd:complexType>
Bottom line is that you need to be careful with your discriminator placement. Keep each discriminator as close as possible to the PoU it is resolving. You can always look down into structures.
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org> Date: 14/07/2020 20:46 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU ------------------------------
Ok, then to clarify, if I put a discriminator inside a branch of a choice with direct dispatch, that discriminator should simply confirm the direct dispatch selection of the choice dispatch key? I.e., it is ignored?
So if I have two nested choices, the outer backtracks, the inner is choice by dispatch, then to discriminate the OUTER choice, I have to issue two discriminators in a row. The first is a noop because it applies to the inner choice. The second affects the outer choice?
This would seem to be the implications of having the choice with direct dispatch be a PoU still.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | *www.owlcyberdefense.com* <http://www.owlcyberdefense.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Tue, Jul 14, 2020 at 4:08 AM Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: I don't agree, because unlike an array with a fixed number of occurrences, it is a *processing error* if the value of the expression does not match any of the dfdl:choiceBranchKey property values for any of the branches. Which currently causes backtracking because there is a PoU.
I consider direct dispatch as more like the use of dfdl:initiatedContent when resolving a choice.
This is not a behaviour that can be changed in DFDL 1.0, it would affect too many existing schemas. For example, IBM's DFDL schemas for SWIFT make heavy use of direct dispatch.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: DFDL-WG <*dfdl-wg@ogf.org* <dfdl-wg@ogf.org>> Date: 13/07/2020 14:50 Subject: [EXTERNAL] [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU.
But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch."
Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be.
I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch."
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | *www.owlcyberdefense.com* <http://www.owlcyberdefense.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg* <https://www.ogf.org/mailman/listinfo/dfdl-wg>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
That's good to hear. I didn't think Daffodil had a different behaviour, as I think it works with the IBM EDI schemas. It is probably worth updating 9.3.3.1 to be explicit on what happens if a second discriminator is encountered in a nested choice branch before a further PoU is encountered. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org> Date: 15/07/2020 18:34 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU So, if the behavior is that after the choice-dispatch is complete, that a discriminator on a branch resolves the outer choice, that is the semantics I am also counting on working exactly that way. If the existing documentation is consistent with that, then no change is needed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Jul 15, 2020 at 5:08 AM Steve Hanson <smh@uk.ibm.com> wrote: No. A discriminator is only ignored if there is no PoU in scope. Otherwise it applies to the nearest in-scope PoU. This is covered in 9.3.3.1 which deals with nested PoUs. It talks about the behaviour of a processing error after a choice has been resolved. Given that an example of a processing error is a discriminator resolving to false, the behaviour of a discriminator evaluating to true is implied.
From the spec for direct dispatch choice ... "When a match is found, it is as if a dfdl:discriminator had evaluated to true on that branch. It is selected as resolution of the choice, and there is no backtracking to try other alternative selections if a processing error occurs."
So in your inner/outer scenario, if you encounter a further discriminator on the resolved branch then that discriminates the OUTER choice. The IBM schemas for EDI rely on this nested choice behaviour. The inner choice has a branch per possible transaction type, with a discriminator to resolve each one. If a subsequent processing error occurs, it causes the first branch of the OUTER choice to fail, which instead drives the 'Bad Transaction' branch. It would make no difference to the behaviour if the inner choice was resolved by direct dispatch or initiatedContent. <xsd:complexType name="Transaction"> <xsd:sequence> <xsd:choice> <xsd:sequence> <xsd:choice> <xsd:element ref="v5010:T997"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator> {fn:contains(./ST/ST01_TransactionSetIdentifierCode,'997')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element ref="v5010:T998"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator> {fn:contains(./ST/ST01_TransactionSetIdentifierCode,'998')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:sequence> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert message="Unsupported message" test="{fn:false()}"/> </xsd:appinfo> </xsd:annotation> </xsd:sequence> </xsd:choice> </xsd:sequence> <xsd:sequence> <xsd:element ref="v5010:BadTransaction"> </xsd:element> </xsd:sequence> </xsd:choice> </xsd:sequence> </xsd:complexType> Bottom line is that you need to be careful with your discriminator placement. Keep each discriminator as close as possible to the PoU it is resolving. You can always look down into structures. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org> Date: 14/07/2020 20:46 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Ok, then to clarify, if I put a discriminator inside a branch of a choice with direct dispatch, that discriminator should simply confirm the direct dispatch selection of the choice dispatch key? I.e., it is ignored? So if I have two nested choices, the outer backtracks, the inner is choice by dispatch, then to discriminate the OUTER choice, I have to issue two discriminators in a row. The first is a noop because it applies to the inner choice. The second affects the outer choice? This would seem to be the implications of having the choice with direct dispatch be a PoU still. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Jul 14, 2020 at 4:08 AM Steve Hanson <smh@uk.ibm.com> wrote: I don't agree, because unlike an array with a fixed number of occurrences, it is a processing error if the value of the expression does not match any of the dfdl:choiceBranchKey property values for any of the branches. Which currently causes backtracking because there is a PoU. I consider direct dispatch as more like the use of dfdl:initiatedContent when resolving a choice. This is not a behaviour that can be changed in DFDL 1.0, it would affect too many existing schemas. For example, IBM's DFDL schemas for SWIFT make heavy use of direct dispatch. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: DFDL-WG <dfdl-wg@ogf.org> Date: 13/07/2020 14:50 Subject: [EXTERNAL] [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU. But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch." Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be. I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Yes I think this clarification in 9.3.3.1 is definitely needed, as we learned daffodil doesn't actually implement this right in the case of two regular nested choices (where the inner one is NOT choice by dispatch). Basically, there's a stack of discriminator booleans, and a discriminator evaluating to true sets the state of the top of stack. That's wrong. It has to find the first non-true discriminator, searching down the stack, and set that one true instead. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Thu, Jul 16, 2020 at 3:09 AM Steve Hanson <smh@uk.ibm.com> wrote:
That's good to hear. I didn't think Daffodil had a different behaviour, as I think it works with the IBM EDI schemas.
It is probably worth updating 9.3.3.1 to be explicit on what happens if a second discriminator is encountered in a nested choice branch before a further PoU is encountered.
Regards
Steve Hanson
IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org> Date: 15/07/2020 18:34 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU ------------------------------
So, if the behavior is that after the choice-dispatch is complete, that a discriminator on a branch resolves the outer choice, that is the semantics I am also counting on working exactly that way.
If the existing documentation is consistent with that, then no change is needed.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | *www.owlcyberdefense.com* <http://www.owlcyberdefense.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Wed, Jul 15, 2020 at 5:08 AM Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: No. A discriminator is only ignored if there is no PoU in scope. Otherwise it applies to the nearest in-scope PoU. This is covered in 9.3.3.1 which deals with nested PoUs. It talks about the behaviour of a processing error after a choice has been resolved. Given that an example of a processing error is a discriminator resolving to false, the behaviour of a discriminator evaluating to true is implied.
From the spec for direct dispatch choice ... *"**When a match is found, it is as if a dfdl:discriminator had evaluated to true on that branch. It is selected as resolution of the choice, and there is no backtracking to try other alternative selections if a processing error occurs.**"*
So in your inner/outer scenario, if you encounter a further discriminator on the resolved branch then that discriminates the OUTER choice.
The IBM schemas for EDI rely on this nested choice behaviour. The inner choice has a branch per possible transaction type, with a discriminator to resolve each one. If a subsequent processing error occurs, it causes the first branch of the OUTER choice to fail, which instead drives the 'Bad Transaction' branch. It would make no difference to the behaviour if the inner choice was resolved by direct dispatch or initiatedContent.
<xsd:complexType name=*"Transaction"*> <xsd:sequence> <xsd:choice> <xsd:sequence> <xsd:choice> <xsd:element ref=*"v5010:T997"*> <xsd:annotation> <xsd:appinfo source=*"**http://www.ogf.org/dfdl/* <http://www.ogf.org/dfdl/>*"*> <dfdl:discriminator>
{fn:contains(./ST/ST01_TransactionSetIdentifierCode,'997')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element ref=*"v5010:T998"*> <xsd:annotation> <xsd:appinfo source=*"**http://www.ogf.org/dfdl/* <http://www.ogf.org/dfdl/>*"*> <dfdl:discriminator>
{fn:contains(./ST/ST01_TransactionSetIdentifierCode,'998')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:sequence> <xsd:annotation> <xsd:appinfo source=*"**http://www.ogf.org/dfdl/* <http://www.ogf.org/dfdl/>*"*> <dfdl:assert message=*"Unsupported message"* test=*"{fn:false()}"*/> </xsd:appinfo> </xsd:annotation> </xsd:sequence> </xsd:choice> </xsd:sequence> <xsd:sequence> <xsd:element ref=*"v5010:BadTransaction"*> </xsd:element> </xsd:sequence> </xsd:choice> </xsd:sequence> </xsd:complexType>
Bottom line is that you need to be careful with your discriminator placement. Keep each discriminator as close as possible to the PoU it is resolving. You can always look down into structures.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> Cc: DFDL-WG <*dfdl-wg@ogf.org* <dfdl-wg@ogf.org>> Date: 14/07/2020 20:46 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU ------------------------------
Ok, then to clarify, if I put a discriminator inside a branch of a choice with direct dispatch, that discriminator should simply confirm the direct dispatch selection of the choice dispatch key? I.e., it is ignored?
So if I have two nested choices, the outer backtracks, the inner is choice by dispatch, then to discriminate the OUTER choice, I have to issue two discriminators in a row. The first is a noop because it applies to the inner choice. The second affects the outer choice?
This would seem to be the implications of having the choice with direct dispatch be a PoU still.
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | *www.owlcyberdefense.com* <http://www.owlcyberdefense.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php>
On Tue, Jul 14, 2020 at 4:08 AM Steve Hanson <*smh@uk.ibm.com* <smh@uk.ibm.com>> wrote: I don't agree, because unlike an array with a fixed number of occurrences, it is a *processing error* if the value of the expression does not match any of the dfdl:choiceBranchKey property values for any of the branches. Which currently causes backtracking because there is a PoU.
I consider direct dispatch as more like the use of dfdl:initiatedContent when resolving a choice.
This is not a behaviour that can be changed in DFDL 1.0, it would affect too many existing schemas. For example, IBM's DFDL schemas for SWIFT make heavy use of direct dispatch.
Regards
Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday
From: Mike Beckerle <*mbeckerle.dfdl@gmail.com* <mbeckerle.dfdl@gmail.com>> To: DFDL-WG <*dfdl-wg@ogf.org* <dfdl-wg@ogf.org>> Date: 13/07/2020 14:50 Subject: [EXTERNAL] [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Sent by: "dfdl-wg" <*dfdl-wg-bounces@ogf.org* <dfdl-wg-bounces@ogf.org>> ------------------------------
Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU.
But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch."
Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be.
I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch."
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | *www.owlcyberdefense.com* <http://www.owlcyberdefense.com> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list *dfdl-wg@ogf.org* <dfdl-wg@ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg* <https://www.ogf.org/mailman/listinfo/dfdl-wg>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Action 319 raised to add clarification to 9.3.3.1. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org> Date: 17/07/2020 13:35 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Yes I think this clarification in 9.3.3.1 is definitely needed, as we learned daffodil doesn't actually implement this right in the case of two regular nested choices (where the inner one is NOT choice by dispatch). Basically, there's a stack of discriminator booleans, and a discriminator evaluating to true sets the state of the top of stack. That's wrong. It has to find the first non-true discriminator, searching down the stack, and set that one true instead. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Thu, Jul 16, 2020 at 3:09 AM Steve Hanson <smh@uk.ibm.com> wrote: That's good to hear. I didn't think Daffodil had a different behaviour, as I think it works with the IBM EDI schemas. It is probably worth updating 9.3.3.1 to be explicit on what happens if a second discriminator is encountered in a nested choice branch before a further PoU is encountered. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org> Date: 15/07/2020 18:34 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU So, if the behavior is that after the choice-dispatch is complete, that a discriminator on a branch resolves the outer choice, that is the semantics I am also counting on working exactly that way. If the existing documentation is consistent with that, then no change is needed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Jul 15, 2020 at 5:08 AM Steve Hanson <smh@uk.ibm.com> wrote: No. A discriminator is only ignored if there is no PoU in scope. Otherwise it applies to the nearest in-scope PoU. This is covered in 9.3.3.1 which deals with nested PoUs. It talks about the behaviour of a processing error after a choice has been resolved. Given that an example of a processing error is a discriminator resolving to false, the behaviour of a discriminator evaluating to true is implied.
From the spec for direct dispatch choice ... "When a match is found, it is as if a dfdl:discriminator had evaluated to true on that branch. It is selected as resolution of the choice, and there is no backtracking to try other alternative selections if a processing error occurs."
So in your inner/outer scenario, if you encounter a further discriminator on the resolved branch then that discriminates the OUTER choice. The IBM schemas for EDI rely on this nested choice behaviour. The inner choice has a branch per possible transaction type, with a discriminator to resolve each one. If a subsequent processing error occurs, it causes the first branch of the OUTER choice to fail, which instead drives the 'Bad Transaction' branch. It would make no difference to the behaviour if the inner choice was resolved by direct dispatch or initiatedContent. <xsd:complexType name="Transaction"> <xsd:sequence> <xsd:choice> <xsd:sequence> <xsd:choice> <xsd:element ref="v5010:T997"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator> {fn:contains(./ST/ST01_TransactionSetIdentifierCode,'997')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:element ref="v5010:T998"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator> {fn:contains(./ST/ST01_TransactionSetIdentifierCode,'998')} </dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> <xsd:sequence> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert message="Unsupported message" test="{fn:false()}"/> </xsd:appinfo> </xsd:annotation> </xsd:sequence> </xsd:choice> </xsd:sequence> <xsd:sequence> <xsd:element ref="v5010:BadTransaction"> </xsd:element> </xsd:sequence> </xsd:choice> </xsd:sequence> </xsd:complexType> Bottom line is that you need to be careful with your discriminator placement. Keep each discriminator as close as possible to the PoU it is resolving. You can always look down into structures. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson <smh@uk.ibm.com> Cc: DFDL-WG <dfdl-wg@ogf.org> Date: 14/07/2020 20:46 Subject: [EXTERNAL] Re: [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Ok, then to clarify, if I put a discriminator inside a branch of a choice with direct dispatch, that discriminator should simply confirm the direct dispatch selection of the choice dispatch key? I.e., it is ignored? So if I have two nested choices, the outer backtracks, the inner is choice by dispatch, then to discriminate the OUTER choice, I have to issue two discriminators in a row. The first is a noop because it applies to the inner choice. The second affects the outer choice? This would seem to be the implications of having the choice with direct dispatch be a PoU still. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Jul 14, 2020 at 4:08 AM Steve Hanson <smh@uk.ibm.com> wrote: I don't agree, because unlike an array with a fixed number of occurrences, it is a processing error if the value of the expression does not match any of the dfdl:choiceBranchKey property values for any of the branches. Which currently causes backtracking because there is a PoU. I consider direct dispatch as more like the use of dfdl:initiatedContent when resolving a choice. This is not a behaviour that can be changed in DFDL 1.0, it would affect too many existing schemas. For example, IBM's DFDL schemas for SWIFT make heavy use of direct dispatch. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: DFDL-WG <dfdl-wg@ogf.org> Date: 13/07/2020 14:50 Subject: [EXTERNAL] [DFDL-WG] clarification needed: choice with direct dispatch is/is-not a PoU Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> Just like an array with a computed number of occurrences, I believe a choice with direct dispatch should have no PoU. But the spec has this phrase "An xs:choice is always a point of uncertainty. It is resolved sequentially, or by direct dispatch." Which suggests there is a role for asserts/discriminators in resolving a choice by direct dispatch even though there shouldn't be. I think we should clarify this to "An xs:choice either is a point of uncertainty, or uses direct dispatch." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
participants (2)
-
Mike Beckerle -
Steve Hanson