Re: [DFDL-WG] Fw: What are the consequences of a failed assert?

11 Jul 2013

      A schema definition error always stops the processor.  So the only thing 
up for debate is whether an assert failure due to a processing error could 
be treated as fatal.

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848

From:   "Garriss Jr., James P." <jgarriss@mitre.org>
To:     "dfdl-wg@ogf.org" <dfdl-wg@ogf.org>, 
Date:   11/07/2013 16:17
Subject:        Re: [DFDL-WG] Fw:  What are the consequences of a failed 
assert?
Sent by:        dfdl-wg-bounces@ogf.org

Helpful, thank you.  If #2 can specify the failureType, then would it be 
reasonable for #3 to do the same?  This would allow the schema designer to 
specify that syntax errors and processing errors are fatal errors that 
should stop processing.  If there’s a syntax error, I really don’t want 
the DFDL schema to run.  Everything’s wrong at that point.  (Catching this 
during static validation is good!)  Things like a divide by 0 error, well, 
I could see that one either way, depending upon what you’re doing.

From: Steve Hanson [mailto:smh@uk.ibm.com] 
Sent: Thursday, July 11, 2013 11:01 AM
To: Garriss Jr., James P.
Cc: dfdl-wg@ogf.org; dfdl-wg-bounces@ogf.org
Subject: Re: [DFDL-WG] Fw: What are the consequences of a failed assert?

A syntax error in the assert will cause a schema definition error. IBM 
DFDL catches these during static validation, but another implementation 
may choose to catch these when parsing. My bad - I should have included 
this in #3 below - I've updated it. 

An example of a #3 that will cause a processing error is a divide by zero 
in the expression. 

(Worth taking a look at errata 2.120 in the errata doc, this lists lots of 
failures and what DFDL error type they result in.) 

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848 

From:        "Garriss Jr., James P." <jgarriss@mitre.org> 
To:        "dfdl-wg@ogf.org" <dfdl-wg@ogf.org>, 
Date:        11/07/2013 15:36 
Subject:        Re: [DFDL-WG] Fw:  What are the consequences of a failed 
assert? 
Sent by:        dfdl-wg-bounces@ogf.org 

What can cause #3?  Syntax error in the assert?  Something else? 

From: Steve Hanson [mailto:smh@uk.ibm.com] 
Sent: Thursday, July 11, 2013 10:31 AM
To: Garriss Jr., James P.
Cc: dfdl-wg@ogf.org; dfdl-wg-bounces@ogf.org
Subject: Re: [DFDL-WG] Fw: What are the consequences of a failed assert? 

Let me give a bit more context. 

As assert can complete in exactly one of three ways: 
1) evaulate to true 
2) evaluate to false -> failureType says what kind of error to throw - 
recoverable, processing, or fatal (proposed) 
3) fail to evaluate -> schema definition error or processing error 

There's only ever one error. There's no concept of multiple errors. 

Let me phrase my question another way. I have an assert which has 
failureType 'fatalError'. If the assert evaluates to false then the parser 
will stop. But if the assert fails to evaluate the parser will throw a 
processing error and backtrack. The latter includes use of fn:error(), as 
that is classed as a failure to evaluate. 

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848 

From:        "Garriss Jr., James P." <jgarriss@mitre.org> 
To:        "dfdl-wg@ogf.org" <dfdl-wg@ogf.org>, 
Date:        11/07/2013 15:03 
Subject:        Re: [DFDL-WG] Fw:  What are the consequences of a failed 
assert? 
Sent by:        dfdl-wg-bounces@ogf.org 

Is there a reason why the processor can’t throw both errors?   

If one of the errors is fatal, it should stop.  IOW, respond the “worse” 
error first. 

From: dfdl-wg-bounces@ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf 
Of Steve Hanson
Sent: Thursday, July 11, 2013 5:07 AM
To: dfdl-wg@ogf.org
Subject: [DFDL-WG] Fw: What are the consequences of a failed assert? 

Let's say I have an assert with 'fatalError'. What happens if the assert 
has a processing error when it is being evaluated. Is this still a 
processing error, or does it get converted into a fatal error?  I include 
the use of fn:error() here as well. 

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848 
----- Forwarded by Steve Hanson/UK/IBM on 11/07/2013 09:59 ----- 

From:        "Garriss Jr., James P." <jgarriss@mitre.org> 
To:        Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB, 
Cc:        "Cranford, Jonathan W." <jcranford@mitre.org>, "dfdl-wg@ogf.org
" <dfdl-wg@ogf.org> 
Date:        10/07/2013 18:06 
Subject:        RE: [DFDL-WG] What are the consequences of a failed 
assert?
...
What is being asked for here sounds like the ability to throw an error 
and stop the parser regardless of what points of uncertainty are in scope
Exactly!
...
James, would this 'fatal error' give you the capability you are looking 
for?
Yes, I think it would.  I would like to encourage the WG to add this 
feature if at all possible. 

From: Steve Hanson [mailto:smh@uk.ibm.com] 
Sent: Monday, July 08, 2013 6:11 AM
To: Tim Kimber
Cc: Cranford, Jonathan W.; dfdl-wg@ogf.org; Garriss Jr., James P.
Subject: RE: [DFDL-WG] What are the consequences of a failed assert? 

Errata 3.4 added the failureType attribute to dfdl:assert so that an 
assert could report the equivalent of a validation error and carry on 
('recoverable error') instead of throwing a 'processing error'. What is 
being asked for here sounds like the ability to throw an error and stop 
the parser regardless of what points of uncertainty are in scope. That 
does sound like a genuinely useful thing to be able to do, and gives 
dfdl:assert the full range of failure actions. We'd have to create a new 
error type, say 'fatal error'. It would have the same effect as a 'schema 
definition error'. 

James, would this 'fatal error' give you the capability you are looking 
for? 

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848 

From:        Tim Kimber/UK/IBM 
To:        "Cranford, Jonathan W." <jcranford@mitre.org>, 
Cc:        "dfdl-wg@ogf.org" <dfdl-wg@ogf.org>, "dfdl-wg-bounces@ogf.org" 
<dfdl-wg-bounces@ogf.org>, "Garriss Jr., James P." <jgarriss@mitre.org>, 
Steve Hanson/UK/IBM@IBMGB 
Date:        08/07/2013 09:29 
Subject:        RE: [DFDL-WG] What are the consequences of a failed 
assert? 

It depends on whether there was exactly one point of uncertainty active 
when the discriminator expression evaluated to 'true'. Points of 
uncertainty can be nested within one another. A POU can be resolved ( or 
made 'inactive' as I was trying to hint at ) by a discriminator that 
evaluates to 'true'. When a processing error occurs, it is caught by the 
nearest ( most deeply nested ) *active* POU. If there is no active POU 
then the parser halts and reports the error. 

regards,

Tim Kimber, DFDL Team,
Hursley, UK
Internet:  kimbert@uk.ibm.com
Tel. 01962-816742 
Internal tel. 37246742

From:        "Cranford, Jonathan W." <jcranford@mitre.org> 
To:        "Garriss Jr., James P." <jgarriss@mitre.org>, Steve 
Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB, 
Cc:        "dfdl-wg@ogf.org" <dfdl-wg@ogf.org>, "dfdl-wg-bounces@ogf.org" 
<dfdl-wg-bounces@ogf.org> 
Date:        05/07/2013 20:31 
Subject:        RE: [DFDL-WG] What are the consequences of a failed 
assert? 

Wouldn’t the use of a discriminator suppress the backtracking behavior in 
the face of a processing error? 

That is, if a discriminator is used to confirm that the Content-Type 
header branch is the correct one, wouldn’t a subsequent assert that causes 
a processing error force the parser to halt with an error? 

-Jonathan Cranford 

From: dfdl-wg-bounces@ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf 
Of Garriss Jr., James P.
Sent: Friday, July 05, 2013 12:09 PM
To: Steve Hanson; Tim Kimber
Cc: dfdl-wg@ogf.org; dfdl-wg-bounces@ogf.org
Subject: Re: [DFDL-WG] What are the consequences of a failed assert? 

Very helpful, guys, as always.  Thank you. 

Question:  Is it possible to force the failure of an assert to be a 
validation error instead of a processing error? 

Here’s why:  All the email headers are modeled as an unordered list.  The 
last item on the list is a catch-all, designed to scoop up all the 
unexpected headers and hide them from the infoset (via hiddenGroupRef).   

So if I have an Content-Type header that is obviously invalid: 

Content-Type: bogus/data 

The assert fails as expected.  But since there is always another choice, 
the catch-all, the Content-Type header quietly disappears.  What I really 
want is for there to be a validation error for “bogus/data,” so that 
processing stops! 

Is that possible?  Is that reasonable? 

From: dfdl-wg-bounces@ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf 
Of Steve Hanson
Sent: Thursday, July 04, 2013 4:26 AM
To: Tim Kimber
Cc: dfdl-wg@ogf.org; dfdl-wg-bounces@ogf.org
Subject: Re: [DFDL-WG] What are the consequences of a failed assert? 

Almost.  A Recoverable Error does not cause backtracking. It was added to 
enable an assert to perform validation that checked the data stream rather 
than just the infoset. For example, checking the max physical length of a 
non-string value. So it has the same effect on the parse as a Validation 
Error. I've corrected your summary. 

The definition and description of the different errors is covered in the 
spec by section 2. There have been some errata in this section to clarify 
the behaviour. Some of the wording has been around since early drafts of 
the spec, so any suggestions for improvement are welcome. 

Regards

Steve Hanson
Architect, IBM Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848 

From:        Tim Kimber/UK/IBM@IBMGB 
To:        dfdl-wg@ogf.org, 
Date:        03/07/2013 21:51 
Subject:        Re: [DFDL-WG] What are the consequences of a failed 
assert? 
Sent by:        dfdl-wg-bounces@ogf.org 

There are four types of error in DFDL: 
- a Schema Definition Error : The schema itself is not valid. ( at least 
three kinds: xsd not valid, xsd not in the DFLD subset, DFDL annotations 
not following the rules ) 
- A Processing Error : The data cannot be parsed. Or if unparsing, the 
info set cannot be unparsed. 
- A Recoverable Error : ( see the Errata ). This is effectively a 
user-defined form of Validation Error, raised while executing a DFDL 
assert, 
- A Validation Error : The info set does not conform to the XSD 

A Schema Definition Error is immediately fatal. Most of these can be 
detected by the processor before parsing/unparsing begins. 
A Processing Error or a Recoverable Error will cause the parser to 
suppress the error and backtrack to the nearest point of uncertainty. So 
it will only stop the parse if there are no points of uncertainty 
currently active. 
A Recoverable Error does not cause backtracking - the parser continues to 
parse after reporting the error. 
A Validation Error is only reported if validation is enabled in the DFDL 
processor. It does not cause backtracking - the parser continues to parse 
after reporting the error. 

That's the gist of it. Further details from other WG members may follow 
shortly, depending on how accurate I have managed to be. 

regards,

Tim Kimber, DFDL Team,
Hursley, UK
Internet:  kimbert@uk.ibm.com
Tel. 01962-816742 
Internal tel. 37246742

From:        "Garriss Jr., James P." <jgarriss@mitre.org> 
To:        "dfdl-wg@ogf.org" <dfdl-wg@ogf.org>, 
Date:        03/07/2013 19:06 
Subject:        [DFDL-WG] What are the consequences of a failed assert? 
Sent by:        dfdl-wg-bounces@ogf.org 

I have an element with an assert, 

      <xsd:element name="Type" dfdl:inputValueCalc="{ 
fn:lower-case(../MixedCaseType) }">
          <xsd:annotation>
              <xsd:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/">
                  <dfdl:assert test="{ dfdl:checkConstraints(.) }" 
message="The type must match one of the values on the enumerated list."/>
              </xsd:appinfo>
          </xsd:annotation>
          <xsd:simpleType>
              <xsd:restriction base="xsd:string">
                  <xsd:enumeration value="application"/>
                  <xsd:enumeration value="multipart"/>
                  <xsd:enumeration value="message"/>
                  <xsd:enumeration value="text"/>
              </xsd:restriction>
          </xsd:simpleType>
      </xsd:element> 

and the assert is failing (as it should in this case!).   

Parse Error: Assertion failed. The type must match one of the values on 
the enumerated list. 

What are the consequences of a failed assert?  I have an old version of 
the spec—is there a place to a get a current, complete copy?—but it says 
“An unsuccessful dfdl:assert causes a processing error.”   

1.     What does “processing error” mean in English?   
2.     Does it mean the input is invalid?   
3.     Does it mean the processor should stop here and go no further?   
4.     Does it mean the process should simply ignore the problem and move 
on to the next item in the schema?   

TIA 
--
dfdl-wg mailing list
dfdl-wg@ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
--
dfdl-wg mailing list
dfdl-wg@ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
--
dfdl-wg mailing list
dfdl-wg@ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
--
 dfdl-wg mailing list
 dfdl-wg@ogf.org
 https://www.ogf.org/mailman/listinfo/dfdl-wg 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
--
  dfdl-wg mailing list
  dfdl-wg@ogf.org
  https://www.ogf.org/mailman/listinfo/dfdl-wg

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU