July 2013 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

clarification needed: validation errors
by Mike Beckerle 14 Aug '13

14 Aug '13

Jessie Chab (tresys) said: ...... The wording of "Validation errors cannot be suppressed by points of uncertainty" was tripping me up. Though I understand what points of uncertainty are, it is unclear exactly what this is saying. When viewed as a continuation of the requirement "validation errors are ignored when resolving points of uncertainty" it seems to make a little more sense. Overall, the requirements together are saying (i believe), Validation errors are not taken into account when resolving points of uncertainty, but they are still going to occur and be present to the user upon a successful parse. I think this could be said a little more clearly in the spec.

2 1

Cancelled: OGF DFDL WG Call
by Steve Hanson 25 Jul '13

25 Jul '13

Cancelled as Steve is on holiday

1 0

Cancelled: OGF DFDL WG Call
by Steve Hanson 25 Jul '13

25 Jul '13

Cancelled as Steve is on holiday

1 0

proposal: add fn:trace function to DFDL
by Mike Beckerle 25 Jul '13

25 Jul '13

I have found working with DFDL expressions basically impossible without the fn:trace function to help debug them. I am not sure how I would write a tutorial about expressions without using fn:trace in it to illustrate what is going on in the expressions. In the interests of keeping the tutorials portable I suggest we add the fn:trace function. fn:trace($value as item()*, $label as xs:string) as item()* Summary: Provides an execution trace intended to be used in debugging queries. The notation item()* means any node sequence including an empty node sequence. The input $value is returned, unchanged, as the result of the function. In addition, the inputs $value, converted to an xs:string, and $label may be directed to a trace data set. The destination, format, and ordering of the output of this trace data set is implementation specific. Related Issue: I don't see any language in the spec where we flat out say that a node sequence in the expression language is a sequence of Infoset Element Information Items. We should add this near the begining of section 23. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 2

Using a discriminator on a simple type
by Steve Hanson 25 Jul '13

25 Jul '13

I know we discussed this on Tuesday's call, but I am really struggling with why I would ever want to do it. It seems to me it is an accident waiting to happen. A type is an abstract reusable thing, I have no idea where it will end up being used, so it is positively dangerous for a discriminator to appear. What is it discriminating? Answer: any surrounding point of uncertainty, no matter where that might be. Please can someone provide me with a real use case for this? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Minutes 2013-07-23/24
by Steve Hanson 25 Jul '13

25 Jul '13

Please find minutes from the above two calls at http://redmine.ogf.org/dmsf_files/13097?download=. Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Next call is in 3 weeks time due to holidays. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

proposal: errata: add rawbytes as optional feature
by Mike Beckerle 24 Jul '13

24 Jul '13

I believe rawbytes should be an optional feature. It is a quite complex feature to implement, and many implementations may not want to bother. Also since lengthKind='delimited' is optional, and that is the primary use case for rawBytes, I think we should be consistent and add rawbytes to the list of optional features. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

could you clarify statement made on the call today
by Cranford, Jonathan W. 24 Jul '13

24 Jul '13

Steve, Could you clarify a statement made during today's DFDL WG call? I didn't quite catch the whole statement, but it sounded like you were saying a design goal of the WG was that the result of parsing a binary format using DFDL would result in a DFDL infoset roughly equivalent to the XML infoset obtained by parsing the same data in an XML format. I don't think I quite captured that correctly, but it sounds like an important point, and I'd like to understand it further. For context, I've been asked to look at building an inspection and sanitization capability on top of DFDL, so I'm weighing the differences between DFDL Infoset and XML Infoset at the moment, and your comparison caught my attention. Thanks in advance, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

2 2

proposal: assert message property as DFDL Expression or String - Fwd: proposal: add fn:error function ....
by Mike Beckerle 24 Jul '13

24 Jul '13

Proposed: Make the message property on dfdl:assert (and dfdl:discriminator) be either a string or a DFDL Expression that returns a string. (Note: this is not a DFDL String Literal, it is just a string.) Example: <dfdl:assert message="{ fn:concat('unknown whatever ', ../foobar) }"> { if (...pred...) then ...expr1... else if (...pred2...) then ...expr2... else fn:false() }</dfdl:assert> If a processing error occurs while evaluating the test predicate, then that processing error occurs. The assert is not considered to have passed or failed because the assert predicate did not return ether true or false. If a processing error occurs while evaluating the message expression, then the assert causes an error of appropriate failureType: recoverableError or processingError, but an implementation-dependent message is provided. ---------- Forwarded message ---------- From: Steve Hanson <smh(a)uk.ibm.com> Date: Fri, Jul 12, 2013 at 4:25 AM Subject: Re: [DFDL-WG] proposal: add fn:error function and dfdl:hex function to DFDL To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org I agree that is simpler and making the message string an expression is an easy change to make. But in the your example if expr1 or expr2 evaluated to false, then in form #1 no message would be issued, but in form #2 it would be issued. Is that what you intended? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 11/07/2013 18:47 Subject: Re: [DFDL-WG] proposal: add fn:error function and dfdl:hex function to DFDL Sent by: dfdl-wg-bounces(a)ogf.org ------------------------------ Ignoring the dfdl:hexBinary thing. I want to suggest a different idea than fn:error function for DFDL: Suppose dfdl:assert (and dfdl:discriminator) message property could be an expression that evaluates to a string. Then instead of writing <dfdl:assert>{ if (...pred...) then ...expr1... else if (...pred2...) then ...expr2... else fn:error(sym, fn:concat("unknown whatever ", ../foobar)) }</dfdl:assert> You could write: <dfdl:assert message="{ fn:concat('unknown whatever ', ../foobar) }"> { if (...pred...) then ...expr1... else if (...pred2...) then ...expr2... else fn:false() }</dfdl:assert> This is almost equivalent. But the message would respect the failureType of the assert. This has advantages. On Wed, Jul 10, 2013 at 3:14 PM, Garriss Jr., James P. <*jgarriss(a)mitre.org*<jgarriss(a)mitre.org>> wrote: > You need a different mechanism, which is a fatalError type for the dfdl:assert annotation. Sounds like we are in agreement. Thanks for clarifying. > But the CDS community uses validation to indicate an immediate need to stop. I.e., invalid == "dangerous". In some cases this is true, though in other cases it is not true. That’s why it seems to me that it would be good if DFDL can handle either mindset. For the work I’ve been doing with DFDL and email, the “invalid=dangerous” mindset was a starting assumption. *From:* Mike Beckerle [mailto:*mbeckerle.dfdl@gmail.com*<mbeckerle.dfdl(a)gmail.com>] * Sent:* Wednesday, July 10, 2013 1:57 PM* To:* Garriss Jr., James P.* Cc:* *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>* Subject:* Re: [DFDL-WG] proposal: add fn:error function and dfdl:hex function to DFDL Actually, fn:error was added in order to allow a DFDL Expression to create a normal processing error. You need a different mechanism, which is a fatalError type for the dfdl:assert annotation. I have noticed that the use of 'validation' in DFDL is the opposite of the way other people use 'validation'. In the DFDL spec, validation generally just creates warnings and continues parsing. It can't even affect the parse unless you call checkConstrants(....) from an assert. But the CDS community uses validation to indicate an immediate need to stop. I.e., invalid == "dangerous". ...mikeb On Wed, Jul 10, 2013 at 1:37 PM, Garriss Jr., James P. <*jgarriss(a)mitre.org*<jgarriss(a)mitre.org>> wrote: > fn:error, if called from inside an assert or discriminator, causes the assert or discriminator to fail (as with any processing error that occurs during evaluation of the assert or discriminator expression). Is this intended to address the issue I raised? If so, I think this is not sufficient. I need to have the option to cause a validation error, not a processing error. If not, please ignore this interruption. :-) *From:* *dfdl-wg-bounces(a)ogf.org* <dfdl-wg-bounces(a)ogf.org> [mailto:* dfdl-wg-bounces(a)ogf.org* <dfdl-wg-bounces(a)ogf.org>] *On Behalf Of *Mike Beckerle* Sent:* Tuesday, July 09, 2013 11:29 AM* To:* *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>* Subject:* Re: [DFDL-WG] proposal: add fn:error function and dfdl:hex function to DFDL Proposal revised for dfdl:hex, and clarification added on fn:error when used inside assert. Revised language below: Proposed: add fn:error function and add dfdl:hexBinary function. DFDL only needs one of the several variations of fn:error. fn:error($error as xs:QName?, $description as xs:string) as none The behavior of fn:error in DFDL expressions is the same as described in XPath 2.0, an example is given below. dfdl:hexBinary($arg as xs:anyAtomicType?) as xs:hexBinary? This function provides an extension to the behavior of the xs:hexBinary constructor. The argument can be a string, in which case the behavior is as the xs:hexBinary constructor. The argument can also be a long, unsignedLong, or any subtype thereof, and in that case a string containing hexadecimal digits is produced. The hexadecimal digits produced that are letters are always uppercase. The number of hex digits in the resulting string is a function of the input type. If byte or unsigned byte, exactly 2 hex digits are produced, for short and unsignedShort, 4 hex digits, and so on. The hex digits correspond to a big-endian representation of a twos-complement binary representation of the argument value. So: dfdl:short(xs:concat('x', dfdl:hexBinary(xs:short(208)))) eq xs:short(208) is true, and a corresponding tautology holds for all the other DFDL functions that construct integers from hexadecimal, if you replace the 208 above with a value in range for the corresponding numeric type. If the argument is a numeric literal, then the smallest signed integer type (long, int, short, byte) is selected that can contain the value, and the number of hexadecimal digits produced corresponds to that type. So: dfdl:hexBinary(208) eq 'D0' dfdl:hexBinary(-2084) eq 'F7FF' Example of fn:error and dfdl:hexBinary: This expression allows a meaningful error message that includes the offending parts of data. fn:error(ex:magic_number, fn:concat("The magic number: 0x", dfdl:hexBinary(.), " was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).")) The $error argument (of type QName) is used in an implementation-dependent manner to provide the context of the error. So for example, if namespace prefix 'ex' is bound to '*http://example.com* <http://example.com/>' then the above call might produce: "Error: {*http://example.com* <http://example.com/>}magic_number - The magic number: 0xFEFF0000 was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian)." Only the part of the above string after the hyphen is standardized. The hyphen and part before the hyphen is implementation dependent. fn:error, if called from inside an assert or discriminator, causes the assert or discriminator to fail (as with any processing error that occurs during evaluation of the assert or discriminator expression). If the assert or discriminator has a message property, then it is implementation dependent how that message string is combined with the description string argument provided to the fn:error function. On Thu, Jun 20, 2013 at 1:09 PM, Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com>> wrote: Proposed: add fn:error function and add dfdl:hex function. DFDL only needs one of the several variations of fn:error fn:error($error as xs:QName?, $description as xs:string) as none dfdl:hex($num as xs:integer) as string The dfdl:hex function inverts the behavior of the dfdl:byte, and related functions (currently also proposed) which take a string specification of a hex value. The argument to dfdl:hex can be a long, unsignedLong, or any subtype thereof. The number of hex digits in the resulting string is a function of the input type. If byte or unsigned byte, exactly 2 hex digits are produced, for short and unsignedShort, 4 hex digits, and so on. The resulting value contains a prefix 'x' character as its first character. So: dfdl:short(dfdl:hex(xs:short(208))) eq xs:short(208) is true, and a corresponding tautology holds for all the other DFDL functions that construct integers from hexadecimal, if you replace the 208 above with a value in range for the corresponding numeric type. Example: This allows a meaningful error message that includes the offending parts of data. fn:error(ex:magic_number, fn:concat("The magic number: 0", dfdl:hex(.), " was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).")) The $error argument (of type QName) is used in an implementation-dependent manner to provide the context of the error. So for example, if namespace prefix 'ex' is bound to '*http://example.com* <http://example.com/>' then the above call might produce: "Error: {*http://example.com* <http://example.com/>}magic_number - The magic number: 0xFEFF0000 was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian)." Only the part of the above string after the hyphen is standardized. The hyphen and part before the hyphen is implementation dependent. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * www.tresys.com* <http://www.tresys.com/> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * www.tresys.com* <http://www.tresys.com/> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * www.tresys.com* <http://www.tresys.com/> -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 2