- dfdl-wg - lists.ogf.org

could you clarify statement made on the call today
by Cranford, Jonathan W. 24 Jul '13

24 Jul '13

Steve, Could you clarify a statement made during today's DFDL WG call? I didn't quite catch the whole statement, but it sounded like you were saying a design goal of the WG was that the result of parsing a binary format using DFDL would result in a DFDL infoset roughly equivalent to the XML infoset obtained by parsing the same data in an XML format. I don't think I quite captured that correctly, but it sounds like an important point, and I'd like to understand it further. For context, I've been asked to look at building an inspection and sanitization capability on top of DFDL, so I'm weighing the differences between DFDL Infoset and XML Infoset at the moment, and your comparison caught my attention. Thanks in advance, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

2 2

proposal: assert message property as DFDL Expression or String - Fwd: proposal: add fn:error function ....
by Mike Beckerle 24 Jul '13

24 Jul '13

Proposed: Make the message property on dfdl:assert (and dfdl:discriminator) be either a string or a DFDL Expression that returns a string. (Note: this is not a DFDL String Literal, it is just a string.) Example: <dfdl:assert message="{ fn:concat('unknown whatever ', ../foobar) }"> { if (...pred...) then ...expr1... else if (...pred2...) then ...expr2... else fn:false() }</dfdl:assert> If a processing error occurs while evaluating the test predicate, then that processing error occurs. The assert is not considered to have passed or failed because the assert predicate did not return ether true or false. If a processing error occurs while evaluating the message expression, then the assert causes an error of appropriate failureType: recoverableError or processingError, but an implementation-dependent message is provided. ---------- Forwarded message ---------- From: Steve Hanson <smh(a)uk.ibm.com> Date: Fri, Jul 12, 2013 at 4:25 AM Subject: Re: [DFDL-WG] proposal: add fn:error function and dfdl:hex function to DFDL To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org I agree that is simpler and making the message string an expression is an easy change to make. But in the your example if expr1 or expr2 evaluated to false, then in form #1 no message would be issued, but in form #2 it would be issued. Is that what you intended? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 11/07/2013 18:47 Subject: Re: [DFDL-WG] proposal: add fn:error function and dfdl:hex function to DFDL Sent by: dfdl-wg-bounces(a)ogf.org ------------------------------ Ignoring the dfdl:hexBinary thing. I want to suggest a different idea than fn:error function for DFDL: Suppose dfdl:assert (and dfdl:discriminator) message property could be an expression that evaluates to a string. Then instead of writing <dfdl:assert>{ if (...pred...) then ...expr1... else if (...pred2...) then ...expr2... else fn:error(sym, fn:concat("unknown whatever ", ../foobar)) }</dfdl:assert> You could write: <dfdl:assert message="{ fn:concat('unknown whatever ', ../foobar) }"> { if (...pred...) then ...expr1... else if (...pred2...) then ...expr2... else fn:false() }</dfdl:assert> This is almost equivalent. But the message would respect the failureType of the assert. This has advantages. On Wed, Jul 10, 2013 at 3:14 PM, Garriss Jr., James P. <*jgarriss(a)mitre.org*<jgarriss(a)mitre.org>> wrote: > You need a different mechanism, which is a fatalError type for the dfdl:assert annotation. Sounds like we are in agreement. Thanks for clarifying. > But the CDS community uses validation to indicate an immediate need to stop. I.e., invalid == "dangerous". In some cases this is true, though in other cases it is not true. That’s why it seems to me that it would be good if DFDL can handle either mindset. For the work I’ve been doing with DFDL and email, the “invalid=dangerous” mindset was a starting assumption. *From:* Mike Beckerle [mailto:*mbeckerle.dfdl@gmail.com*<mbeckerle.dfdl(a)gmail.com>] * Sent:* Wednesday, July 10, 2013 1:57 PM* To:* Garriss Jr., James P.* Cc:* *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>* Subject:* Re: [DFDL-WG] proposal: add fn:error function and dfdl:hex function to DFDL Actually, fn:error was added in order to allow a DFDL Expression to create a normal processing error. You need a different mechanism, which is a fatalError type for the dfdl:assert annotation. I have noticed that the use of 'validation' in DFDL is the opposite of the way other people use 'validation'. In the DFDL spec, validation generally just creates warnings and continues parsing. It can't even affect the parse unless you call checkConstrants(....) from an assert. But the CDS community uses validation to indicate an immediate need to stop. I.e., invalid == "dangerous". ...mikeb On Wed, Jul 10, 2013 at 1:37 PM, Garriss Jr., James P. <*jgarriss(a)mitre.org*<jgarriss(a)mitre.org>> wrote: > fn:error, if called from inside an assert or discriminator, causes the assert or discriminator to fail (as with any processing error that occurs during evaluation of the assert or discriminator expression). Is this intended to address the issue I raised? If so, I think this is not sufficient. I need to have the option to cause a validation error, not a processing error. If not, please ignore this interruption. :-) *From:* *dfdl-wg-bounces(a)ogf.org* <dfdl-wg-bounces(a)ogf.org> [mailto:* dfdl-wg-bounces(a)ogf.org* <dfdl-wg-bounces(a)ogf.org>] *On Behalf Of *Mike Beckerle* Sent:* Tuesday, July 09, 2013 11:29 AM* To:* *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>* Subject:* Re: [DFDL-WG] proposal: add fn:error function and dfdl:hex function to DFDL Proposal revised for dfdl:hex, and clarification added on fn:error when used inside assert. Revised language below: Proposed: add fn:error function and add dfdl:hexBinary function. DFDL only needs one of the several variations of fn:error. fn:error($error as xs:QName?, $description as xs:string) as none The behavior of fn:error in DFDL expressions is the same as described in XPath 2.0, an example is given below. dfdl:hexBinary($arg as xs:anyAtomicType?) as xs:hexBinary? This function provides an extension to the behavior of the xs:hexBinary constructor. The argument can be a string, in which case the behavior is as the xs:hexBinary constructor. The argument can also be a long, unsignedLong, or any subtype thereof, and in that case a string containing hexadecimal digits is produced. The hexadecimal digits produced that are letters are always uppercase. The number of hex digits in the resulting string is a function of the input type. If byte or unsigned byte, exactly 2 hex digits are produced, for short and unsignedShort, 4 hex digits, and so on. The hex digits correspond to a big-endian representation of a twos-complement binary representation of the argument value. So: dfdl:short(xs:concat('x', dfdl:hexBinary(xs:short(208)))) eq xs:short(208) is true, and a corresponding tautology holds for all the other DFDL functions that construct integers from hexadecimal, if you replace the 208 above with a value in range for the corresponding numeric type. If the argument is a numeric literal, then the smallest signed integer type (long, int, short, byte) is selected that can contain the value, and the number of hexadecimal digits produced corresponds to that type. So: dfdl:hexBinary(208) eq 'D0' dfdl:hexBinary(-2084) eq 'F7FF' Example of fn:error and dfdl:hexBinary: This expression allows a meaningful error message that includes the offending parts of data. fn:error(ex:magic_number, fn:concat("The magic number: 0x", dfdl:hexBinary(.), " was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).")) The $error argument (of type QName) is used in an implementation-dependent manner to provide the context of the error. So for example, if namespace prefix 'ex' is bound to '*http://example.com* <http://example.com/>' then the above call might produce: "Error: {*http://example.com* <http://example.com/>}magic_number - The magic number: 0xFEFF0000 was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian)." Only the part of the above string after the hyphen is standardized. The hyphen and part before the hyphen is implementation dependent. fn:error, if called from inside an assert or discriminator, causes the assert or discriminator to fail (as with any processing error that occurs during evaluation of the assert or discriminator expression). If the assert or discriminator has a message property, then it is implementation dependent how that message string is combined with the description string argument provided to the fn:error function. On Thu, Jun 20, 2013 at 1:09 PM, Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com>> wrote: Proposed: add fn:error function and add dfdl:hex function. DFDL only needs one of the several variations of fn:error fn:error($error as xs:QName?, $description as xs:string) as none dfdl:hex($num as xs:integer) as string The dfdl:hex function inverts the behavior of the dfdl:byte, and related functions (currently also proposed) which take a string specification of a hex value. The argument to dfdl:hex can be a long, unsignedLong, or any subtype thereof. The number of hex digits in the resulting string is a function of the input type. If byte or unsigned byte, exactly 2 hex digits are produced, for short and unsignedShort, 4 hex digits, and so on. The resulting value contains a prefix 'x' character as its first character. So: dfdl:short(dfdl:hex(xs:short(208))) eq xs:short(208) is true, and a corresponding tautology holds for all the other DFDL functions that construct integers from hexadecimal, if you replace the 208 above with a value in range for the corresponding numeric type. Example: This allows a meaningful error message that includes the offending parts of data. fn:error(ex:magic_number, fn:concat("The magic number: 0", dfdl:hex(.), " was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).")) The $error argument (of type QName) is used in an implementation-dependent manner to provide the context of the error. So for example, if namespace prefix 'ex' is bound to '*http://example.com* <http://example.com/>' then the above call might produce: "Error: {*http://example.com* <http://example.com/>}magic_number - The magic number: 0xFEFF0000 was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian)." Only the part of the above string after the hyphen is standardized. The hyphen and part before the hyphen is implementation dependent. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * www.tresys.com* <http://www.tresys.com/> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * www.tresys.com* <http://www.tresys.com/> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * www.tresys.com* <http://www.tresys.com/> -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 2

proposed wording: scanable and 'results are not predictable' improvement - was Fwd: issue: scannable and 'results are not predictable'
by Mike Beckerle 24 Jul '13

24 Jul '13

The upshot of this whole thread is that we need to fix the description of testPattern for asserts/discriminators. That motivates another correction in the prose description of encodingErrorPolicy. Proposed rewording: This paragraph in the testPattern description for asserts/discriminators: - In order for a testPattern to be used, the data subject to the pattern must be scannable using a DFDL regular expression otherwise the results are not predictable. Change to: - In order for a testPattern to be used, the data subject to the pattern must be scannable using a DFDL regular expression. If the pattern regular expression reads data that cannot be decoded into characters of the current encoding, then the behavior is controlled by the dfdl:encodingErrorPolicy property. See Section 11.2.1 Property dfdl:encodingErrorPolicy for details. In addition, consider the paragraph in section 11.2.1.2 - The Unicode Replacement Character must not appear in any delimiter, pad character, nil value, regular expression, number pattern or calendar pattern, or in any other DFDL property value where the Unicode Replacement Character would be expected in the data being parsed. It is a schema definition error if the Unicode Replacement Character appears in any of these locations of a DFDL schema, or is part of the value of an expression that returns a string to be used as the value of a DFDL property. I believe the above paragraph is a mistake. It precludes a very useful technique which is to use a negated character class in the regex like [^\uFFFD] This regex searches for any character except the unicode replacement character which is very useful. I suggest the above paragraph be dropped. This sentence (same section) can be modified: - Schema authors are advised that bounded length regular expressions can help in this case. E.g., ".{0,50}" says to match any character (including Unicode Replacement Characters), but only up to length 50. Change to: - Schema authors are advised that bounded length regular expressions and negated character classes can improve the schema. E.g., "[^\uFFFD].{0,50}" says to match any character (*excluding *specifically Unicode Replacement Characters U+FFFD), but only up to length 50. ---------- Forwarded message ---------- From: Steve Hanson <smh(a)uk.ibm.com> Date: Tue, Jul 23, 2013 at 9:02 AM Subject: Re: [DFDL-WG] issue: scannable and 'results are not predictable' To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Tim Kimber <KIMBERT(a)uk.ibm.com > Mike, would you like to attempt some words to improve the 'results not predictable' sentence? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: Tim Kimber/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 11/07/2013 18:18 Subject: Re: [DFDL-WG] issue: scannable and 'results are not predictable' ------------------------------ Ok. let me summarize. I think this is clear now: lengthKind pattern requires everything to be statically known to be text. Fake binary data as iso-8859-1 text since that passes all bytes. assert pattern just tries to decode in current encoding. Binary data might cause decode errors or might not depending on what is in the actual data. Use encoding iso-8859-1 to preclude this possibility, or user-beware/know-the-data. The behavior of decode errors is controlled by encodingErrorPolicy, which clearly states that if the policy is 'error' then a processing error is issued. It specifically states that this applies in all situations including lengthKind pattern, and pattern asserts. The description does not leave any wiggle room here. (There's a separate email thread on asserts with expressions that get errors but it does not yet discuss pattern asserts.) So the wording that says "results are not predictable" should instead explain and provide a reference to the description of encodingErrorPolicy. On Thu, Jul 11, 2013 at 5:57 AM, Steve Hanson <*smh(a)uk.ibm.com*<smh(a)uk.ibm.com>> wrote: In the original DFDL 1.0 spec this is what we used to say about lengthKind 'pattern'. *12.3.5.1 Pattern-Based Lengths - Scanability* *Any element (complex, simple text, simple binary) may have a dfdl:lengthKind 'pattern' as long as the bytes in the content region of the element **are legal in the stated encoding of that element. Where a complex element has children with binary representation in practice this means an 8-bit ASCII encoding. Binary data can be handled by way of treating it as text with encoding='iso-8859-1'. In this case the text is interpreted as in the iso-8859-1 character encoding, and the correspondence of byte values in the data to a string in the DFDL infoset is one to one. That is, byte with value N, produces an infoset character with character code N.* This was changed by errata 3.9 back in the 4th revision of the Errata document. At the time, the same limit was applied to asserts & discriminators as well. Here is the original errata wording. * 3.9.** Section 12.3.5.1. The spec currently allows lengthKind ‘pattern’ to be used when the representation of the current element, or of a child element, is binary, but imposes restrictions on the encoding that can be in force. However encoding is not necessarily examined for binary elements, so this would introduce another reason for needing encoding.* * Change the spec so that lengthKind ‘pattern’ is only applicable ** o **elements of simple type with representation 'text'* * o **elements of complex type * * For an element of complex type:* * 1. all simple child elements must have representation 'text' and have the same encoding as the parent complex element, and* * 2. all complex child elements must themselves follow 1 and 2 (recursively). * * Similar wording to apply to dfdl:assert testKind="pattern" in section 7.3.1. * In the 11th revision of the Errata document, the last sentence was changed to... * Note that the same restrictions do not** apply to testKind="pattern" on asserts and discriminators* This was done because an assert or discriminator with testKind 'pattern' is peeking ahead into the data stream from the start of the representation of the object (element / sequence / choice). This was recorded by action 190. *190* *Clarify rules for assert/discriminator testKind 'pattern' (All)* * 23/10: Need to be clear on data position and whether it is just for text representations. * * 30/10: Closed.** To comply with the timing rules being proposed in action 186, where these things are executed first before a 'format' annotation, the data position must be the beginning of the representation (note warning useful when alignment present). As these things can be used on various objects, the only rule regarding text is that dfdl:encoding must have a value in scope. Errata taken**.* Personally I am happy for DFDL 1.0 to stick with the current errata, and improve the wording in the testPattern description. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:*+44-1962-815848* <%2B44-1962-815848> From: Tim Kimber/UK/IBM@IBMGB To: *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>, Date: 11/07/2013 10:08 Subject: Re: [DFDL-WG] issue: scannable and 'results are not predictable' Sent by: *dfdl-wg-bounces(a)ogf.org* <dfdl-wg-bounces(a)ogf.org> ------------------------------ There was a time when we disallowed lengthKind='delimited' when representation is 'binary'. Binary data can, in general, contain any sequence of bytes so it might contain the terminating markup. In other words it is not guaranteed to be 'scannable'. We relaxed that rule because we found that there are industry formats out there which contain non-text delimited fields. In other words, the general rule ( binary data is not scannable ) does not always apply in specific formats. I think that point is relevant to this discussion. Just because the DFDL properties indicate that the data is not *guaranteed* to be scannable, that does not mean that the actual data is not scannable. I believe we should - define the term 'scannable' - acknowledge that when a complex type is not 'scannable' according to the definition, the data still might be parse-able in a reliable way - not prohibit the use of lengthKind='pattern' ( i.e. not issue an SDE ) just because the element is not 'scannable'. It may well be appropriate for an implementation to issue a warning when lengthKind is 'delimited' or 'pattern' and the element's content is not 'scannable'. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: *kimbert(a)uk.ibm.com* <kimbert(a)uk.ibm.com> Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com> > To: *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>, Date: 10/07/2013 18:22 Subject: [DFDL-WG] issue: scannable and 'results are not predictable' Sent by: *dfdl-wg-bounces(a)ogf.org* <dfdl-wg-bounces(a)ogf.org> ------------------------------ I was editing the definition of scannable into the glossary and when I looked at usage of 'scannable' in testPattern I found this: In the box for testPattern it says if the data is not scannable "the results are not predictable". Is that sufficient? We can sometimes statically determine that the schema says the data should all be scannable (e.g., no change of encoding, no binary elements), and that would rule out one non-predictability. So, if data is non-scannable in the sense that the schema contains say, a binary element, we can issue an SDE if lengthKind is pattern or a testPattern assert is being used. We could also SDE if runtime-valued encoding properties are used and the encoding changes inside a scannable context. Well, I guess testKind pattern asserts/discriminators are an issue because they may look only at the first part of the data of a complex component, so they don't require everything to be scannable, only the part the regex actually examines. So in this case it's user-beware, and if non-scannable I suppose we could issue a warning. But the spec does not say this is an SDE or warning currently. It just says results are not predictable. There is also the fact that the data might be broken, i.e., the schema might say the data is scannable, but at parse time character decode errors occur. I believe our policy on this is that these cause processing errors. This really is orthogonal to scannable, which is a property of a schema component. Comments? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * www.tresys.com* <http://www.tresys.com/> -- dfdl-wg mailing list* **dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>* **https://www.ogf.org/mailman/listinfo/dfdl-wg*<https://www.ogf.org/mailman/listinfo/dfdl-wg> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU-- dfdl-wg mailing list *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg*<https://www.ogf.org/mailman/listinfo/dfdl-wg> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> *https://www.ogf.org/mailman/listinfo/dfdl-wg*<https://www.ogf.org/mailman/listinfo/dfdl-wg> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * www.tresys.com* <http://www.tresys.com/> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 1

Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language
by Tim Kimber 24 Jul '13

24 Jul '13

Interesting...that never occurred to me. I have always considered the # to be a qualifier on the XML character reference entity. In XML, the carriage-return character can be specified in any of the following ways: &cr; - A named entity  - A numeric entity ( starts with '#' )  - A numeric entity in base 16 ( the numeric part starts with 'x' ) So the '#' character is signalling that the entity is a numeric entity. But there's no need for that to be signalled in this case - the parameter for these integer constructors will always be a number. On that basis, the prefix should be 'x'. I would prefer that to '#x' on the grounds of readability. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 20/06/2013 14:01 Subject: Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language I was thinking along the same lines as Tim. I'd prefer the hex string to start with '#x' for consistency with DFDL hex entity syntax. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 20/06/2013 09:44 Subject: Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language Sent by: dfdl-wg-bounces(a)ogf.org As an alternative, we could - drop the 'h' prefix - use the same names as the XPath integer constructors, including the use of the full name for unsigned variants ( e.g. 'dfdl:unsignedInt' ) - behave exactly like xs:byte / xs:short etc when the string does not begin with '0x' The only difference would be the namespace of the function. So the modeller would use the dfdl variant if they needed the '0x' notation, and they could choose to use the dfdl variant always when constructing integers from string literals. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 19/06/2013 23:28 Subject: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language Sent by: dfdl-wg-bounces(a)ogf.org Rationale: in writing DFDL schemas for binary data formats, there is a strong need to express binary data constants in hexadecimal. Proposed new functions: dfdl:hByte dfdl:hUByte dfdl:hShort dfdl:hUShort dfdl:hInt dfdl:hUInt dfdl:hLong dfdl:hULong The prefix 'h' is a reminder that the function converts from hexadecimal. The prefix 'U' on the name denotes that the function creates an unsigned type result. Normally one might like to write this out fully, but in this case brevity is helpful given the expected usage of these functions to construct literal constants in DFDL expressions. All the functions take a single string argument. The string must begin with "0x" and contain at least one hex digit after that. The string must contain only hexadecimal digits, that is, the characters 0-9a-zA-Z. It is a schema definition error otherwise. The hex digits represent a big endian twos complement representation of a binary number. Each function has a limit on the number of hex digits, with no more digits than 2, 4, 8, or 16 for the byte, short, int, and long versions. That is to say for dfdl:byte and dfdl:uByte, there can be at most 2 hex digits. For dfdl:short and dfdl:uShort, there can be at most 4 hex digits, and so on. It is a schema definition error if more digits are encountered than are suitable for the type being created. Examples: dfdl:hUInt("0xa1b2c3d4") is the value 2712847316. Note that in the first byte 'a1' the most significant bit is set, but since this is an unsigned type, this is not interpreted as a sign bit. dfdl:hInt("0xFFFFFFFF") is the int value -1. The sign bit indicates that the number is negative, and this twos complement value represents -1. dfdl:hUByte("0xFF") is the unsigned byte value 255 dfdl:hByte("0xff") is the byte value -1. dfdl:hByte("0x7F") is the byte value 127 dfdl:hByte("0x80") is the byte value -128 dfdl:hUByte("0x80") is the unsigned byte value 128 dfdl:hByte("0x0A3") is a schema definition error as the leading zero is not allowed because at most 2 digits are allowed for byte types. dfdl:hShort("0x0A3") is short value 208. Leading zero causes no issue here because up to 4 digits are allowed. Example of usage in expressions: <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:setVariable ref="ex:bOrd"> { if (xs:unsignedInt(.) eq dfdl:hUInt('0xa1b2c3d4')) then 'bigEndian' else if (xs:unsignedInt(.) eq dfdl:hUInt('0xd4c3b2a1')) then 'littleEndian' else fn:error(ex:magic_number, "Magic number was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).") } </dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 6

for errata r14 - Fwd: DFDL regular expressions and Unicode - conformance
by Mike Beckerle 24 Jul '13

24 Jul '13

I am assuming this issue will get handled as part of a r14 erratum. ---------- Forwarded message ---------- From: Steve Hanson <smh(a)uk.ibm.com> Date: Tue, Jul 16, 2013 at 11:13 AM Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode - conformance To: Andrew Edwards <andy.edwards(a)uk.ibm.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Jonathan No need for us to contact ICU, as Andy indicates below ICU and Java both claim conformance. Here's the words from errata 3.29. Please can you rephrase to combine the conformance requirement and the restrictions, so that we end up with a form you are happy with, then we can update the errata? *A DFDL regular expression is defined by a set of valid pattern characters. For portability, a DFDL regular expression pattern is restricted to the inclusive subset of the ICU regular expression [ICURE] and the Java(R) 7 regular expression [JAVARE] with the Unicode flags UNICODE_CASE and UNICODE_CHARACTER_CLASS turned on.* Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, "Cranford, Jonathan W." <jcranford(a)mitre.org> Date: 11/07/2013 14:19 Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode ------------------------------ Hi Jonathan, Sorry for the delay; first week back in the office... As you've noted, errata 3.29 describes what DFDL regexes are supported. Specifically, it is a subset of Java 7's java.util.regex ( http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html) and ICU's regular expression support ( http://userguide.icu-project.org/strings/regexp) both of which conform with level 1 of Unicode technical standard #18 It looks like there are 2 stages to checking conformance: - Logical - do the available regex constructs provide conformance to the technical standard. This is probably just a couple of hours of reading the Unicode standard rules and cross-checking the constructs in each matching engine. - Actual - do Java 7 and ICU really match properly for each of the conformance statements. This can take an ever increasing amount of time testing various sets of data and regex patterns, and it risks the only reward being that we find bugs in Java 7 or ICU. Minimum would be 3 or 4 days of test generation. Does that answer the issue? Andy *Andy Edwards* - *IBM Integration Bus*<http://www-03.ibm.com/software/products/us/en/integration-bus>- *DFDL*<https://w3-connections.ibm.com/wikis/home?lang=en-gb#!/wiki/IBM%20Data%20Fo…> *Email:* *andy.edwards(a)uk.ibm.com* <andy.edwards(a)uk.ibm.com> *Snail Mail:* MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN *Tel int:* 247222 *Tel ext:* +44 (0)1962 817222 *Desk:* DE3 V17 *The Feynman problem solving Algorithm* 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU *Steve Hanson/UK/IBM* 08/07/2013 11:08 To "Cranford, Jonathan W." <jcranford(a)mitre.org>, cc "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] DFDL regular expressions and UnicodeLink<Notes://D06ML014/80256D7F004ED63A/38D46BF5E8F08834852564B500129B2C/8054F31FB22A8880A1C918FA98057ED6> Jonathan I've copied Andy who added regexs support into IBM DFDL recently. He might have an idea as to the effort involved in stating conformance. We will discuss your other two emails on next DFDL-WG call or so. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 06/07/2013 00:56 Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode Sent by: dfdl-wg-bounces(a)ogf.org ------------------------------ Update: I just found errata 3.29, which answers this question, I think. >From the description in the errata, and looking at the documentation for java 7 regular expressions, it looks like DFDL regular expressions conform to level 1 of Unicode Regular expressions (UTS#18). I still think there would be value in stating such conformance in the DFDL spec, but I suppose that would take some legwork for someone to actually confirm the conformance of ICU and Java7 to level 1. Very respectfully, -- Jonathan Cranford >-----Original Message----- >From: Cranford, Jonathan W. >Sent: Friday, July 05, 2013 1:36 PM >To: dfdl-wg(a)ogf.org >Subject: DFDL regular expressions and Unicode > >I've been going through the spec recently, and I have a few questions about DFDL >regular expressions. > >Rather than put them into one long email, I'll break them up into separate emails. > >First question: What level of conformance to Unicode Technical Standard #18 >UNICODE > REGULAR EXPRESSIONS do DFDL regular expressions claim? > > For example, > * XML Schema regular expressions are "targeted at support of 'Level 1' >features" > (http://www.w3.org/TR/xmlschema-2/#dt-ccesN) > * Java 1.4 regular expressions "implement its second level of support" > ( http://docs.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.html) > * Perl 5.18 seems to implement most of Level 1 > ( http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression- >Support-Level) > > I think the conformance level should be specified in the DFDL spec so that it is >clear to schema > designers what a regular expression would really match against. Details > like case conversion and canonical equivalence make a difference when > matching against a Unicode string. > >Thanks in advance, > >-- >Jonathan W. Cranford <jcranford(a)mitre.org> >Senior Information Systems Engineer >The MITRE Corporation (http://www.mitre.org) -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

Broadcast: OGF DFDL WG Extra Call (24 Jul 15:00 GDT in No room booked)
by Steve Hanson 24 Jul '13

24 Jul '13

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

questions/feedback on dfdl:assert
by Cranford, Jonathan W. 23 Jul '13

23 Jul '13

DFDL WG, Some questions and feedback on Section 7.3.1 Properties for dfdl:assert. I don't think any of these are major issues. > "Schema authors can insert xs:sequence constructs to control the timing of evaluation of statements more precisely." An example here would illustrate and clarify the concept. This same verbiage is used in other sections of the spec, but I would think that only a single example would be sufficient to demonstrate the technique. Under the "test" property: > "The expression must have been evaluated by the time this element and it descendents have been processed." I'm lost in the grammar here. Is that equivalent to "The expression must be evaluated before this element and its descendants are fully processed"? (Changes highlighted). Also, should the spec define what it means to "process" an element for this statement to have any weight? > "It is a schema definition error if dfdl:test is the empty string and the value is not specified and dfdl:testKind is 'expression' or not specified." Huh? If the value of what isn't specified? The value of the dfdl:assert annotation? Under the "testPattern" property: > "It is a schema definition error if dfdl:testPattern is the empty string and the value is not specified and dfdl:testKind is 'pattern'." Same comment here. Should this be "... and the value of the dfdl:assert annotation is not specified..."? > "In order for a testPattern to be used, the data subject to the pattern must be scannable using a DFDL regular expression otherwise the results are not predictable." What does it mean for data to be "scannable"? Does this mean that the representation has to be text, since it's the physical representation of the data that is compared to the regular expression? Very respectfully, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

3 2

OGF DFDL WG Call Minutes 2013-07-23
by Steve Hanson 23 Jul '13

23 Jul '13

Please find minutes from the above call attached. These are intermediate minutes as there is a follow-on call Wed 24th @ 3pm UK / 10am Eastern. Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

issue: scannable and 'results are not predictable'
by Mike Beckerle 23 Jul '13

23 Jul '13

I was editing the definition of scannable into the glossary and when I looked at usage of 'scannable' in testPattern I found this: In the box for testPattern it says if the data is not scannable "the results are not predictable". Is that sufficient? We can sometimes statically determine that the schema says the data should all be scannable (e.g., no change of encoding, no binary elements), and that would rule out one non-predictability. So, if data is non-scannable in the sense that the schema contains say, a binary element, we can issue an SDE if lengthKind is pattern or a testPattern assert is being used. We could also SDE if runtime-valued encoding properties are used and the encoding changes inside a scannable context. Well, I guess testKind pattern asserts/discriminators are an issue because they may look only at the first part of the data of a complex component, so they don't require everything to be scannable, only the part the regex actually examines. So in this case it's user-beware, and if non-scannable I suppose we could issue a warning. But the spec does not say this is an SDE or warning currently. It just says results are not predictable. There is also the fact that the data might be broken, i.e., the schema might say the data is scannable, but at parse time character decode errors occur. I believe our policy on this is that these cause processing errors. This really is orthogonal to scannable, which is a property of a schema component. Comments? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 4

Rescheduled: OGF DFDL WG Call (23 Jul 14:30 GDT in DE3S09)
by Steve Hanson 22 Jul '13

22 Jul '13

Rescheduled: OGF DFDL WG Call 23/07/2013 - (Repeats) Location: DE3S09 Steve Hanson has rescheduled this meeting. You have not added this entry to your calendar. Repeats This entry repeats Required: mbeckerle.dfdl(a)gmail.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org, Stephanie Fetzer/Charlotte/IBM@IBMUS OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0