- dfdl-wg - lists.ogf.org

proposal: functions to allow reasonable syntax for hex constants in DFDL expression language
by Mike Beckerle 20 Jun '13

20 Jun '13

Rationale: in writing DFDL schemas for binary data formats, there is a strong need to express binary data constants in hexadecimal. Proposed new functions: dfdl:hByte dfdl:hUByte dfdl:hShort dfdl:hUShort dfdl:hInt dfdl:hUInt dfdl:hLong dfdl:hULong The prefix 'h' is a reminder that the function converts from hexadecimal. The prefix 'U' on the name denotes that the function creates an unsigned type result. Normally one might like to write this out fully, but in this case brevity is helpful given the expected usage of these functions to construct literal constants in DFDL expressions. All the functions take a single string argument. The string must begin with "0x" and contain at least one hex digit after that. The string must contain only hexadecimal digits, that is, the characters 0-9a-zA-Z. It is a schema definition error otherwise. The hex digits represent a big endian twos complement representation of a binary number. Each function has a limit on the number of hex digits, with no more digits than 2, 4, 8, or 16 for the byte, short, int, and long versions. That is to say for dfdl:byte and dfdl:uByte, there can be at most 2 hex digits. For dfdl:short and dfdl:uShort, there can be at most 4 hex digits, and so on. It is a schema definition error if more digits are encountered than are suitable for the type being created. Examples: dfdl:hUInt("0xa1b2c3d4") is the value 2712847316. Note that in the first byte 'a1' the most significant bit is set, but since this is an unsigned type, this is not interpreted as a sign bit. dfdl:hInt("0xFFFFFFFF") is the int value -1. The sign bit indicates that the number is negative, and this twos complement value represents -1. dfdl:hUByte("0xFF") is the unsigned byte value 255 dfdl:hByte("0xff") is the byte value -1. dfdl:hByte("0x7F") is the byte value 127 dfdl:hByte("0x80") is the byte value -128 dfdl:hUByte("0x80") is the unsigned byte value 128 dfdl:hByte("0x0A3") is a schema definition error as the leading zero is not allowed because at most 2 digits are allowed for byte types. dfdl:hShort("0x0A3") is short value 208. Leading zero causes no issue here because up to 4 digits are allowed. Example of usage in expressions: <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:setVariable ref="ex:bOrd"> { if (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) then 'bigEndian' else if (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) then 'littleEndian' else fn:error(ex:magic_number, "Magic number was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).") } </dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 2

how to do mix of case sensitive and insensitive delimiters
by Mike Beckerle 20 Jun '13

20 Jun '13

I have a wierd case where the initiator wants to be case insensitive matching, but the terminator wants to be case sensitive. The only way I can think of dealing with this is to use the initiator, but handle the length via lengthKind='pattern' to grab the value, doing lookahead so it will stop before the terminator. Then an empty sequence with a case sensitive terminator to pick off that part of the data stream. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

4 6

Cancelled: OGF DFDL WG Call
by Steve Hanson 20 Jun '13

20 Jun '13

Steve unavailable

1 0

How do I ignore the case of my initiator and my elements?
by Garriss Jr., James P. 20 Jun '13

20 Jun '13

Issue #1 Email header names have a set value, but in practice SMTP servers do what they want. So this is the required header name for content type: Content-Type: <some value> But variations exist, such as this: Content-type: <some value> Rather than trying to guess every possible combination, I think it would be good to simply ignore the case of the header name. (I have modeled header names as initiators.) Issue #2 The spec says that the value of the content type is case insensitive, so all the following possibilities (and more) are allowed: Content-Type: text/html Content-Type: TEXT/HTML Content-Type: Text/Html Content-Type: tExT/hTmL So obviously I need to be able to ignore the case of the header's value. (I have modeled the type (e.g., text) and the subtype (e.g., html) as elements. And as there's obviously a limited set of allowed for each, I've constrained these with an enumerated list.) I have attached a simplified version of the schema I'm using for the content header. How would I modify it in order to ignore the case of the header name and the header value? TIA!

2 1

clarification needed: Ignore case
by Mike Beckerle 20 Jun '13

20 Jun '13

The ignoreCase property description says this: Whether mixed case data is accepted when matching delimiters and data values on input. This affects the behavior of matching for these properties: initiator, terminator, separator, nilValue, textStandardExponentRep, textStandardInfinityRep, textStandardNaNRep, textStandardZeroRep, textBooleanTrue, textBooleanFalse, Ignoring that dangling comma at the end, the question is about what this leaves out, and making sure this is intentional. 1. If an element of type string has the XSD attribute fixed="...." with some value, is that comparison of the fixed value to the data content guided by dfdl:ignoreCase? 2. If an element of simple type string has enumerations, are the comparisons of the enum values to what is found in the data guided by dfdl:ignoreCase? My thought is No and No, as these are purely XSD issues, not DFDL related, so we want our behavior to be exactly the same as XSD here. The XSD/XML world is simply a very case-sensitive one. This page (http://www.ibm.com/developerworks/library/x-case/) is about a user asking how to do case-insensitive enum comparisons in XSD. The hack workaround is changing to use a pattern/regex match. I.e., instead of case insensitive "red" you get a pattern match of "[r|R][e|E][d|D]" which is blechy, but effective. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 2

clarification on when escape characters are needed
by Mike Beckerle 19 Jun '13

19 Jun '13

Suppose I have a sequence. It has an infix separator which is "=". <sequence dfdl:separator="=" dfdl:separatorPosition="infix"> <element name="a" type="xs:string"/> <element name="b" type="xs:string"/> </sequence> Now, consider this data: password=f82+=7&%q I want <a>password</a> <b>f82+=7&%q</b> Notice how the b element contains an '=' which was not escaped in any way in the sequence. Element b is statically known to be last, the separator is infix; hence, things are unambiguous even if there is no escaping. However, there is an alternative interpretation, which is that the above data should fail, because it produces <a>password</a><b>f82+</b> but then does not find the expected stuff next. Rather it finds the '=7&%q' data. In other words, the sequence separator divides the sequence content into 3 content regions, but there aren't 3 things to consume those, so it is a processing error. Which is correct? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

4 10

Re: [DFDL-WG] DFDL Expressions and hex literal constants
by Tim Kimber 17 Jun '13

17 Jun '13

Even that won't work, unfortunately. XPath 2.0 does not allow a decimal or integer to be cast to xs:hexBinary in this way: if (xs:hexBinary(../magicNum)... I think we're back where we started - either we define an optional pair of DFDL extension functions, or we accept the limitations of XPath 2.0. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 17/06/2013 14:52 Subject: Re: [DFDL-WG] DFDL Expressions and hex literal constants Isn't this where you use the xs:hexBinary constructor? I have examples in my IBM4690 TLOG ACE schema where I need to check that the content of a hexBinary field is a certain value and it works as desired with IBM DFDL. <xs:element dfdl:length="1" dfdl:lengthKind="explicit" fixed="13" name="SubStringType" type="type_PS"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator>{. eq xs:hexBinary('13')}</dfdl:discriminator> </xs:appinfo> </xs:annotation> </xs:element> <xs:element dfdl:length="1" dfdl:lengthKind="explicit" fixed="BB" name="SubStringType" type="type_PS"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator>{. eq xs:hexBinary('BB')}</dfdl:discriminator> </xs:appinfo> </xs:annotation> </xs:element> Then Mike's example would look like: { if (xs:hexBinary(../magicNum) eq xs:hexBinary('a1b2c3d4') then 'bigEndian' else if (xs:hexBinary(../magicNum) eq xs:hexBinary('d4c3b2a1') then 'littleEndian' else error() } Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 13/06/2013 11:27 Subject: Re: [DFDL-WG] DFDL Expressions and hex literal constants Sent by: dfdl-wg-bounces(a)ogf.org XPath uses XML schema lexical formats for all simple types. The XML Schema lexical format for an integer does not allow a hexadecimal option. The standard solution is to create a pair of custom XPath functions that convert between hex and decimal lexical formats. But that's not a viable solution for a DFDL modeller because they cannot define custom XPath functions. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 13/06/2013 10:39 Subject: Re: [DFDL-WG] DFDL Expressions and hex literal constants Sent by: dfdl-wg-bounces(a)ogf.org Mike I think there might be a wider problem in specifying any hex in XPath expressions. Tim's been looking into this as part of an IBM DFDL defect. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 13/06/2013 00:24 Subject: [DFDL-WG] DFDL Expressions and hex literal constants Sent by: dfdl-wg-bounces(a)ogf.org I haven't found a way in XPath to specify a hexadecimal integer number. E.g., I want to write expressions like { if (../magicNum eq 0xa1b2c3d4) then 'bigEndian' else if (.../magicNum eq 0xd4c3b2a1) then 'littleEndian' else error() } But it rejects the 0x notation. Do I really have to write decimal everywhere? When dealing with binary data this is a bit of a nightmare. I think the above code is clear to anyone, but if I convert those numbers to decimal it becomes completely obscure. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2013-06-18
by Steve Hanson 17 Jun '13

17 Jun '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13089 Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

DFDL Expressions and hex literal constants
by Mike Beckerle 17 Jun '13

17 Jun '13

I haven't found a way in XPath to specify a hexadecimal integer number. E.g., I want to write expressions like { if (../magicNum eq 0xa1b2c3d4) then 'bigEndian' else if (.../magicNum eq 0xd4c3b2a1) then 'littleEndian' else error() } But it rejects the 0x notation. Do I really have to write decimal everywhere? When dealing with binary data this is a bit of a nightmare. I think the above code is clear to anyone, but if I convert those numbers to decimal it becomes completely obscure. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 4

Re: [DFDL-WG] optional whitespace entity
by Steve Hanson 13 Jun '13

13 Jun '13

Or, assuming you have dfdl:separator '%SP;' on the parent sequence, change that to '%SP;%WSP*;' Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: "Garriss Jr., James P." <jgarriss(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 13/06/2013 13:50 Subject: Re: [DFDL-WG] optional whitespace entity You could use dfdl:textTrimKind 'padChar', dfdl:textStringPadCharacter'%SP;' and dfdl:textStringJustification 'right', and trim off the excess space. I'm guessing that your pad char is a '0' right now but the '0' is harmless. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 13/06/2013 13:20 Subject: Re: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org > can you show us the scenario where you want to apply this? Sure. Consider the Date header. Note that the day of the month is *always* 2 digits (that is, it’s 04 instead of just 4): Date: Fri, 04 Feb 2013 08:54:52 -0500 Now consider the Received header, which finishes with a date. Sometimes the day of the month is 2 digits, when the day is 10 or higher: Received: by mail-wi0-f178.google.com with SMTP id hj6so339193wib.11 for <jgarriss(a)mitre.org>; Thu, 30 May 2013 22:28:57 -0700 (PDT) Sometimes it is 2 characters but instead of a leading 0 (like the Date header above), there is a blank space preceding the day. If you look closely in this example, you will see that there are 2 spaces between “Tue,” and “4 Jun”: Received: from 131.28.34.56 ([131.28.34.56]) by VFOHMLAO03.Enterprise.afmc.ds.af.mil ([131.28.34.43]) via Exchange Front-End Server webmail.afmc.af.mil ([131.28.34.85]) with Microsoft Exchange Server HTTP-DAV ; Tue, 4 Jun 2013 18:02:13 +0000 And sometimes it is 1 digit: Received: from smtpksrv1.mitre.org (129.83.31.51) by IMCCAS03.MITRE.ORG (129.83.29.80) with Microsoft SMTP Server id 14.2.342.3; Tue, 4 Jun 2013 09:31:47 -0400 The problem is how to model the Day element when it’s part of the Received header. If the length is merely set to 2 characters, then the value can be “ 4”, which Daffodil complains that it can’t convert it into an integer. So I settled on this:  <xsd:element name="Day" dfdl:lengthKind="delimited" dfdl:initiator="%WSP*;"> <xsd:annotation> <xsd:appinfo source=" http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="There cannot be more than 31 days in a month"/> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base= "xsd:unsignedInt"> <xsd:maxInclusive value= "31"/> </xsd:restriction> </xsd:simpleType> </xsd:element> (I have a different version of the Day element for the Date header.) I started this thread because I wanted to make sure that this initiator would work whether the extra space is present or absent. BTW, happy ending: it works as expected in Daffodil 0.10. > I am wondering whether whether %WSP*; on its own should be allowed as a delimiter? If not, then please provide some solution for the above problem. From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Thursday, June 13, 2013 5:30 AM To: Garriss Jr., James P. Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] optional whitespace entity James, please can you show us the scenario where you want to apply this? I ask because I think it is the only example in DFDL where you can specify a DFDL delimiter and for there to be nothing for that delimiter in the data. I suspect this might have some ramifications for things like delimiter scanning and dfdl:initiatedContent. It clearly solves a problem for James, but I am wondering whether whether %WSP*; on its own should be allowed as a delimiter? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 11/06/2013 16:19 Subject: Re: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org You know, Tim, that is what I meant, but I just copied the syntax directly from Table 4. Upon further review, I see that this table doesn’t give the complete syntax. I wonder how many other people will just cut-and-paste directly from these tables. It might be a good idea to put the complete syntax there. In any case, thanks for the explanation. That’s what I hoped it would do. From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Tim Kimber Sent: Tuesday, June 11, 2013 11:02 AM To: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] optional whitespace entity I assume that you meant to write dfdl:initiator="%WSP*;" That will match zero or more whitespace characters. It will match and consume any leading white space before the element, and it will never fail to match. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 11/06/2013 15:48 Subject: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org <xsd:element name="Day" dfdl:lengthKind="delimited" dfdl:initiator="WSP*"> Will this element match if the initiator is not found (that is, even if there is not a space before element? TIA-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1