- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] DFDL Expressions and hex literal constants
by Tim Kimber 17 Jun '13

17 Jun '13

Even that won't work, unfortunately. XPath 2.0 does not allow a decimal or integer to be cast to xs:hexBinary in this way: if (xs:hexBinary(../magicNum)... I think we're back where we started - either we define an optional pair of DFDL extension functions, or we accept the limitations of XPath 2.0. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 17/06/2013 14:52 Subject: Re: [DFDL-WG] DFDL Expressions and hex literal constants Isn't this where you use the xs:hexBinary constructor? I have examples in my IBM4690 TLOG ACE schema where I need to check that the content of a hexBinary field is a certain value and it works as desired with IBM DFDL. <xs:element dfdl:length="1" dfdl:lengthKind="explicit" fixed="13" name="SubStringType" type="type_PS"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator>{. eq xs:hexBinary('13')}</dfdl:discriminator> </xs:appinfo> </xs:annotation> </xs:element> <xs:element dfdl:length="1" dfdl:lengthKind="explicit" fixed="BB" name="SubStringType" type="type_PS"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator>{. eq xs:hexBinary('BB')}</dfdl:discriminator> </xs:appinfo> </xs:annotation> </xs:element> Then Mike's example would look like: { if (xs:hexBinary(../magicNum) eq xs:hexBinary('a1b2c3d4') then 'bigEndian' else if (xs:hexBinary(../magicNum) eq xs:hexBinary('d4c3b2a1') then 'littleEndian' else error() } Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 13/06/2013 11:27 Subject: Re: [DFDL-WG] DFDL Expressions and hex literal constants Sent by: dfdl-wg-bounces(a)ogf.org XPath uses XML schema lexical formats for all simple types. The XML Schema lexical format for an integer does not allow a hexadecimal option. The standard solution is to create a pair of custom XPath functions that convert between hex and decimal lexical formats. But that's not a viable solution for a DFDL modeller because they cannot define custom XPath functions. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 13/06/2013 10:39 Subject: Re: [DFDL-WG] DFDL Expressions and hex literal constants Sent by: dfdl-wg-bounces(a)ogf.org Mike I think there might be a wider problem in specifying any hex in XPath expressions. Tim's been looking into this as part of an IBM DFDL defect. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 13/06/2013 00:24 Subject: [DFDL-WG] DFDL Expressions and hex literal constants Sent by: dfdl-wg-bounces(a)ogf.org I haven't found a way in XPath to specify a hexadecimal integer number. E.g., I want to write expressions like { if (../magicNum eq 0xa1b2c3d4) then 'bigEndian' else if (.../magicNum eq 0xd4c3b2a1) then 'littleEndian' else error() } But it rejects the 0x notation. Do I really have to write decimal everywhere? When dealing with binary data this is a bit of a nightmare. I think the above code is clear to anyone, but if I convert those numbers to decimal it becomes completely obscure. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2013-06-18
by Steve Hanson 17 Jun '13

17 Jun '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13089 Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

DFDL Expressions and hex literal constants
by Mike Beckerle 17 Jun '13

17 Jun '13

I haven't found a way in XPath to specify a hexadecimal integer number. E.g., I want to write expressions like { if (../magicNum eq 0xa1b2c3d4) then 'bigEndian' else if (.../magicNum eq 0xd4c3b2a1) then 'littleEndian' else error() } But it rejects the 0x notation. Do I really have to write decimal everywhere? When dealing with binary data this is a bit of a nightmare. I think the above code is clear to anyone, but if I convert those numbers to decimal it becomes completely obscure. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 4

Re: [DFDL-WG] optional whitespace entity
by Steve Hanson 13 Jun '13

13 Jun '13

Or, assuming you have dfdl:separator '%SP;' on the parent sequence, change that to '%SP;%WSP*;' Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: "Garriss Jr., James P." <jgarriss(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 13/06/2013 13:50 Subject: Re: [DFDL-WG] optional whitespace entity You could use dfdl:textTrimKind 'padChar', dfdl:textStringPadCharacter'%SP;' and dfdl:textStringJustification 'right', and trim off the excess space. I'm guessing that your pad char is a '0' right now but the '0' is harmless. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 13/06/2013 13:20 Subject: Re: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org > can you show us the scenario where you want to apply this? Sure. Consider the Date header. Note that the day of the month is *always* 2 digits (that is, it’s 04 instead of just 4): Date: Fri, 04 Feb 2013 08:54:52 -0500 Now consider the Received header, which finishes with a date. Sometimes the day of the month is 2 digits, when the day is 10 or higher: Received: by mail-wi0-f178.google.com with SMTP id hj6so339193wib.11 for <jgarriss(a)mitre.org>; Thu, 30 May 2013 22:28:57 -0700 (PDT) Sometimes it is 2 characters but instead of a leading 0 (like the Date header above), there is a blank space preceding the day. If you look closely in this example, you will see that there are 2 spaces between “Tue,” and “4 Jun”: Received: from 131.28.34.56 ([131.28.34.56]) by VFOHMLAO03.Enterprise.afmc.ds.af.mil ([131.28.34.43]) via Exchange Front-End Server webmail.afmc.af.mil ([131.28.34.85]) with Microsoft Exchange Server HTTP-DAV ; Tue, 4 Jun 2013 18:02:13 +0000 And sometimes it is 1 digit: Received: from smtpksrv1.mitre.org (129.83.31.51) by IMCCAS03.MITRE.ORG (129.83.29.80) with Microsoft SMTP Server id 14.2.342.3; Tue, 4 Jun 2013 09:31:47 -0400 The problem is how to model the Day element when it’s part of the Received header. If the length is merely set to 2 characters, then the value can be “ 4”, which Daffodil complains that it can’t convert it into an integer. So I settled on this:  <xsd:element name="Day" dfdl:lengthKind="delimited" dfdl:initiator="%WSP*;"> <xsd:annotation> <xsd:appinfo source=" http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="There cannot be more than 31 days in a month"/> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base= "xsd:unsignedInt"> <xsd:maxInclusive value= "31"/> </xsd:restriction> </xsd:simpleType> </xsd:element> (I have a different version of the Day element for the Date header.) I started this thread because I wanted to make sure that this initiator would work whether the extra space is present or absent. BTW, happy ending: it works as expected in Daffodil 0.10. > I am wondering whether whether %WSP*; on its own should be allowed as a delimiter? If not, then please provide some solution for the above problem. From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Thursday, June 13, 2013 5:30 AM To: Garriss Jr., James P. Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] optional whitespace entity James, please can you show us the scenario where you want to apply this? I ask because I think it is the only example in DFDL where you can specify a DFDL delimiter and for there to be nothing for that delimiter in the data. I suspect this might have some ramifications for things like delimiter scanning and dfdl:initiatedContent. It clearly solves a problem for James, but I am wondering whether whether %WSP*; on its own should be allowed as a delimiter? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 11/06/2013 16:19 Subject: Re: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org You know, Tim, that is what I meant, but I just copied the syntax directly from Table 4. Upon further review, I see that this table doesn’t give the complete syntax. I wonder how many other people will just cut-and-paste directly from these tables. It might be a good idea to put the complete syntax there. In any case, thanks for the explanation. That’s what I hoped it would do. From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Tim Kimber Sent: Tuesday, June 11, 2013 11:02 AM To: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] optional whitespace entity I assume that you meant to write dfdl:initiator="%WSP*;" That will match zero or more whitespace characters. It will match and consume any leading white space before the element, and it will never fail to match. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 11/06/2013 15:48 Subject: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org <xsd:element name="Day" dfdl:lengthKind="delimited" dfdl:initiator="WSP*"> Will this element match if the initiator is not found (that is, even if there is not a space before element? TIA-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

optional whitespace entity
by Garriss Jr., James P. 13 Jun '13

13 Jun '13

<xsd:element name="Day" dfdl:lengthKind="delimited" dfdl:initiator="WSP*"> Will this element match if the initiator is not found (that is, even if there is not a space before element? TIA

3 5

Zoned decimals: spec errata 2.92 & 2.88
by Steve Hanson 10 Jun '13

10 Jun '13

2.92. Section 13.6. When property textNumberRep is ‘zoned’, the property description should state that ‘zoned’ is only allowed for SBCS encodings (schema definition error otherwise). When I came to implement this for IBM DFDL, I noticed there were already tests for UTF-8 and Shift_JIS which succeeded. The point being that both these character sets are ASCII compatible for the first 128 code points (Shift_JIS has two code points that differ, x5C and x7E). I am wondering if this errata is therefore too strict? I am particularly concerned that there might be Japanese users who will have COBOL data in Shift_JIS or MS_Kanji. 2.88. Section 13.5. Add support for HP NonStop Tandem zoned decimals. In this architecture, the negative sign is incorporated in the last byte of the number in the usual manner, but the overpunching occurs on the highest bit (ie, value 8) of the nibble. Consequently, a new enum value 'asciiTandemModified’ is added to property textZonedSignStyle. The range of ASCII code points that are used in a zoned number is x30-x39 and either x70-x79 (standard overpunch) or x7B, x41-x49 (translated EBCDIC overpunch) or x20-x29 (CA Realia overpunch). This errata adds x80-x89. But these are not code points in standard ASCII, so the modeller must specify something like ISO-8859-1 in order for this to parse without an encoding error. The wording in the spec for this errata alludes to this but could make this clearer. asciiTandemModified: In this style the ascii characters ‘0-9’ represent positive sign and digits 0 to 9, but bytes from 0x80 to 0x89 are used to represent overpunched negative sign and a digit. There are no corresponding character codepoints in the standard ASCII encoding since these values are all above 128 (decimal). (Note that neither ISO-8859-1 encoding nor Unicode have assigned glyphs for these codepoints. They are considered control characters.) Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Ignore extraneous CRLF w/ space?
by Garriss Jr., James P. 07 Jun '13

07 Jun '13

Long IMF headers, such as Received, can be wrapped onto the next line by using a CRLF and then a space. This example has 3 such wrappings: Received: from smtpksrv1.mitre.org (localhost.localdomain [127.0.0.1]) by localhost (Postfix) via Exchange Front-End Server webmail.afmc.af.mil ([131.28.34.85]) with SMTP id 0A8791F116E for <jgarriss(a)mitre.org>; Tue, 4 Jun 2013 14:03:24 -0400 (EDT) How do I get DFDL to ignore these wrappings? For most of the header, it's not an issue, because I can use a lengthPattern to lookahead to the ; before the date starts. But once the date starts, I have no way of knowing when it ends, so I need to ignore any CRLF with a space. TIA

2 7

of check constraints and fixed
by Garriss Jr., James P. 07 Jun '13

07 Jun '13

Check constraints can be used... <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="There cannot be more than 59 seconds in a minute."/> </xsd:appinfo> </xsd:annotation> ...to validate (while parsing) the restrictions placed on an element: <xsd:simpleType> <xsd:restriction base="xsd:unsignedInt"> <xsd:maxInclusive value="59"/> </xsd:restriction> </xsd:simpleType> Is check constraints similarly required to validate the fixed value of an element while parsing? <xsd:element name="MimeVersion" dfdl:initiator="MIME-Version:%SP;" dfdl:terminator="%NL;" type="xsd:string" dfdl:length="3" dfdl:lengthKind="explicit" fixed="1.0"> If not, then how do I force it to validate while parsing? TIA

2 1

Cancelled: OGF DFDL WG Call
by Steve Hanson 06 Jun '13

06 Jun '13

Steve is on leave

1 0

OGF DFDL WG Call Minutes 2013-06-04
by Steve Hanson 05 Jun '13

05 Jun '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/13082?download= Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0