June 2013 - dfdl-wg - lists.ogf.org

DFDL Expressions and hex literal constants
by Mike Beckerle 17 Jun '13

17 Jun '13

I haven't found a way in XPath to specify a hexadecimal integer number. E.g., I want to write expressions like { if (../magicNum eq 0xa1b2c3d4) then 'bigEndian' else if (.../magicNum eq 0xd4c3b2a1) then 'littleEndian' else error() } But it rejects the 0x notation. Do I really have to write decimal everywhere? When dealing with binary data this is a bit of a nightmare. I think the above code is clear to anyone, but if I convert those numbers to decimal it becomes completely obscure. ...mike -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 4

Re: [DFDL-WG] optional whitespace entity
by Steve Hanson 13 Jun '13

13 Jun '13

Or, assuming you have dfdl:separator '%SP;' on the parent sequence, change that to '%SP;%WSP*;' Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: "Garriss Jr., James P." <jgarriss(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 13/06/2013 13:50 Subject: Re: [DFDL-WG] optional whitespace entity You could use dfdl:textTrimKind 'padChar', dfdl:textStringPadCharacter'%SP;' and dfdl:textStringJustification 'right', and trim off the excess space. I'm guessing that your pad char is a '0' right now but the '0' is harmless. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 13/06/2013 13:20 Subject: Re: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org > can you show us the scenario where you want to apply this? Sure. Consider the Date header. Note that the day of the month is *always* 2 digits (that is, it’s 04 instead of just 4): Date: Fri, 04 Feb 2013 08:54:52 -0500 Now consider the Received header, which finishes with a date. Sometimes the day of the month is 2 digits, when the day is 10 or higher: Received: by mail-wi0-f178.google.com with SMTP id hj6so339193wib.11 for <jgarriss(a)mitre.org>; Thu, 30 May 2013 22:28:57 -0700 (PDT) Sometimes it is 2 characters but instead of a leading 0 (like the Date header above), there is a blank space preceding the day. If you look closely in this example, you will see that there are 2 spaces between “Tue,” and “4 Jun”: Received: from 131.28.34.56 ([131.28.34.56]) by VFOHMLAO03.Enterprise.afmc.ds.af.mil ([131.28.34.43]) via Exchange Front-End Server webmail.afmc.af.mil ([131.28.34.85]) with Microsoft Exchange Server HTTP-DAV ; Tue, 4 Jun 2013 18:02:13 +0000 And sometimes it is 1 digit: Received: from smtpksrv1.mitre.org (129.83.31.51) by IMCCAS03.MITRE.ORG (129.83.29.80) with Microsoft SMTP Server id 14.2.342.3; Tue, 4 Jun 2013 09:31:47 -0400 The problem is how to model the Day element when it’s part of the Received header. If the length is merely set to 2 characters, then the value can be “ 4”, which Daffodil complains that it can’t convert it into an integer. So I settled on this:  <xsd:element name="Day" dfdl:lengthKind="delimited" dfdl:initiator="%WSP*;"> <xsd:annotation> <xsd:appinfo source=" http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="There cannot be more than 31 days in a month"/> </xsd:appinfo> </xsd:annotation> <xsd:simpleType> <xsd:restriction base= "xsd:unsignedInt"> <xsd:maxInclusive value= "31"/> </xsd:restriction> </xsd:simpleType> </xsd:element> (I have a different version of the Day element for the Date header.) I started this thread because I wanted to make sure that this initiator would work whether the extra space is present or absent. BTW, happy ending: it works as expected in Daffodil 0.10. > I am wondering whether whether %WSP*; on its own should be allowed as a delimiter? If not, then please provide some solution for the above problem. From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Thursday, June 13, 2013 5:30 AM To: Garriss Jr., James P. Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org Subject: Re: [DFDL-WG] optional whitespace entity James, please can you show us the scenario where you want to apply this? I ask because I think it is the only example in DFDL where you can specify a DFDL delimiter and for there to be nothing for that delimiter in the data. I suspect this might have some ramifications for things like delimiter scanning and dfdl:initiatedContent. It clearly solves a problem for James, but I am wondering whether whether %WSP*; on its own should be allowed as a delimiter? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 11/06/2013 16:19 Subject: Re: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org You know, Tim, that is what I meant, but I just copied the syntax directly from Table 4. Upon further review, I see that this table doesn’t give the complete syntax. I wonder how many other people will just cut-and-paste directly from these tables. It might be a good idea to put the complete syntax there. In any case, thanks for the explanation. That’s what I hoped it would do. From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Tim Kimber Sent: Tuesday, June 11, 2013 11:02 AM To: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] optional whitespace entity I assume that you meant to write dfdl:initiator="%WSP*;" That will match zero or more whitespace characters. It will match and consume any leading white space before the element, and it will never fail to match. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "Garriss Jr., James P." <jgarriss(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 11/06/2013 15:48 Subject: [DFDL-WG] optional whitespace entity Sent by: dfdl-wg-bounces(a)ogf.org <xsd:element name="Day" dfdl:lengthKind="delimited" dfdl:initiator="WSP*"> Will this element match if the initiator is not found (that is, even if there is not a space before element? TIA-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

optional whitespace entity
by Garriss Jr., James P. 13 Jun '13

13 Jun '13

<xsd:element name="Day" dfdl:lengthKind="delimited" dfdl:initiator="WSP*"> Will this element match if the initiator is not found (that is, even if there is not a space before element? TIA

3 5

Zoned decimals: spec errata 2.92 & 2.88
by Steve Hanson 10 Jun '13

10 Jun '13

2.92. Section 13.6. When property textNumberRep is ‘zoned’, the property description should state that ‘zoned’ is only allowed for SBCS encodings (schema definition error otherwise). When I came to implement this for IBM DFDL, I noticed there were already tests for UTF-8 and Shift_JIS which succeeded. The point being that both these character sets are ASCII compatible for the first 128 code points (Shift_JIS has two code points that differ, x5C and x7E). I am wondering if this errata is therefore too strict? I am particularly concerned that there might be Japanese users who will have COBOL data in Shift_JIS or MS_Kanji. 2.88. Section 13.5. Add support for HP NonStop Tandem zoned decimals. In this architecture, the negative sign is incorporated in the last byte of the number in the usual manner, but the overpunching occurs on the highest bit (ie, value 8) of the nibble. Consequently, a new enum value 'asciiTandemModified’ is added to property textZonedSignStyle. The range of ASCII code points that are used in a zoned number is x30-x39 and either x70-x79 (standard overpunch) or x7B, x41-x49 (translated EBCDIC overpunch) or x20-x29 (CA Realia overpunch). This errata adds x80-x89. But these are not code points in standard ASCII, so the modeller must specify something like ISO-8859-1 in order for this to parse without an encoding error. The wording in the spec for this errata alludes to this but could make this clearer. asciiTandemModified: In this style the ascii characters ‘0-9’ represent positive sign and digits 0 to 9, but bytes from 0x80 to 0x89 are used to represent overpunched negative sign and a digit. There are no corresponding character codepoints in the standard ASCII encoding since these values are all above 128 (decimal). (Note that neither ISO-8859-1 encoding nor Unicode have assigned glyphs for these codepoints. They are considered control characters.) Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Ignore extraneous CRLF w/ space?
by Garriss Jr., James P. 07 Jun '13

07 Jun '13

Long IMF headers, such as Received, can be wrapped onto the next line by using a CRLF and then a space. This example has 3 such wrappings: Received: from smtpksrv1.mitre.org (localhost.localdomain [127.0.0.1]) by localhost (Postfix) via Exchange Front-End Server webmail.afmc.af.mil ([131.28.34.85]) with SMTP id 0A8791F116E for <jgarriss(a)mitre.org>; Tue, 4 Jun 2013 14:03:24 -0400 (EDT) How do I get DFDL to ignore these wrappings? For most of the header, it's not an issue, because I can use a lengthPattern to lookahead to the ; before the date starts. But once the date starts, I have no way of knowing when it ends, so I need to ignore any CRLF with a space. TIA

2 7

of check constraints and fixed
by Garriss Jr., James P. 07 Jun '13

07 Jun '13

Check constraints can be used... <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert test="{ dfdl:checkConstraints(.) }" message="There cannot be more than 59 seconds in a minute."/> </xsd:appinfo> </xsd:annotation> ...to validate (while parsing) the restrictions placed on an element: <xsd:simpleType> <xsd:restriction base="xsd:unsignedInt"> <xsd:maxInclusive value="59"/> </xsd:restriction> </xsd:simpleType> Is check constraints similarly required to validate the fixed value of an element while parsing? <xsd:element name="MimeVersion" dfdl:initiator="MIME-Version:%SP;" dfdl:terminator="%NL;" type="xsd:string" dfdl:length="3" dfdl:lengthKind="explicit" fixed="1.0"> If not, then how do I force it to validate while parsing? TIA

2 1

Cancelled: OGF DFDL WG Call
by Steve Hanson 06 Jun '13

06 Jun '13

Steve is on leave

1 0

OGF DFDL WG Call Minutes 2013-06-04
by Steve Hanson 05 Jun '13

05 Jun '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/13082?download= Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

outputValueCalc to compute array elements - not possible?
by Mike Beckerle 05 Jun '13

05 Jun '13

I want to create an array of short strings from a single long string using outputValueCalc. Let's call the initial single long string s. I want the max length of each short string to be m. So ceiling(s/m) is the number of array elements. Given m, it should be clear that each array element's contents can be computed as a specific sub-string of s, determined using m, and the index of the array element within the array, i.e., the position() function in XPath. The substring starts at m*(position() - 1), the end is length(s) - m*(position() -1) if position is ceiling(s/m), and start + m otherwise. However, current rules in the spec say that elements must exist in the infoset to be unparsed unless they are filled in by way of defaulting up to minOccurs. In my scenario I don't want to default them, I want to compute their values, but I do want to start with an infoset which has zero of the array elements pre-existing. I'm pretty sure there is no way for DFDL to will n array-elements into existence in the augmented infoset just knowing n, such that they then can each compute their sub-element's values using outputValueCalc. Also I think we currently don't allow outputValueCalc on an array element anyway. But my array element could be a complex type containing an element with an outputValueCalc. So I think there is no way to achieve what I want, eventhough there's no issue with figuring out how many array elements there should be nor any issue with figuring out the contents of each array element. I raise this by way of wanting to be able to unparse somehow, anything I can parse. I can parse a variable-length array of strings, and compute a string which is their concatenation. It's tricky but doable. I cannot break up a long string into pieces which are in an array. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1

Re: [DFDL-WG] terminate by next field's initiator aka lengthKind="endAtStartOfNext" or something like that
by Steve Hanson 04 Jun '13

04 Jun '13

Try modelling the < as an 'infix' separator, with suppression policy 'anyEmpty' which allows for it to be absent when the preceding field is empty. And model the > as part of the terminator of the record, so a list '%NL; >%NL;'. Then speculative parsing will then sort out 1 and 2, I hope. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 04/06/2013 15:02 Subject: Re: [DFDL-WG] terminate by next field's initiator aka lengthKind="endAtStartOfNext" or something like that MicroSoft's RTF example: {\rtf1\ansi\ansicpg1252\deff0\deflang1033{\fonttbl{\f0\fnil\fcharset0 Calibri;}} {\*\generator Msftedit 5.41.21.2510;}\viewkind4\uc1\pard \sa200\sl276\slmult1\lang9\f0\fs22\par Line 1: xxxx\par \b Line 2:\b0 yyyy\par } Elements are delimited by the \ (either initiator or prefix separator) of simple fields, or by the { (initiator) of complex fields. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 04/06/2013 14:47 Subject: [DFDL-WG] terminate by next field's initiator aka lengthKind="endAtStartOfNext" or something like that Sent by: dfdl-wg-bounces(a)ogf.org I know we omitted this from DFDL v1.0 (I am quite sure I advocated that position), and we're too late to add it back now, but while theoretically possible I had never seen this before, but now I have seen it and I'm wondering if it is more common than I originally thought. The situation is this. I have an element. It wants to be delimited in that it has an escape scheme, and it is delimited by something in the common-sense of the word, but the terminator is actually what one thinks of as the initiator of the next element. It comes up in Internet Message Format headers as one example: Reply-To: joe(a)foo.com Reply-To: <joe(a)foo.com> Reply-To: joe smith<joe(a)foo.com> Reply-To: "joe <Mr. XML> smith"<joe(a)foo.com> Reply-To: <> In the 3rd and fourth case, there is no terminator, just the required < which begins the next field. Modeling this whole reply-to construct requires a choice of several different elements which model the different formats. For example I see no way to model a format which accepts either line one or line 2 of the above without using a choice. That said, my real concern is with lines 3 and 4. The natural model for lines 3 and 4 (and perhaps 5) seems like it should be a display-name field followed by an email address field. The "<" really does not want to be used in some situations as the terminator of the prior field and in others as the initiator of the next field. That affects reuse of the validation regex's, etc. Right now the only way to model this is for the display name field to use a regex which re-invents the escape-scheme-like behavior of the optional quotation mark surround, and uses regex lookahead to sense the "<" when it appears unescaped, without consuming it. That's not too bad really, but I am curious what others have seen out there in the world of data that also has this idiom where a string is delimited by a unique structure at the beginning of the next element. Do we have collective knowledge of several more such formats, or have we all just seen this same IMF header example as the motivation. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0