March 2014 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording
by Andrew Edwards 15 Apr '14

15 Apr '14

Steve (et al) - Resending as the last one bounced. I'll usurp Tim and respond :) Currently the IBM implementation insists on using a fixed-length encoding and returns an "unsupported" error message for a variable width encoding. With a fixed width encoding, we "do the maths" using the bytes-per-character and the bytes written by this complex element. HTH, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Steve Hanson/UK/IBM 24/03/2014 14:52 To "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, cc Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Note errata 3.9, my bolding: "3.9. Section 12.3.5, 7.3.1, 7.3.2. The spec originally allows lengthKind ‘pattern’ to be used when the representation of the current element, or of a child element, is binary, but imposes restrictions on the encoding that can be in force. Clarify that the encoding property must be defined for the element (else schema definition error), and that a decoding processing error is possible if the match of the regex encounters data that does not decode in that encoding, dependent on the setting of encodingErrorPolicy. Remove section 12.3.5.1. Same clarifications needed for testKind ”pattern” property for asserts and discriminators. For consistency, the restriction that a complex element of specified length and lengthUnits ‘characters’ must have children that are all text and that have the same encoding as the complex element, is dropped." That's the restriction that I was referring to in my comment below. I can see why it was dropped - basically the parser now just tries to decode n characters using the complex element's encoding (and encodingErrorPolicy). We could apply the same principle for dfdl:valueLength & dfdl:contentLength - you build the stream from the bottom up, and then decode it using the complex element's encoding (and encodingErrorPolicy ?) to get the length in characters. Note that's how unparsing for lengthKind 'prefixed' with lengthUnits 'characters' would work as well - the spec just says "For a complex element, the length is that of the ComplexContent region" which is not sufficient (12.3.4). Similar deal for lengthKind 'explicit' - in order to know the size in chars of ElementUnused the unparser needs to know the size in chars of the data first (12.3.7.3). (Of course, for a fixed width encoding, you don't need to decode, you can just do the maths, but for the general case you need to decode. Also just doing the maths does not take encodingErrorPolicy into account). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 24/03/2014 12:55 Subject: Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Mike 23.5.3.1. Value length is only a function of the dfdl:encoding property if the element has a text representation. Not sure this needs to be (re)stated here. 23.5.3.1. "The value length is computed from the DFDL infoset value, ignoring the dfdl:length or dfdl:textOutputMinLength property. Other DFDL properties which affect the length of a text or binary representation are respected, it is only an explicit length which is ignored." Last sentence is too imprecise - should be phrased in terms of the grammar. 23.5.3.1. "If the second argument is 'characters' then the element must have text representation and it is a schema definition error otherwise". Yes but only for a simple type, so should be qualified. 23.5.3.1. "If the second argument, giving the length units, is 'characters', then recursively, this complex type element must have text representation throughout all its contained elements and framing, all of which must also use a uniform character set encoding." I can't see that restriction elsewhere in the spec when it talks about length of ComplexContent and lengthUnits 'characters' - I was expecting it to be in section 12.3.4 or 12.3.7.3 which face the same issue - but it isn't. Did we decide not to have this restriction? Without such a restriction, how does the unparser come up with a meaningful length (unless it re-parses)? (Tim - what does IBM DFDL do here?) What about delimiters and padding of children that use %#r entities? 23.5.3.2. The points in 23.5.3.1 about escape characters, length as a function of encoding, and bottom up for complex elements, apply equally to 23.5.3.2. It might be easier just to say in 23.5.3.2 that dfdl:contentLength for complex elements is same as dfdl:valueLength, and for simple elements differs only by the additional inclusion of LeftPadding and RightPadOrFill regions. Also noted in passing: Specified length - An item has specified length when dfdl:lengthKind is "implicit", "explicit", or "prefixed". should be Specified length - An element has specified length when dfdl:lengthKind is "implicit" (simple type only), "explicit", or "prefixed". Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 20/03/2014 17:21 Subject: [DFDL-WG] Action 242 - valueLength and contentLength function wording Sent by: dfdl-wg-bounces(a)ogf.org See attached doc which is proposed revisions to section 23.5.3 Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy [attachment "Action-252-DFDL-Functions-23.5.3.docx" deleted by Andrew Edwards/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 3

Re: [DFDL-WG] newbie question
by Tim Kimber 28 Mar '14

28 Mar '14

I didn't see the requirement for a pattern facet in your problem description. Feel free to use a pattern facet if you require one. If you want to perform cross-field validation then you should probably use DFDL asserts ( not discriminators ) for that. The best place to put the asserts is on the parent element, addressing ./A and ./C. You could use the exceptionType attribute on the assert (if the DFDL processor supports it) to make it throw a recoverable error instead of a processing error. That would enable invalid data to be parsed, and you would still be notified of any ( and all ) violations of the validation rules. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "mfcplus(a)yahoo.com" <mfcplus(a)yahoo.com> To: Tim Kimber/UK/IBM@IBMGB, Date: 28/03/2014 13:21 Subject: Re: [DFDL-WG] newbie question so if i am not to use a pattern, how do i validate against the alphanumeric part? also i have a conditional in the message type, should i use discriminator with xpath rxpression for that? elemA /elemB/elemC CRLF elemC/elemD/elemE CRLF elemF/elemG/elemH CRLF for the above strcuture, i have a requirement that elemC must not exist if elemA is not there, and must exist if elemA is there. sequenceC must not exist if sequenceA is not there, sequenceC must exist if sequenceA is there. just wondering how do i do that with dfdl discriminator. mant thanks again Sent from Yahoo Mail on Android. From: Tim Kimber <KIMBERT(a)uk.ibm.com>; To: mfcplus(a)yahoo.com <mfcplus(a)yahoo.com>; Cc: <dfdl-wg(a)ogf.org>; Subject: Re: [DFDL-WG] newbie question Sent: Fri, Mar 28, 2014 11:34:12 AM The key point here is that the data format is delimited, and there is a requirement to validate the parsed info set against some rules ( length constraints, in this case). I would - set terminator="/" - set lengthKind='delimited' - use xs:minLength and xs:maxLength facets to check the length ( or use a pattern facet if you must, but min/maxLength facets will use less CPU ) - switch on validation in the DFDL parser so that the length of the delimited value gets checked against the facets in the schema. Other possibilities exist ( e.g. DFDL assert to check the length ) but this is the most efficient. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "mfcplus(a)yahoo.com" <mfcplus(a)yahoo.com> To: Tim Kimber/UK/IBM@IBMGB, Date: 28/03/2014 11:25 Subject: Re: [DFDL-WG] newbie question a field that is 3-5 chars long alphanumeric, basically a \w{3,5}, it should give out parsing error if it doesnt fit by the pattern, no initiator terminator is / Sent from Yahoo Mail on Android From: Tim Kimber <KIMBERT(a)uk.ibm.com>; To: <dfdl-wg(a)ogf.org>; Subject: Re: [DFDL-WG] newbie question Sent: Fri, Mar 28, 2014 10:19:47 AM It depends on the data format. Normally a variable length field is terminated by a delimiter, so a data pattern/regex is not required. Can you describe the format in a little more detail? regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Bing Lu <mfcplus(a)yahoo.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 28/03/2014 03:35 Subject: [DFDL-WG] newbie question Sent by: dfdl-wg-bounces(a)ogf.org what's the best dfdl approache to describe a variable length alpha numeric field? should i use simpletype restriction pattern, or use dfdl:assert testPattern? thanks in advance-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Re: [DFDL-WG] newbie question
by Tim Kimber 28 Mar '14

28 Mar '14

The key point here is that the data format is delimited, and there is a requirement to validate the parsed info set against some rules ( length constraints, in this case). I would - set terminator="/" - set lengthKind='delimited' - use xs:minLength and xs:maxLength facets to check the length ( or use a pattern facet if you must, but min/maxLength facets will use less CPU ) - switch on validation in the DFDL parser so that the length of the delimited value gets checked against the facets in the schema. Other possibilities exist ( e.g. DFDL assert to check the length ) but this is the most efficient. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: "mfcplus(a)yahoo.com" <mfcplus(a)yahoo.com> To: Tim Kimber/UK/IBM@IBMGB, Date: 28/03/2014 11:25 Subject: Re: [DFDL-WG] newbie question a field that is 3-5 chars long alphanumeric, basically a \w{3,5}, it should give out parsing error if it doesnt fit by the pattern, no initiator terminator is / Sent from Yahoo Mail on Android From: Tim Kimber <KIMBERT(a)uk.ibm.com>; To: <dfdl-wg(a)ogf.org>; Subject: Re: [DFDL-WG] newbie question Sent: Fri, Mar 28, 2014 10:19:47 AM It depends on the data format. Normally a variable length field is terminated by a delimiter, so a data pattern/regex is not required. Can you describe the format in a little more detail? regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Bing Lu <mfcplus(a)yahoo.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 28/03/2014 03:35 Subject: [DFDL-WG] newbie question Sent by: dfdl-wg-bounces(a)ogf.org what's the best dfdl approache to describe a variable length alpha numeric field? should i use simpletype restriction pattern, or use dfdl:assert testPattern? thanks in advance-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

newbie question
by Bing Lu 28 Mar '14

28 Mar '14

what's the best dfdl approache to describe a variable length alpha numeric field? should i use simpletype restriction pattern, or use dfdl:assert testPattern? thanks in advance

2 1

alignment questions
by Mike Beckerle 26 Mar '14

26 Mar '14

Are leadingSkip always byte aligned, or are they N * 8 bits added on. Similarly trailingSkip E.g., <element name="x" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:lengthUnits="bits" dfdl:trailingSkip="10" dfdl:alignmentUnits="bytes" dfdl:alignment="1"/> So, after this element am I at bit 81 or bit 88? In other words, does the trailingSkip region get aligned per size of alignment unit, or does alignment really only happen at the beginning of an element. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call Minutes 2014-03-25
by Steve Hanson 26 Mar '14

26 Mar '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13220?download= Next call is Fri 11th April @ 3pm UK. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: DFDL Working Group Call (11 Apr 15:00 GDT in DE3S09)
by Steve Hanson 26 Mar '14

26 Mar '14

Rescheduled: DFDL Working Group Call 11/04/2014 - (Repeats) Location: DE3S09 Steve Hanson has rescheduled this meeting. You have not yet responded. Repeats This entry repeats Required: jcranford(a)mitre.org, mbeckerle(a)tresys.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: Alex Wood1/UK/IBM@IBMGB, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, jonathan.w.cranford(a)gmail.com, Mark Frost/UK/IBM, mbeckerle.dfdl(a)gmail.com, Stephanie Fetzer/Charlotte/IBM@IBMUS OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

Cancelled: DFDL Working Group Call
by Steve Hanson 26 Mar '14

26 Mar '14

Cancelled: DFDL Working Group Call 01/04/2014 - (Repeats) Location: DE3S09 Steve Hanson has cancelled this meeting. Your calendar will be updated to reflect this change. Repeats This entry repeats Required: jcranford(a)mitre.org, mbeckerle(a)tresys.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: Alex Wood1/UK/IBM@IBMGB, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, jonathan.w.cranford(a)gmail.com, Mark Frost/UK/IBM, mbeckerle.dfdl(a)gmail.com, Stephanie Fetzer/Charlotte/IBM@IBMUS

1 0

lengthUnits bits not allowed for complex type - Unnecessary restriction on length units?
by Mike Beckerle 26 Mar '14

26 Mar '14

In section 12.3.7.3 we have this sentence: "The dfdl:lengthUnits may be 'bytes' or 'characters' and it is a schema definition error otherwise. " Does anyone recall why we have this? I have data formats which are bit oriented and there are complex types which would naturally not be a multiple of 8 bytes long, e.g., 1 bit field, 3 bit field, 10 bit field, 6 bit field = 20 bits. I can't think of any reason for this restriction other than to explain how fillByte is used to fill in unused bits. But I think we can say that any unused bits are filled in with bits from the fillByte, and we don't have to be specific about which bits from the fillByte. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 2