- dfdl-wg - lists.ogf.org

DFDL: Calculated Values section rewrite
by Mike Beckerle 15 May '08

15 May '08

My deliverable for the next draft (32) was to revise the calculated values section. I have rewritten it and the draft is attached. Of note: I have removed the troublesome "outputLengthCalc" property, as I no longer see a critical need for it. One of the examples computes the number of occurrences for a byte array, and I think that mechanism along with alignment is sufficient to handle the troublesome cases I was considering where size of padding had to be dynamically computed. The section now consists of a short table of definition, followed by illustrative examples, however, I think this is fine. The semantics of inputValueCalc and outputValueCalc aren't really that complex. They can be described in a paragraph. It is the motivation for them that is complicated, so I think examples in the spec, while sometimes considered problematic, are OK in this case.

3 5

DFDL: Time change for weekly calls?
by Mike Beckerle 15 May '08

15 May '08

I believe all active participants are either ET.US time, or UK time. We chose Noon ET.US because we used to have active participants from PT.US, but currently we don't, so I for one would like to free up Wednesday lunch time. I suggest moving back either 1 hour, or 90 minutes to either 11am ET.US 4pm UK, or 10:30am ET.US, 3:30pm UK. Any day of the week is acceptable to me. Keeping it Wednesday would be my preference. Comments on feasibility? .mikeb

2 1

DFDL: Agenda for DFDL WG call - 2008-05-14
by Mike Beckerle 14 May '08

14 May '08

Proposed agenda for today's call. Please provide additional topics. - Draft 32 status - Unparsing property precedence (left over from agenda of last meeting) - plans for meetings in Hursley next week o topics and prep needed o logistics - Mike to go to what building, and ask for whom, etc.

1 1

DFDL: Minutes from OGF WG call, 30 Apr 2008
by Ian W Parkinson 14 May '08

14 May '08

Hi all, I've just noticed - I didn't distribute the minutes from last week's WG call. Apologies for the delay. Cheers, Ian Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 16:00 GMT, 30 Apr 2008 Attendees Mike Beckerle (Oco) Steve Hanson (IBM) Ian Parkinson (IBM) Alan Powell (IBM) 1. F2F meeting at IBM Hursley Mike has confirmed that he will travel to IBM Hursley to further design discussions with the IBM team, later in May. 2. Review of specification line items The meeting began a review of the outstanding work required for specification draft 32, but this was not completed during the meeting as a number of technical discussions were triggered. valueCalc Mike has provided material to Alan. No further work is anticipated on this item. nulls/defaults/optionals There has been some discussion on nilIndicatorPath and forward references, and this discussion needs to continue. The rest of this section is now complete. UML description of schema components complete Property Precedence This item is currently marked complete; however it has sparked some discussion. On unparse, if the infoset contains less than the required number of occurances of some field, then the unparser may be expected to output sufficient instances of the default value. We need to craft a policy to determine exactly how this achieved; for example, do we use the xsd:minOccurs value or do we try to match the result of the dfdl:occurs expression? Mike proposed that if an occurs expression is present, it must evaluate to a valid number of occurances. Steve suggested that we disambiguate dfdl:occurs from xsd:minOccurs and xsd:maxOccurs by renaming the property dfdl:occursPath - however, this property can be an expression involving calculations, not just a simple path. A number of simplifications have been proposed, and these require further discussion. Alan noted that an XPath expression has no way to access a field's default value, and suggested introducing a function to handle this. Also, we may need to define count() such that it returns the number of instances of an element after any necessary padding and introduce a different way to obtain the number of instances found in the infoset. Stop values and terminators are not considered part of the logical data. Steve will update the diagram. Variable Markup Steve had previously proposed a scheme whereby properties, such as iniitator or terminator, could be described by another DFDL element or type definition. We could use a similar mechanism to handle strings with embedded complex structures. This subject will be tackled during the F2F. Regular Expressions for Lengths The meeting agreed to adopt the regular expression subset previously distributed by Alan. Regular expressions are ignored for unparsing, and are only used to determine the length of an element during parsing. Considering repeating elements, Steve contrasted the behaviour of regular expressions in DFDL (where the regular expression matches just a single instance) with that in IBM's MRM technology (where a regular expression matches the entire array), and the meeting discussed whether a regular expressions should include any necessary terminators. Meeting closed, 17:10 GMT Ian Parkinson WebSphere ESB Development Mail Point 211, Hursley Park, Hursley, Winchester, SO21 2JN, UK Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

DFDL: Unable to attend Call on 2008-05-07
by Mike Beckerle 06 May '08

06 May '08

1 0

Fw: Agenda for DFDL WG call - 30-04-2008
by Alan Powell 30 Apr '08

30 Apr '08

Proposed agenda for todays call - Draft 32 status - Nulls and Defaults. - confirm updates OK subject to nilIndicatorPath discussion below - Forward references nilIndicatorPath, occurs {expression}, length {expression}, expressions in general I would like to propose that we make nilIndicatorPath does not update indicator flag when nil so that it operates in the same way as occurs etc. - Regular expression language. - Unparsing property precedence Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Agenda for DFDL WG call - 2008-04-30
by Mike Beckerle 30 Apr '08

30 Apr '08

( Reprising last week's agenda since we didn't have quorum last week. ) Proposed agenda for todays call * Nulls and Defaults. - confirm updates OK subject to nilIndicatorPath discussion * Forward references nilIndicatorPath, occurs {expression}, length {expression}, expressions in general * Regular expression language. * Unparsing property precedence

1 0

DFDL: Minutes from OGF WG call, 14 Apr 2008
by Alan Powell 16 Apr '08

16 Apr '08

The meeting was abandoned as most participants were unavailable Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] DFDL Regular Expression proposal
by Alan Powell 16 Apr '08

16 Apr '08

Steve 1) A good summary of full XML schema regular expressions is here http://www.xmlschemareference.com/regularExpression.html Quick summary of subset XML Schema - Regular Expression - Meta Characters - all supported Normal Characters - no restrictions Single Character Escape Sequence - all supported Multiple Character Escape Sequences - Multiple Character Escape Sequences Description . Any character except '\n' (newline) and '\r' (return). \s Whitespace, specifically '' (space), '\t' (tab), '\n' (newline) and '\r' (return). \S Any character except those matched by '\s'. \i The first character in an XML identifier. Specifically, any letter, the character '_', or the character ':', See the XML Recommendation for the complex specification of a letter. This character represents a subset of letter that might appear in '\c'. \I Any character except those matched by '\i'. \c Any character that might appear in the built-in NMTOKEN datatype. See the XML Recommendation for the complex specification of a NameChar. \C Any character except those matched by '\c'. \d Any Decimal digit. A shortcut for '\p{Nd}'. \D Any character except those matched by '\d'. \w Any character that might appear in a word. A shortcut for '[#X0000-#x10FFFF]-[\p{P}\p{Z}\p{C}]' (all characters except the set of "punctuation", "separator", and "other" characters). \W Any character except those matched by '\w'. Character Categories Character Category Description Notes L Letter, Any Lu Letter, Uppercase Ll Letter, Lowercase Lt Letter, Titlecase Lm Letter, Modifier Lo Letter, Other L Letter, uppercase, lowercase, and titlecase letters (Lu, Ll, and Lt) Optional in The Unicode Standard; not supported by the Schema Recommendation. M Mark, Any Mn Mark, Nonspacing Mc Mark, Spacing Combining Me Mark, Enclosing N Number, Any Nd Number, Decimal Digit Nl Number, Letter No Number, Other P Punctuation, Any Pc Punctuation, Connector Pd Punctuation, Dash Ps Punctuation, Open Pe Punctuation, Close Pi Punctuation, Initial quote (may behave like Ps or Pe, depending on usage) Pf Punctuation, Final quote (may behave like Ps or Pe, depending on usage) Po Punctuation, Other S Symbol, Any Sm Symbol, Math Sc Symbol, Currency Sk Symbol, Modifier So Symbol, Other Z Separator, Any Zs Separator, Space Zl Separator, Line Zp Separator, Paragraph C Other, Any Cc Other, Control Cf Other, Format Cs Other, Surrogate (not supported by Schema Recommendation). Explicitly not supported by Schema Recommendation. Co Other, Private Use Cn Other, Not Assigned (no characters in the file have this property). Character Blocks - None supported (only really meaningful in unicode) XML Character References - supported 2) - 6) see below Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 From: Steve Hanson/UK/IBM To: Alan Powell/UK/IBM Cc: dfdl-wg(a)ogf.org, mbeckerle.dfdl(a)gmail.com Date: 09/04/2008 13:58 Subject: Re: [DFDL-WG] DFDL Regular Expression proposal Comments from Steve and Ian: 1) The subset proposed is basically lifted from the IBM MRM parser help. If I ever knew what the rationale for the subset was, I don't know it now. What features have we excluded? 2) IBM MRM parser has extended the xsd regular expression syntax to allow hexadecimal characters using the following syntax: \xNN hexadecimal digits in the range 0 to F MRM makes much wider use of regular expressions, as an alternative to speculative parsing, so I can see why MRM needed this (one concrete use case was for TLOG retail messages). Do we need to support this in DFDL? Is this documented? I think we need to allow hex characters 3) If we don't add the hex support, what are the use cases for using a dfdl:lengthPattern versus using an xsd pattern facet? It looks like pattern facets apply to all supported schema simple types, so not clear why dfdl:lengthPattern would be needed. The only use case I can think of is where we have length on a complex element or sequence or choice. If this is the only use case perhaps dfdl:lengthPattern should only be used in those cases? MRM allows this use. (It might also answer 2 as it allows embedded binary data to appear). Or is there a distinction between validation and parsing? The xsd:pattern operates on the logical contents, Lengthpattern operates on the physical contents including markup. 4) What is the behaviour on unparsing? I believe that MRM simply takes the value presented to it and outputs it (it does not attempt to match it against the pattern), so DFDL equivalent would be to outout the infoset value. Agree 5) For a repeating element, presumably we would consume only as match as the number of occurs dictates. Good question. I had assumed lengthPattern had the same semantics as length. 6) Should state explicitly that DFDL entity references are not allowed. The XML character reference is used instead &#xNN; Need to support DFDL entities to allow x00 Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Sent by: dfdl-wg-bounces(a)ogf.org 09/04/2008 01:34 Please respond to mbeckerle.dfdl(a)gmail.com To Alan Powell/UK/IBM@IBMGB, <dfdl-wg(a)ogf.org> cc Subject Re: [DFDL-WG] DFDL Regular Expression proposal Suggest add to ?lengthPattern? that the longest possible match is taken. This is the usual behavior for regular expressions, but it?s a clarification I?ve seen other places. From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Alan Powell Sent: Thursday, April 03, 2008 12:44 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] DFDL Regular Expression proposal Attached is the proposal for the regular expression syntax used to determine element length. Highlights Based on the XML Schema regular expression subset used by WebSphere Message Broker. Only applies to representation = text Uses LengthPattern property rather than decorated syntax to distinguish from literals and regular expressions as it is only used in one place, this avoids everywhere else having to escape the decoration character and we are running out of decoration characters. Assumes the pattern is converted to the data code page before matching against the data stream. Comments and improvements as soon as possible please. Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for DFDL WG call - 16-04-2008
by Alan Powell 16 Apr '08

16 Apr '08

Proposed agenda for todays call Nulls and Defaults. - confirm updates OK subject to nilIndicatorPath discussion below Forward references nilIndicatorPath, occurs {expression}, length {expression}, expressions in general Regular expression language. Unparsing property precedence Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0