- dfdl-wg - lists.ogf.org

Notes from 2007-09-12 call
by Mike Beckerle 19 Sep '07

19 Sep '07

Mike Beckerle, Alan Powell, Steve Hanson, Suman Kalia attended. Discussed these questions from Alan about expression language. 1. Accessing hidden values - it seems inconsistent to allow access to hidden values when xpath is used within the DFDL domain but not when used outside. 2. Where xpath is allowed in the schema - It is currently allowed in an arbitrary set of properties (initiator, terminator, separator, occurseparator, null, etc ). Why not allow it everywhere? Wr.t. (1) we decided this is correct. path expressions for dfdl properties can see hidden elements, path expressions in other places (e.g., schematron assertions) cannot. Wr.t (2) we decided that expressions should be allowed in principle everywhere for the value of any property; however, there may be exceptions for certain properties. Particularly, it seems some enum-valued properties are unlikely to ever want to be expressions. Example: dfdl:representation. However, it was also pointed out that once we put selectors back into the language you can interleave multiple formats in the same schema, and for any enumerated property you could just have one selector-chosen format for each possible value of the enumerated property. The reason we don't want a blanket statement that you can have expressions anywhere you need a property value is that there is some potential that this makes implementations unnecessarily complex due to the excess flexibility. Digression: (This added by MikeB - was not part of the call today.) Consider dfdl:byteOrder=" if (../../x = 'B') then 'bigEndian' else if (../../x='L') then 'littleEndian' else 'I don't know' }" DFDL implementations must be prepared to cope with recieving "I don't know" as the proposed value for the byteOrder. This is a schema definition error, but it is happening at run time so becomes a processing error. The only way to rule this out is to treat enumerated property values not as strings but as an enum type and force the expressions that compute them to return an enum type, not a string. This is a kind of type inference I had hoped implementations would not need. Selectors have the advantage of being statically verifiable. i.e., each selected format is known to use a value of the enum that is valid or a diagnostic could be issued by the DFDL processor. If we allow an arbitrary expression to return the value of an enumerated property then it presumably could also return a nonsense value: We discussed proposals circulated by MikeB: Here's an update to the first one. We decided sequences shouldn't be another way to carry opaque data. Easy and conservative way to fix this is to require the length of an empty sequence to be zero. Second proposal to eliminate hexBinary and base64Binary was discussed lightly. It was suggested that one could have both, and that would make it easy to explain what the hexBinary type is, because it is a shorthand for a string with encoding="hex", and similarly for base64Binary. We did not resolve this issue on the call. Finally, we discussed regular expression features for DFDL. There does appear to be need for regexp features to support parsing data which is delimited by changing data content. E.g. consider "12345Mike Beckerle". and a two-element sequence. One is a number which continues until the first non-digit character. The other is a string which begins with a non-digit character. Regexp length appears to be a good way to handle this kind of thing. Alan Powell has the action item to talk with the IBM internal TX product group. They have a speculative parser and so have fewer regular-expression features in their language. We want to understand how they deal with the header, body[], trailer use case. This case is where the data is lines of text, the header is the first line, the trailer is the last line, the body records are everything in between and there's no content that can be used to distinguish the record types. This is handled in some format-description systems with regexp features. In TX this is handled by speculative parsing and we want to understand how this comes out and if it is preferable to adding regexp features. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 2

DFDL core draft 025 released
by Mike Beckerle 14 Sep '07

14 Sep '07

I've released internal draft 025 which you can find here: https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… I didn't get everything into it that I'd hoped. But I promised a number of people an improved draft by COB today, and this one is a big improvement over 019. Here's things I wanted to get in but couldn't: - we've settled the opaque data discussion. Someone should figure out where in 025 this treatment belongs and how the text of the proposal (circulated in email earlier this week) should be redone inside the core spec. - the discussion of variables is now in back matter which is gradually being phased out. (too formal/mathematical suitable for implementors only) But somebody needs to read the discussion of variables there and create a more user-centric discussion of the same. - null/default/optional - this is the focus of my work right now. Should have a proposal out next week. Next week on our wednesday call I want to go over the new organizational structure of this draft. Note: change tracking is turned ON in this draft. Please DO NOT send me comments on this draft in any form except using MS-WORD COMMENTS in the document. I really really don't want to piece through emails that say "on page xxx section yyy I think ....". It is much easier if people use MS-WORD Comments feature and then we can easily just merge stuff. Thanks Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

Conference
by Simon Parker 12 Sep '07

12 Sep '07

Sorry, an urgent assignment has come up and I can't attend today's conference. See you next week, Simon Simon Parker Software Consultant PolarLake 80 Harcourt Street Dublin 2, Ireland Phone +353 (1) 449-1075 Fax +353 (1) 449-1011 Web http://www.polarlake.com/ The information transmitted in this email is intended for the addressee only and may contain confidential and/or privileged material. Any review, retransmission, dissemination, reliance upon or other use of this information by persons or entities other than the addressee is prohibited. If you think, for any reason, that this message may have been addressed to you in error, we would ask you to notify the sender immediately by return email and delete the material. PolarLake Limited | Registered in Dublin, Ireland | Number 357324 | Registered office as above.

1 0

WG call 2007-09-12
by Mike Beckerle 12 Sep '07

12 Sep '07

I have two things that we can discuss: The above is v3. It differs in that sequences with length were added. There's an issue in comments of the doc we can discuss. Also: a new related proposal below - I can overview this, but we don't really need people to weigh in with opinions/decisions until more of a chance to think it through. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

v4 on length/occurs simplification proposal
by Mike Beckerle 06 Sep '07

06 Sep '07

How's lengthKind="maximum" sound for the "endOfData" fitting into a box thing? Other possibilities in this doc: Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 2

Fw: Minutes from 2007-08-08 Call - comments from Steve
by Suman Kalia 30 Aug '07

30 Aug '07

Comments in Green Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 08/16/2007 12:46 PM ----- "Simon Parker" <simon.parker(a)polarlake.com> Sent by: dfdl-wg-bounces(a)ogf.org 08/16/2007 12:28 PM To <dfdl-wg(a)ogf.org> cc Subject Re: [DFDL-WG] Minutes from 2007-08-08 Call - comments from Steve Responses embedded below Simon From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: 15 August 2007 12:23 To: Mike Beckerle Cc: dfdl-wg(a)ogf.org; dfdl-wg-bounces(a)ogf.org; Simon Parker Subject: [DFDL-WG] Minutes from 2007-08-08 Call - comments from Steve I've spent today catching up with the recent DFDL spec discussions around Simon's comments to v0.19. Some comments of my own on the content of these and previous call minutes. - General principle: The eventual consumers of DFDL will be users the majority of whom will not be data modelling experts, that's certainly the experience at IBM. Most see data modelling as a black art and find it difficult. I think that an over-reliance on hidden elements is not going to go down well. I would err on the side of caution here, and only if we are convinced a property will be very rarely used should we remove it and replace by a hidden element. [Simon] Accepted, providing we can specify everything. Ideally we'll publish a rigorous, orthogonal language and a convenient, intuitive library with controlled redundancy. - Leading/Trailing Skip Bytes is a property intended to handle the byte skipping added by compilers, over and above simple byte alignment rules. The formulae for setting the values is beyond the ken of users to set manually, it would invariably be done using an automated COBOL -> DFDL translator, etc. I would not be too troubled if that went 'hidden'. SKK -- For complex scenarios ( e.g. occurs depending on elements in COBOL), different compilers follow quite complicated algorithm to add slack bytes at the end or front of structures to properly align array elements and current set of technologies for COBOL->DFDL may not be able to extract this information from compilers/interpreters as they may not be exposed through well defined interfaces in which case the user may have to manually adjust the values for Leading/Trailing skip counts. I would not vote for this attributes to be hidden, they are certainly advanced properties used occasionally to cater to such complex scenarios. 'finalTerminatorCanBeMissing' property. The rules for interpreting what trailing markup actually means are complex and properties like this will almost certainly be needed. SKK -- I tend to agree with Steve (Aside: For Mike's second example, though, where data of max length n is terminated by markup only if actual length < n, wouldn't that be better expressed using a regular expression? finalTerminatorCanBeMissing is too general, and could lead the parser to validly parse data where the terminator was accidentally omitted). - Infix/prefix/postfix separators. I believe this should be retained. It's in IBM WTX (Mercator) and I frequently have to apologise for the absence of postfix in IBM MRM. When a user sees (eg) x,y,z it's easier for him to comprehend that the comma after z is a postfix separator rather than the terminator of the parent group. - Simon had a comment on the removal of 'applies' which I haven't seen discussed ("I find this cumbersome. I suggest this alternative: drop ‘applies’ and ‘dfdl:format’, insist on ‘dfdl:sequence’ and friends instead, and add local variants like ‘dfdl:sequenceLocal’. For attribute shorthand, add boolean attributes with the same name: sequenceLocal=”true” (optional, default false)."). SKK - I am not comfortable with using names like sequenceLocal, if we go with this you will quickly compound the problem with allLocal, choiceLocal etc..The intent here is to specify the scope and it is best expressed through one generic property with different set of enumeration values identifying scope. I don't follow, the use of 'applies' is orthogonal to whether you use dfdl:format or one of the specific elements such as dfdl:sequence. [Simon] You're right, the ideas should be discussed separately. My hasty comment throws it all in together. 1 Replace this: <dfdl:format applies="hereOnly"> with this: <dfdl:formatLocal> Why? Because 'applies' is a metaproperty that doesn't describe the representation, and should be prominent. Also, for brevity. 2 Replace this: <dfdl:format> with one of these: <dfdl:element> <dfdl:sequence> <dfdl:complexType>... Why? For ease of validation and interpretation, to make mistakes more obvious to human readers, and to support more rigorous specification of the relationship between properties and xsd constructs. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 14/08/2007 14:23 To dfdl-wg(a)ogf.org, "Simon Parker" <simon.parker(a)polarlake.com> cc Subject Re: [DFDL-WG] Minutes from 2007-08-08 Call I forgot to clarify Simon's question on sp165. This was the 'finalTerminatorCanBeMissing" property. We considered the comment that this might be unnecessary. Use case: file of text format. Each "record" in the file is terminated by a CRLF so sez the user. At the top level this file contains an array of these records. The file might or might not have a CRLF at the end of the file because human beings might have edited the file with a text editor, and either inserted or neglected to insert this final CRLF. We want the file format to be legal with or without the final CRLF; however, all prior CRLFs in the file must be present. So how to express this: 1) CRLF is a terminator of the record 2) CRLF is an occursSeparator of the enclosing array, records have no terminator. We enclose the array in a sequence group where the array is followed by a hidden "optional" (minOccurs=0 max=1) element of fixed="CRLF" string value. Choice (1) requires that we have finalTerminatorCanBeMissing Choice (2) is just modeling the behavior that is required directly via hidden elements. This is tantamount to saying that this keyword is not worth having because there is a way to model it already. This is true of many keywords. If we deem this one too obscure, then we need to revisit many others. (Leading/Trailing Skip Bytes is a good example. Trivially represented by a hidden element). What are our criteria for inclusion? Up until now our criteria have been to include things that existing systems already have found a need for. However, existing systems don't have hidden field capability. Note that this same missing final terminator issue can come up not only with End-of-data, but with any bounded size structure. E.g., suppose we say that an array has occursUnits="bytes" and occursPath="874". Then it is 874 bytes long. The array elements can be terminated by a particular data. E.g., semicolon. For the same reasons as the CRLF example above, we want to be able to tolerate a missing final semicolon before the end of the 874 bytes. In effect the byte-length-limit creates an implicit "end-of-data" for a sub-stream consisting of just those bytes. Conclusion: finalTerminatorCanBeMissing seems to be useful enough and comes up often enough that I think the keyword is worthwhile. Implication: we should create a list of keywords or enumerated values for properties that we think are in the grey area where perhaps we want to drop them. Here's some candidates: byteOrderMarkPolicy, leading/trailingSkipBytes. Both these can be modeled readily as hidden elements. There are probably others. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Mike Beckerle/Worcester/IBM 08/14/2007 08:40 AM To "Simon Parker" <simon.parker(a)polarlake.com> cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Minutes from 2007-08-08 CallLink In conjunction with the annotated document these notes are clear, except for 'sp165'. Perhaps someone will recapitulate the discussion briefly at Wednesday's conference. I think only three annotations remain: sp167 Absent and missing (expanded discussion on the wiki already) This will be a major topic on a call. sp172 separatorType="infix" I'm happy to drop this strange stuff about separatorType=prefix or postfix and just say separator means infix. However, I would note that at least two major integration products (IBM WebSphere Transformation Extender - formerly Mercator, and Microsoft Biztalk, have this concept, so we may end up putting it back in. Presumably MS copied the earlier Mercator style, or both got it from common requirements in some EDI standard. sp173 defaultWhenMissing (expanded discussion on the wiki already) Same topic as sp167 above. Will have a call topic to discuss. I've added another contribution to the wiki discussion on 'require'. This seems to be at resolution I think, which is that we can express this using assertions. The general style of using DFDL to describe what fixed-data syntactic constructs look like is a good one. However, I've amended the Wiki thread on this with a further issue for group consideration. See bottom of page: https://forge.gridforum.org/sf/wiki/do/viewPage/projects.dfdl-wg/wiki/Requi… The 'length and occurs' proposal is an improvement, though I still have reservations to discuss; likewise the 'opaque data' proposal. For a call, this week or soon. I will send out an agenda. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 "Simon Parker" <simon.parker(a)polarlake.com> Sent by: dfdl-wg-bounces(a)ogf.org 08/13/2007 10:56 AM To <dfdl-wg(a)ogf.org> cc Subject Re: [DFDL-WG] Minutes from 2007-08-08 Call In conjunction with the annotated document these notes are clear, except for 'sp165'. Perhaps someone will recapitulate the discussion briefly at Wednesday's conference. I think only three annotations remain: sp167 Absent and missing (expanded discussion on the wiki already) sp172 separatorType="infix" sp173 defaultWhenMissing (expanded discussion on the wiki already) I've added another contribution to the wiki discussion on 'require'. The 'length and occurs' proposal is an improvement, though I still have reservations to discuss; likewise the 'opaque data' proposal. Regards, Simon From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Mike Beckerle Sent: 08 August 2007 18:00 To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] Minutes from 2007-08-08 Call MikeB, Geoff Judd, Alan Powell attended. Continued through SP's comments. sp37 - got it. sp45 - agree. This whole part to be rewritten. sp115 - ok. strict and "lax" as enums. No built-in default - we never use defaults in the processor itself. Only in the predefined formats. sp118 - ok sp123 - Proposal to simplify length, lengthKind, lengthUnits, and also occursKind, occursPath, occursPathUnits needed. (along the lines of byteCount, itemCount, length='delimited' enum, etc.) sp154 - Need specific proposal to eliminate hexBinary and use what for opaque (consider also string with encoding='bytes'. ) Or introduce a dfdl:byteString type or dfdl:opaque type. (derived type - just a standard name). sp158 - see sp123 sp165 - needed to have composition property for enclosing groups and or end-of-data. Regexp doesn't fix this. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

4 5

MikeB cannot attend OGF21 in Seattle
by Mike Beckerle 29 Aug '07

29 Aug '07

Our WG session is Monday Oct 15th at 11am. I have an unmovable conflict that day, and there's no way I could travel on Monday to be in seattle by tuesday. So anyone else? ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

Proposal to simlify null/optional/default handling
by Mike Beckerle 29 Aug '07

29 Aug '07

This proposal is a radical simplification of this stuff. Really collapses it massively. Please give it some thought. I did go through this with Geoff Judd who did the original research into the complexity of this area. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

DFDL call today at Noon US.EDT.
by Mike Beckerle 29 Aug '07

29 Aug '07

We have a WG call today. Proposed agenda: - continue length/occurs simplification (hopefully finish this - new doc just sent out before this email) - discuss null/default/optional handling proposal - plan for upcoming OGF meeting in Oct. in Seattle (?)

1 0

Minutes of DFDL working group call 23rd Aug 2007
by Steve Hanson 22 Aug '07

22 Aug '07

Minutes Attendees: Geoff Judd, Steve Hanson, Simon Parker a) Discussed Mike's length proposal #2: Length: Plan A prefered - Noted that decoupling length and lengthUnits allows each to be specified at different scopes (pros and cons to this) Occurs: Plan C discussed and preferred in principle to Plans A and B. However: - What happens when we have both an occurs separator and a stop value, eg, a,b,c,d,, - it looks like occursKind can't handle this - Currently dfdl:length can not be used to define the length of an enclosing group, only a simple value - Not clear therefore that the 'Number of occurrences defined by size of bucket' is supported? b) Discussed Mike's opaque proposal #2 - Looks good - Need a description of modelling an array of bits where overall length not a multiple of 8 bytes - no simple type for a bit - Therefore need to state how this is supported using hexBinary or base64Binary (pad with 0's on parse, lose trailing 0's on write) Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0