- dfdl-wg - lists.ogf.org

Proposal to simlify null/optional/default handling
by Mike Beckerle 29 Aug '07

29 Aug '07

This proposal is a radical simplification of this stuff. Really collapses it massively. Please give it some thought. I did go through this with Geoff Judd who did the original research into the complexity of this area. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

DFDL call today at Noon US.EDT.
by Mike Beckerle 29 Aug '07

29 Aug '07

We have a WG call today. Proposed agenda: - continue length/occurs simplification (hopefully finish this - new doc just sent out before this email) - discuss null/default/optional handling proposal - plan for upcoming OGF meeting in Oct. in Seattle (?)

1 0

Minutes of DFDL working group call 23rd Aug 2007
by Steve Hanson 22 Aug '07

22 Aug '07

Minutes Attendees: Geoff Judd, Steve Hanson, Simon Parker a) Discussed Mike's length proposal #2: Length: Plan A prefered - Noted that decoupling length and lengthUnits allows each to be specified at different scopes (pros and cons to this) Occurs: Plan C discussed and preferred in principle to Plans A and B. However: - What happens when we have both an occurs separator and a stop value, eg, a,b,c,d,, - it looks like occursKind can't handle this - Currently dfdl:length can not be used to define the length of an enclosing group, only a simple value - Not clear therefore that the 'Number of occurrences defined by size of bucket' is supported? b) Discussed Mike's opaque proposal #2 - Looks good - Need a description of modelling an array of bits where overall length not a multiple of 8 bytes - no simple type for a bit - Therefore need to state how this is supported using hexBinary or base64Binary (pad with 0's on parse, lose trailing 0's on write) Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Minutes from 2007-08-08 Call
by Mike Beckerle 16 Aug '07

16 Aug '07

I forgot to clarify Simon's question on sp165. This was the 'finalTerminatorCanBeMissing" property. We considered the comment that this might be unnecessary. Use case: file of text format. Each "record" in the file is terminated by a CRLF so sez the user. At the top level this file contains an array of these records. The file might or might not have a CRLF at the end of the file because human beings might have edited the file with a text editor, and either inserted or neglected to insert this final CRLF. We want the file format to be legal with or without the final CRLF; however, all prior CRLFs in the file must be present. So how to express this: 1) CRLF is a terminator of the record 2) CRLF is an occursSeparator of the enclosing array, records have no terminator. We enclose the array in a sequence group where the array is followed by a hidden "optional" (minOccurs=0 max=1) element of fixed="CRLF" string value. Choice (1) requires that we have finalTerminatorCanBeMissing Choice (2) is just modeling the behavior that is required directly via hidden elements. This is tantamount to saying that this keyword is not worth having because there is a way to model it already. This is true of many keywords. If we deem this one too obscure, then we need to revisit many others. (Leading/Trailing Skip Bytes is a good example. Trivially represented by a hidden element). What are our criteria for inclusion? Up until now our criteria have been to include things that existing systems already have found a need for. However, existing systems don't have hidden field capability. Note that this same missing final terminator issue can come up not only with End-of-data, but with any bounded size structure. E.g., suppose we say that an array has occursUnits="bytes" and occursPath="874". Then it is 874 bytes long. The array elements can be terminated by a particular data. E.g., semicolon. For the same reasons as the CRLF example above, we want to be able to tolerate a missing final semicolon before the end of the 874 bytes. In effect the byte-length-limit creates an implicit "end-of-data" for a sub-stream consisting of just those bytes. Conclusion: finalTerminatorCanBeMissing seems to be useful enough and comes up often enough that I think the keyword is worthwhile. Implication: we should create a list of keywords or enumerated values for properties that we think are in the grey area where perhaps we want to drop them. Here's some candidates: byteOrderMarkPolicy, leading/trailingSkipBytes. Both these can be modeled readily as hidden elements. There are probably others. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Mike Beckerle/Worcester/IBM 08/14/2007 08:40 AM To "Simon Parker" <simon.parker(a)polarlake.com> cc dfdl-wg(a)ogf.org Subject Re: [DFDL-WG] Minutes from 2007-08-08 Call In conjunction with the annotated document these notes are clear, except for 'sp165'. Perhaps someone will recapitulate the discussion briefly at Wednesday's conference. I think only three annotations remain: sp167 Absent and missing (expanded discussion on the wiki already) This will be a major topic on a call. sp172 separatorType="infix" I'm happy to drop this strange stuff about separatorType=prefix or postfix and just say separator means infix. However, I would note that at least two major integration products (IBM WebSphere Transformation Extender - formerly Mercator, and Microsoft Biztalk, have this concept, so we may end up putting it back in. Presumably MS copied the earlier Mercator style, or both got it from common requirements in some EDI standard. sp173 defaultWhenMissing (expanded discussion on the wiki already) Same topic as sp167 above. Will have a call topic to discuss. I've added another contribution to the wiki discussion on 'require'. This seems to be at resolution I think, which is that we can express this using assertions. The general style of using DFDL to describe what fixed-data syntactic constructs look like is a good one. However, I've amended the Wiki thread on this with a further issue for group consideration. See bottom of page: https://forge.gridforum.org/sf/wiki/do/viewPage/projects.dfdl-wg/wiki/Requi… The 'length and occurs' proposal is an improvement, though I still have reservations to discuss; likewise the 'opaque data' proposal. For a call, this week or soon. I will send out an agenda. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 "Simon Parker" <simon.parker(a)polarlake.com> Sent by: dfdl-wg-bounces(a)ogf.org 08/13/2007 10:56 AM To <dfdl-wg(a)ogf.org> cc Subject Re: [DFDL-WG] Minutes from 2007-08-08 Call In conjunction with the annotated document these notes are clear, except for 'sp165'. Perhaps someone will recapitulate the discussion briefly at Wednesday's conference. I think only three annotations remain: sp167 Absent and missing (expanded discussion on the wiki already) sp172 separatorType="infix" sp173 defaultWhenMissing (expanded discussion on the wiki already) I've added another contribution to the wiki discussion on 'require'. The 'length and occurs' proposal is an improvement, though I still have reservations to discuss; likewise the 'opaque data' proposal. Regards, Simon From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Mike Beckerle Sent: 08 August 2007 18:00 To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] Minutes from 2007-08-08 Call MikeB, Geoff Judd, Alan Powell attended. Continued through SP's comments. sp37 - got it. sp45 - agree. This whole part to be rewritten. sp115 - ok. strict and "lax" as enums. No built-in default - we never use defaults in the processor itself. Only in the predefined formats. sp118 - ok sp123 - Proposal to simplify length, lengthKind, lengthUnits, and also occursKind, occursPath, occursPathUnits needed. (along the lines of byteCount, itemCount, length='delimited' enum, etc.) sp154 - Need specific proposal to eliminate hexBinary and use what for opaque (consider also string with encoding='bytes'. ) Or introduce a dfdl:byteString type or dfdl:opaque type. (derived type - just a standard name). sp158 - see sp123 sp165 - needed to have composition property for enclosing groups and or end-of-data. Regexp doesn't fix this. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

3 3

encoding="bytes" not needed? - was Re: Fw: Proposal - simplifying opaque types
by Mike Beckerle 16 Aug '07

16 Aug '07

Can anyone recall the use case driving the need for encoding="bytes". I've been trying to reconstruct it and I am not able to. It was supposedly for using string literal syntax to discuss padding fields and constant data found surrounding binary data. The spec now has % as the hex escape for string literals, and the bytes specified using it are not subject to character set translation. So, if you mean the data has an initiator that contains hex 6B followed by hex 2C, and you don't know nor care what characters those correspond to, then just write "%6B%2C". You don't have to deal with character set encodings at all. So, I can't see why we need encoding="bytes" anymore. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 Suman Kalia/Toronto/IBM@IBMCA 08/16/2007 08:54 AM To dfdl-wg(a)ogf.org cc Mike Beckerle/Worcester/IBM@IBMUS Subject Fw: [DFDL-WG] Proposal - simplifying opaque types Sorry I could not call-in yesterday as I had conflict with another meeting. Here are my comments on the proposal >> A byte string is described like this: >> <element name=”foo” type=”string” >> dfdl:encoding=”bytes” dfdl:lengthKind=”fixed” dfdl:length=”17”/> I am not sure if dfdl:encoding=byte is the correct way to model opaque blob of data which may not be a string but complex structure containing mixture of text and binary data and that structure blob may have alignment requirements within the memory layout. I believe the correct way to model blob is really the hexBinary or base64binary - these types do describe the encoding and by default you would specify the length in terms of number of bytes which could be described through length facet. The example HexBinary Type provided in section 7.2 When you model something as hexBinary, it is wrong to say the representation is text because data represented through hexBinary may not be string at all ( could be a complex structure) and the DFDL representations given below in the example would not be correct. The example provided by Simon was simple where the data just happened to be string data. >> Suppose we wanted to model this same data logically as a hexBinary 'blob'. >> <element name="d" type="hexBinary" dfdl:representation="text" dfdl:encoding="utf-8" dfdl:length="11" dfdl:lengthUnitKind="characters"/> In some respect having dfdl:encoding=byte and making it not sensitive to character encoding, we are simulating what hexBinary specifies already. Question : Why promote an additional DFDL specific construct when it could be catered through built-in schema type? PS: I am not against having dfdl:encoding=byte as specified in the spec but promoting it as a mechanism to model opaque binary data will not rest well with XML Schema folks. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 08/16/2007 08:49 AM ----- Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 08/10/2007 03:26 PM To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Proposal - simplifying opaque types Please review for discussion on a future call. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

Fw: Proposal - simplifying opaque types
by Suman Kalia 16 Aug '07

16 Aug '07

Sorry I could not call-in yesterday as I had conflict with another meeting. Here are my comments on the proposal >> A byte string is described like this: >> <element name=”foo” type=”string” >> dfdl:encoding=”bytes” dfdl:lengthKind=”fixed” dfdl:length=”17”/> I am not sure if dfdl:encoding=byte is the correct way to model opaque blob of data which may not be a string but complex structure containing mixture of text and binary data and that structure blob may have alignment requirements within the memory layout. I believe the correct way to model blob is really the hexBinary or base64binary - these types do describe the encoding and by default you would specify the length in terms of number of bytes which could be described through length facet. The example HexBinary Type provided in section 7.2 When you model something as hexBinary, it is wrong to say the representation is text because data represented through hexBinary may not be string at all ( could be a complex structure) and the DFDL representations given below in the example would not be correct. The example provided by Simon was simple where the data just happened to be string data. >> Suppose we wanted to model this same data logically as a hexBinary 'blob'. >> <element name="d" type="hexBinary" dfdl:representation="text" dfdl:encoding="utf-8" dfdl:length="11" dfdl:lengthUnitKind="characters"/> In some respect having dfdl:encoding=byte and making it not sensitive to character encoding, we are simulating what hexBinary specifies already. Question : Why promote an additional DFDL specific construct when it could be catered through built-in schema type? PS: I am not against having dfdl:encoding=byte as specified in the spec but promoting it as a mechanism to model opaque binary data will not rest well with XML Schema folks. Suman Kalia IBM Toronto Lab WebSphere Business Integration Application Connectivity Tools Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 08/16/2007 08:49 AM ----- Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 08/10/2007 03:26 PM To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Proposal - simplifying opaque types Please review for discussion on a future call. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

version 2 of length properties simplification proposal
by Mike Beckerle 16 Aug '07

16 Aug '07

For your consideration. My preferences are Plan A for scalars and Plan C for arrays. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

V2 of propsal on opaque types
by Mike Beckerle 15 Aug '07

15 Aug '07

1 0

proposal for changing length properties
by Mike Beckerle 14 Aug '07

14 Aug '07

Proposal to revise length and occurs specification properties. Please consider for discussion on a future call. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 1

Minutes from 2007-08-08 Call
by Mike Beckerle 14 Aug '07

14 Aug '07

MikeB, Geoff Judd, Alan Powell attended. Continued through SP's comments. sp37 - got it. sp45 - agree. This whole part to be rewritten. sp115 - ok. strict and "lax" as enums. No built-in default - we never use defaults in the processor itself. Only in the predefined formats. sp118 - ok sp123 - Proposal to simplify length, lengthKind, lengthUnits, and also occursKind, occursPath, occursPathUnits needed. (along the lines of byteCount, itemCount, length='delimited' enum, etc.) sp154 - Need specific proposal to eliminate hexBinary and use what for opaque (consider also string with encoding='bytes'. ) Or introduce a dfdl:byteString type or dfdl:opaque type. (derived type - just a standard name). sp158 - see sp123 sp165 - needed to have composition property for enclosing groups and or end-of-data. Regexp doesn't fix this. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 2