- dfdl-wg - lists.ogf.org

Floating elements and unordered groups
by Tim Kimber 12 Jan '10

12 Jan '10

Hi all, I know this area of the specification was only recently resolved, and I think there may be an inconsistency in the v0.37 wording. Section 16, re: sequenceKind says: "The children of an unordered sequence must be xs:element." Section 16.5 Floating Elements says: "An ordered sequence of n element children with either n or n-1 of those children with dfdl:floating="true" is equivalent to an unordered sequence with the same n element children with dfdl:floating="false". A complex element with dfdl:floating="true" can have as its content model a sequence with elements that also have dfdl:floating="true". " Now suppose that, instead of N element children, there are N-1 floating element children + one non-floating group. This group will be equivalent to an unordered group with a non-element member. If the specification was intending to make life easy for implementers, then it should probably disallow groups in any non-ordered context, including when sequenceKind='ordered' and there is at least one floating component. But I think that would be too restrictive. I would be happy for the restriction to be lifted entirely. Given that unordered groups can have dfdl:initiated="false", it will sometimes be necessary to find the correct member by trial and error ( speculative parsing ) anyway. I don't think it's any more difficult to speculatively parse a group than to speculatively parse a complex element. If I've missed something, and it turns out that the restriction is useful, then we should a) tighten up the wording to say that if a group with N members has N or N-1 floating members, then it must be validated as if it was an unordered group. b) consider lifting the restriction in cases where dfdl:initiated="true" ( because it makes things so much easier for the DFDL processor ) regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

Re: [DFDL-WG] dfdl-wg Digest, Vol 41, Issue 8
by Tim Kimber 08 Jan '10

08 Jan '10

My understanding was different from what is quoted below. 'delimited' can always be terminated by the end of the data or the end of an enclosing known-length element. Without that rule, the simplest scenario involving an infix delimiter will not work correctly ( or will force the user to set lengthKind='endOfParent' on the final group member, which would be very unintuitive ). There is a difference between endOfParent and delimited though. Delimited is not allowed when representation='binary'. 'endOfParent' is. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: dfdl-wg-request(a)ogf.org To: dfdl-wg(a)ogf.org Date: 08/01/2010 18:02 Subject: dfdl-wg Digest, Vol 41, Issue 8 Send dfdl-wg mailing list submissions to dfdl-wg(a)ogf.org To subscribe or unsubscribe via the World Wide Web, visit http://www.ogf.org/mailman/listinfo/dfdl-wg or, via email, send a message with subject or body 'help' to dfdl-wg-request(a)ogf.org You can reach the person managing the list at dfdl-wg-owner(a)ogf.org When replying, please edit your Subject line so it is more specific than "Re: Contents of dfdl-wg digest..." Today's Topics: 1. dfdl:lentgKind= 'delimited' and 'endOfParent' (Alan Powell) ----- Message from Alan Powell <alan_powell(a)uk.ibm.com> on Fri, 8 Jan 2010 17:10:51 +0000 ----- To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] dfdl:lentgKind= 'delimited' and 'endOfParent' Since we extended the meaning of dfdl:lengthKind= 'delimited' to include 'delimited' means the item is delimited by the item’s terminator (if specified) or an enclosing construct’s separator or the end of the enclosing construct designated by its known length or its terminator. the only difference with dfdl:lentghKind='endOfParent' is that the latter includes the 'end of the data stream'. We should either Add 'end of data stream' to delimited and remove 'endOfParent' Make 'endOfParent' be specifically for only 'end of data stream' Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

dfdl:lentgKind= 'delimited' and 'endOfParent'
by Alan Powell 08 Jan '10

08 Jan '10

Since we extended the meaning of dfdl:lengthKind= 'delimited' to include 'delimited' means the item is delimited by the item?s terminator (if specified) or an enclosing construct?s separator or the end of the enclosing construct designated by its known length or its terminator. the only difference with dfdl:lentghKind='endOfParent' is that the latter includes the 'end of the data stream'. We should either Add 'end of data stream' to delimited and remove 'endOfParent' Make 'endOfParent' be specifically for only 'end of data stream' Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Minutes for OGF DFDL Working Group Call, January-06-2010
by Alan Powell 07 Jan '10

07 Jan '10

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, January-06-2010 Attendees Mike Beckerle (Oco) Steve Hanson (IBM) Alan Powell (IBM) Steve Marting (Progeny) Stephanie Fetzer (IBM) Suman Kalia (IBM) Apologies Peter Lambros (IBM) 1. 045 - Disciminators Bob Connolly was not available to discuss WTX. Stephanie discussed some of the WTX behaviour and agreed to provide more for the next call. We need a definition of the restricted positions where discriminators can be used and the semantics 2. LengthUnits = 'bits' Reviewed v4 of Mike's proposal and suggested editorial changes. If possible would like some words to describe how 'little-endian' is dealt with. Approved. 3. Selector support in V1 Implementation of selectors is proving more difficult than expected so IBM requested that selectors were deferred. In order to give reviewers and indiction of how the function might look in a future release and the opportunity to say if it is a higher priority the description will be move to an appendix named 'Features under consideration for a future version' 4. Go through remaining actions updated below 5 Draft 037 review Mike has reviewed and raised many comments -Significant issues: The spec is not consistent about how different types of errors are dealt with. Schema definition errors can occur at design and runtime and stop processing 'Grammar ambiguity' errors may not be statically detectable so are processing errors. - Use of entities in regular expressions. 'Special' entities such as WSP+ will not be allowed. 5A. AOB Steve H raised an issue with zoned number support. Customers have requested the ability control whether the sign is overpunched for positive numbers. Action raised 6 Review Schedule Activity Schedule Who Complete Action items - 18 Dec 2009 WG Complete Spec Write up work items ? 23 Dec 2009 AP Restructure and complete specification - 23 Dec 2009 AP Issue Draft 038 23 Dec 2009 WG review WG review 7 Dec ? 08 Jan 2010 WG Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 039 15 Jan 2010 Incorporate review comments 4 Jan - 29 Jan 2010 AP + Issue Draft 040 29 Jan 2010 Initial OGF Editor Review Initial Editor review 1 Feb - 1 Mar 2010 OGF Initial GFSG review 1 Feb - 1 Mar 2010 Issue Draft 041 1 Mar 2010 OGF Public Comment period (60 days) 1 Mar - 30 Apr 2010 OGF OGF 28 Munich 15-19 March 2010 Incorporate comments Incorporate comments 28 May 2010 Issue Draft 042 28 May 2010 Final OGF Editor Review Final Editor review June 2010 OGF final GFSG review June 2010 Issue Final specification 30 June 2010 Publish proposed recommendation 1 July 2010 Grid recommendation process 1 Jan - 1 April 2011 Meeting closed, 15:00 Next call 13 January 2010 13:00 UK Next action: 074 Actions raised at this meeting No Action 073 SH: Control of overpunching zoned positive sign Current Actions: No Action 045 20/05 AP: Speculative Parsing 27/05: Psuedo code has been circulated. Review for next call 03/06: Comments received and will be incorporated 09/06: Progress but not discussed 17/06: Discussed briefly 24/06: No Progress 01/07: No Progress 15/07: No progress. MB not happy with the way the algorithm is documented, need to find a better way. 29/07: No Progress 05/08: No Progress. Will document behaviour as a set of rules. 12/08: No Progress ... 16/09: no progress 30/09: AP distributed proposal and others commented. Brief discussion AP to incorporate update and reissue 07/10: Updated proposal was discussed.Comments will be incorporated into the next version. 14/10: Alan to update proposal to include array scenario where minOccurs > 0 21/10: Updated proposal reviewed 28/10: Updated proposal reviewed see minutes 04/11: Discussed semantics of disciminators on arrays. MB to produce examples 11/11: Absorbing action 033 into 045. Maybe decorated discrminator kinds are needed after all. MB and SF to continue with examples. 18/11: Went through WTX implementation of example. SF to gather more documentation about WTX discriminator rules. 25/11: Further discussion. Will get more WTX documentation. Need to confirm that no changes need to Resolving Uncertainty doc. 04/11: Further discussion about arrays. 09/12: Reviewed proposed discriminator semantic. 16/12: Reviewed discriminator examples and WTX semantic. 23/12: SF to provide better description of WTX behaviour and invite B Connolley to next call 06/01:B Connolly not available. SF to provide more complete description. 049 20/05 AP Built-in specification description and schemas 03/06: not discussed 24/06: No Progress 24/06: No Progress (hope to get these from test cases) 15/07: No progress. Once available, the examples in the spec should use the dfdl:defineFormat annotations they provide. ... 14/10: no progress 21/10: Discussed the real need for this being in the specification. It seemed that the main value is it define a schema location for downloading 'known' defaults from the web. 28/10: no progress 04/11: no progress 11/11: no update 18/11: no update 25/11: Agreed to try to produce for CSV and fixed formats 04/12: no update 09/12: no update 16/12: no update 23/12: no update 06/01: no progress. If there is no resource to complete this action it can be deffered 056 MB Resolve lengthUnits=bits including fillbytes 12/08: No Progress ... 28/10: no progress 04/11: MB to look at lengthUnits = bits 11/11: no update 18/11: no update 25/11: no update 04/12: no update. ALan will set up a separate call to progress this action. 09/12: no update. ALan will set up a separate call to progress this action. 16/12: MB, SH and AP had a separate call. MB to distribute proposal 23/12: Discussed proposal. MB will updated 06/01: V4 discussed and approved 064 MB/SH Request WG presentation at OGF 28 25/11: Session requested 04/12: no update 09/12: no update 16/12: SH has changed request to a general session rather tha WG in the hope of attracting more people. 23/12: no update 06/01: not heard anything yet 066 Investigate format for defining test cases 25/11:IBM to see if it is possible to publish its test case format. 04/12: no update 09/12: no update 16/12: reminded dent to project manager 23/12: SH will send another reminder. 06/01: Another reminder will be sent 068 Should the roots of messages be designated.? 09/12: Yes. New dfdl:documentRoot property Closed 16/12: reopened and decided to drop property subject to agreement from SKK and SF 23/12: SKK review decision to drop dfdl:documentRoot 071 Semantics of length=0, nil handling and defaults. 23/12:SH no update 06/01: SH has started 073 SH: Control of overpunching zoned positive sign Closed actions No Action 072 Byte Order Mark and Unicode signature 23/12: Confirm BOM will be treated and part of the content. 06/01: MB v37 comments suggest additional description. closed Work items: No Item target version status 005 Improvements on property descriptions not started 011 How speculative parsing works (combining choice and variable-occurence - currently these are separate) (from action 045) awaiting completion of actions 045 012 Reordering the properties discussion: move representation earlier, improve flow of topics not started 036 Update dfdl schema with change properties ongoing 038 Improve length section including bit handling some improvement in 036 042 Mapping of the DFDL infoset to XDM none not required for V1 specification 069 ICU fractional seconds 070 Write DFDL primer 071 Write test cases. 072 it is a processing error if the number of occurrences in the data does not match the value of the expression or prefix 073 Rename dfdl:separatorPolicy="required" to "always". 074 - Last 'postFix' separator is not optional - Terminators are mandatory. - dfdl:documentFinalTerminatorCanBeMissing - dfdl:documentFinalSeparatorCanBeMissing (Action (70)) 075 Remove occursCountKind="useAvailableSpace". 076 dfdl:documentRoot, will be defined that can only be on global elements. The DFDL spec does not have to define the format of parameters to the DFDL processor but will indicate that it must be possible to adresss any element. Agreed that ANY element within the schema cane be the starting point for parsing or unparsing. 077 'delimited' means the item is delimited by the item?s terminator (if specified) or an enclosing construct?s separator or end of the enclosing construct designated by its known length or its terminator. The definition of EndOfParent also needs improving. 078 document UPA checks 079 Restrictions on use of 'special' entities in regular expressions Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Trackers posted from 037 review
by Mike Beckerle 06 Jan '10

06 Jan '10

I've posted several new tracker artifacts from my review of draft 037. Major issues have separate artifacts, and there is a big consolidated list of 61 smaller (In my opinion that is) items. http://forge.gridforum.org/sf/go/projects.dfdl-wg/tracker.specification_ver…

1 0

zoned having leading overpunched sign - evidence for existence of
by Mike Beckerle 06 Jan '10

06 Jan '10

This google search: http://www.google.com/search?rlz=1C1CHMA_enUS361US361&sourceid=chrome&ie=UT… has a number of pages that hit, Only http://www.uni.edu/coboldoc/cobrm_042.htm suggests there really is a leading overpunched sign. It suggests that CDO (an Oracle Cobol technology) supports SIGNED NUMERIC LEFT OVERPUNCHED l s, equivalent to Compaq Cobol S9(m)V9(n) LEADING, both of which mean zoned with leading overpunched sign, from what I can infer. I've never seen such data, but this suggests there are Cobol systems, or were, which can create this format. Whether anyone using such compilers has ever exercised this format is also in question of course.

2 1

DFDL bits - version 5
by Mike Beckerle 06 Jan '10

06 Jan '10

I made the title corrections below, and cleaned up the "pseudo code" a little, changing fieldWidth to bitStringLength. This is the best I can do for the v1.0 draft. On Wed, Jan 6, 2010 at 5:47 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > > Mike > > Minor comments before today's call: > > I'm happy for 1.0 to restrict the use of lengthUnits='bits' to a specific > set of logical types as per your doc. > > Section *Bits within Bit Strings of Length < 8* should really be *Bits > within Bit Strings of Length <= 8* > > Section *Numeric Value of a Bit String* is better titled *Bits within Bit > Strings of Length > 8 *for symmetry > > Regards > > Steve Hanson > Programming Model Architect, WebSphere Message Brokers, > OGF DFDL WG Co-Chair, > Hursley, UK, > Internet: smh(a)uk.ibm.com, > Phone (+44)/(0) 1962-815848 > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org Date: 06/01/2010 > 04:53 Subject: [DFDL-WG] DFDL bits - version 4 Sent by: > dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > I've fixed the issues identified by Suman and Alan. > > RE: > > 1. On the last WG call Steve H suggested allowing lengthUnit='bits' for any > logical type as long as the length was a multiple of 8. Did you decide > against that? > > > I did not decide on this either way. Which ever requires the least writing > in the spec is fine with me. > > Consider: dfdl:lengthUnits="bits" for dfdl:representation="text". This is > exactly equivalent to dfdl:lengthUnits="bytes" with the length, however it > is determined (prefix, or explicit) to be divided by 8. Now, what if the > length comes back from the expression or prefix as not a multiple of 8. Then > we have to detect a processing error..... length in bits not a multiple of > 8, and we have to explain why this is an error, etc. > > Unless there is a use case that requires this, I'd rather not put it in > just for uniformity's sake. I'd rather say that lengthKind="bits" is > available only for a specific narrow set of types. > > [attachment "DFDL Bits-bigEndian-bias.doc" deleted by Steve Hanson/UK/IBM] > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/dfdl-wg > > > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > >

1 0

DFDL bits - version 4
by Mike Beckerle 06 Jan '10

06 Jan '10

I've fixed the issues identified by Suman and Alan. RE: > > 1. On the last WG call Steve H suggested allowing lengthUnit='bits' for any > logical type as long as the length was a multiple of 8. Did you decide > against that? > > > I did not decide on this either way. Which ever requires the least writing in the spec is fine with me. Consider: dfdl:lengthUnits="bits" for dfdl:representation="text". This is exactly equivalent to dfdl:lengthUnits="bytes" with the length, however it is determined (prefix, or explicit) to be divided by 8. Now, what if the length comes back from the expression or prefix as not a multiple of 8. Then we have to detect a processing error..... length in bits not a multiple of 8, and we have to explain why this is an error, etc. Unless there is a use case that requires this, I'd rather not put it in just for uniformity's sake. I'd rather say that lengthKind="bits" is available only for a specific narrow set of types.

1 0

Information Update - Description has changed: OGF DFDL Working Group weekly call
by Steve Hanson 05 Jan '10

05 Jan '10

IBM Hursley attendees - DE3S09 Added Hursley location info

1 0

DFDL bits - version 3
by Mike Beckerle 05 Jan '10

05 Jan '10

Attached is revised per today's WG call.

3 2