June 2014 - dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2014-06-17
by Steve Hanson 18 Jun '14

18 Jun '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13265?download= Note next week's call is on Wednesday, same time. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: DFDL Working Group Call
by Steve Hanson 18 Jun '14

18 Jun '14

Mike unavailable all week

1 0

Rescheduled: DFDL Working Group Call (25 Jun 16:00 GDT in DE3S09)
by Steve Hanson 17 Jun '14

17 Jun '14

Rescheduled: DFDL Working Group Call 25/06/2014 - (Repeats) Location: DE3S09 Steve Hanson has rescheduled this meeting. You have not yet responded. Repeats This entry repeats Required: jcranford(a)mitre.org, mbeckerle(a)tresys.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: Alex Wood1/UK/IBM@IBMGB, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, jonathan.w.cranford(a)gmail.com, Mark Frost/UK/IBM, mbeckerle.dfdl(a)gmail.com, Stephanie Fetzer/Charlotte/IBM@IBMUS OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

Rescheduled: DFDL Working Group Call (2 Jul 16:00 GDT in DE3S09)
by Steve Hanson 17 Jun '14

17 Jun '14

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

OGF DFDL WG Call Agenda 2014-06-17
by Steve Hanson 16 Jun '14

16 Jun '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13265?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] data where inactive escape character is to be retained
by Steve Hanson 13 Jun '14

13 Jun '14

I think I have come across a concrete example of this. It's from a standard called vCard 3.0 (RFC 2426 - http://tools.ietf.org/html/rfc2426) A backslash is used to a) escape itself; b) escape in-scope delimiters; c) indicate an embedded linefeed. The backslash would need removing for a) and b) but not for c). ESCAPED-CHAR = "\\" / "\;" / "\," / "\n" / "\N") ; \\ encodes \, \n or \N encodes newline ; \; encodes ;, \, encodes , Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 04/06/2014 08:53 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained WG call 3rd June: The generalised use case is perhaps speculative, so it was agreed not to change the DFDL spec to handle this unless a concrete use case emerges. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 20/05/2014 18:11 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained Sent by: dfdl-wg-bounces(a)ogf.org All, I’ll chime in with an observation and few more details on how Roger Costello got around a similar problem. Observation An escape block allows the escapeBlockEnd to be escaped with the escapeEscapeCharacter, while allowing the escapeEscapeCharacter itself to appear in the data without any special semantics as long as it is NOT followed by escapeBlockEnd. (From section 13.2.1 of the spec: “On parsing the dfdl:escapeBlockStart is removed from the beginning of the data and dfdl:escapeBlockEnd is removed from end of the data and any dfdl:escapeEscapeCharacters are removed when they precede a dfdl:escapeBlockEnd.”) This is really similar to the problem that Mike posed to the group below, where an escape character is sometimes an escape character but sometimes isn’t. While an escape block might not be suitable in all circumstances, the original problem that sparked Mike’s post was amenable to using an escape block, and that is how Roger Costello got around the problem. Some more details Roger Costello was using a quotation mark (“) as the initiator and terminator for quoted values in a data format. In this format, quotation marks can be escaped with a backslash (\); however, within a quoted string, the data could have a backslash as a normal data character (e.g. \n, representing two characters, not a single newline character). Roger posed his challenge to the Daffodil team, and then Mike created the example below to demonstrate the problem to the DFDL WG. In contrast to Mike’s example, Roger was having the problem with initiators and terminators, not a separator. At the time, we thought that that an escape block couldn’t be applied to the data format in question, so Mike may have altered the problem in order to prevent an escape block from clouding the issue as posed to the WG. It turns out, after more analysis, that an escape block could be used, and that solved the problem: escapeBlockStart=”"” escapeBlockEnd=”"” escapeEscapeCharacter=”\” Closing Observations In general, DFDL supports two different escape schemes with different behavior for the escape character. * When escapeKind=”escapeCharacter”, the escape character is always an escape character. * When escapeKind=”escapeBlock”, the escape character (escapeEscapeCharacter) is only an escape character in front of escapeBlockEnd. In this case, we were able to use an escape block to model the data format. While there may be a data format that has a character that is sometimes an escape character and sometimes isn’t, without a real world example, I echo Mike’s hesitance to add this feature to DFDL. HTH, Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Mike Beckerle Sent: Friday, May 02, 2014 2:47 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] data where inactive escape character is to be retained We have data that has ; as separator and has fields that look like this: abcd \; efgh That's a single field. The backslash escapes the ; so that the data is abcd ; efgh. This same data set also has abcd \n efgh Here the backslash precedes an ordinary non-delimiter. The data is supposed to be abcd \n efgh. That is, this data set requires the backslash to be retained in the data when it is not preceding the start of a delimiter. Am I missing something or is it impossible to model this? It would seem there needs to be a flag to indicate whether the escape characters that don't actually escape a delimiter are to be retained or not. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 261
by Tim Kimber 11 Jun '14

11 Jun '14

comments in <tk>tags regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 11/06/2014 10:47 Subject: Re: [DFDL-WG] Action 261 Some thoughts on this... I agree that the definition of positional sequence in the spec needs tightening as it is ambiguous as it stands and could be interpreted as a) or b). If we adopted b) then that would appear to allow 'expression' to appear in a positional sequence, but wouldn't it also allow 'stopValue'? <tk>Yes - according to definition b) stopValue would be allowable in a positional sequence. We could still disallow it if we do not believe there is any benefit in allowing it. I don't believe it introduces any particular complexities for an implementer.</tk> occursCountKind 'expression' is analogous to lengthKind 'explicit' with an expression and to lengthKind 'prefixed'. Both these lengthKinds are classified as 'specified length' when parsing but 'variable length' when unparsing. We are observing that occursCountKind 'expression' is like 'fixed' when parsing but not quite so like 'fixed' when unparsing - which is why section 16 groups 'expression' with 'parsed' for unparsing. <tk>Yes - we took a decision that the unparser should ignore the expression in lengthKind/occursCountKind, and just output whatever data happens to be in the info set. I'm not sure that it saves a lot of effort in the implementation and it certainly is not easy to justify as a consistent behaviour. For me, the unparser should treat lengthKind='explicit' the same way whether the value is static or calculated. And the unparser should treat lengthKind='expression' the same way as lengthKind='fixed'. </tk> When unparsing occursCountKind 'expression' you don't always have the calculated array length N. If the infoset was derived from XML, there is likely no 'count' element, just a bunch of elements with the same name that make up the 'array'. DFDL gives you the choice whether to manually set the count element, or to have the parser set it automatically via outputValueCalc. In the former case, you can create a document that can not be parsed; <tk>You can with the current rules too. In fact, you can parse a document with trailing optional empty array occurrences and when it is unparsed the trailing empty occurrences will have been discarded.</tk> the unparser could check the 'count' element matches the infoset, but that would involve reverse engineering an arbitrarily complex expression and is why the specification does not say that. <tk>It would involve evaluating the expression. In most cases, that will not require any lookahead because the Length/Count field will precede the array or element. Not sure where the reverse engineering comes in?</tk> Here's a real example of such an expression (albeit with lengthKind 'explicit' but the principle is the same): dfdl:length="{xs:nonNegativeInteger(fn:floor((../Length + 1) div 2))}" Alex brought up the case where the expression evaluates to 0. In a positional sequence, would you still expect a delimiter for this case? <tk>Yes, unless it is in the trailing optional region of the group and SSP='trailingEmpty'. In a positional sequence, every delimiter must be present until suppression begins ( if allowed )</tk> If 'yes' then the resultant zero length string must be treated as the 'absent representation' and ignored. If 'no' then is the sequence still positional? <tk>I don't understand the point. Why would it not be the 'empty representation'? Why must it be 'ignored' if it does happen to be the 'absent representation'? What does 'ignored' mean?</tk> Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 10/06/2014 21:22 Subject: [DFDL-WG] Action 261 Sent by: dfdl-wg-bounces(a)ogf.org Implied separatorSuppressionPolicy for occursCountKind 'expression ' (All) 10/6: Spec says it is 'never' (positional sequence) but you have to parse to identify the position, so isn't that non-positional? I think there are two alternative definitions of 'positional': a) the identity of every delimited field is known before parsing of the sequence group begins b) the identity of every delimited field is known before parsing of the field begins As an implementer, b) is sufficient because it means that the parser never needs to backtrack while parsing the group. a) allows the field identities to be statically known, but that is less important - it does not allow optimised extraction of a particular field as would be the case for a fixed-length group ( the possibility of escaped separators/terminators means that every character will need to be scanned anyway ). It may sound like a small point, but it affects two decisions 1. whether ock='expression' should be allowed within a positional sequence group ( action 261 ) 2. what the behaviour of the unparser should be w.r.t. ock='expression'. My own feeling is that ock='expression' should be treated almost exactly like ock='fixed', except that the calculated array length N is used instead of maxOccurs. - When parsing a positional sequence group it should cause N delimiters to be expected for the array. - When unparsing a positional sequence group it should cause N delimiters to be written. These rules are consistent and straightforward to describe and implement. The current rule ( unparser outputs the occurrences that are in the info set only ) allows the unparser to write a document that cannot be parsed using the same schema. regards, Tim Kimber, ----- Forwarded by Tim Kimber/UK/IBM on 10/06/2014 20:34 ----- From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 10/06/2014 17:57 Subject: [DFDL-WG] OGF DFDL WG Call Minutes 2014-06-10 Sent by: dfdl-wg-bounces(a)ogf.org Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13263?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Action 261
by Tim Kimber 11 Jun '14

11 Jun '14

Implied separatorSuppressionPolicy for occursCountKind Implied separatorSuppressionPolicy for occursCountKind ''''expression expression expression '''' (All) 10/6: Spec says it is 'never' (positional sequence) but you have to parse to identify the position, so isn't that non-positional? I think there are two alternative definitions of 'positional': a) the identity of every delimited field is known before parsing of the sequence group begins b) the identity of every delimited field is known before parsing of the field begins As an implementer, b) is sufficient because it means that the parser never needs to backtrack while parsing the group. a) allows the field identities to be statically known, but that is less important - it does not allow optimised extraction of a particular field as would be the case for a fixed-length group ( the possibility of escaped separators/terminators means that every character will need to be scanned anyway ). It may sound like a small point, but it affects two decisions 1. whether ock='expression' should be allowed within a positional sequence group ( action 261 ) 2. what the behaviour of the unparser should be w.r.t. ock='expression'. My own feeling is that ock='expression' should be treated almost exactly like ock='fixed', except that the calculated array length N is used instead of maxOccurs. - When parsing a positional sequence group it should cause N delimiters to be expected for the array. - When unparsing a positional sequence group it should cause N delimiters to be written. These rules are consistent and straightforward to describe and implement. The current rule ( unparser outputs the occurrences that are in the info set only ) allows the unparser to write a document that cannot be parsed using the same schema. regards, Tim Kimber, ----- Forwarded by Tim Kimber/UK/IBM on 10/06/2014 20:34 ----- From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 10/06/2014 17:57 Subject: [DFDL-WG] OGF DFDL WG Call Minutes 2014-06-10 Sent by: dfdl-wg-bounces(a)ogf.org Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13263?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Minutes 2014-06-10
by Steve Hanson 10 Jun '14

10 Jun '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13263?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Specification draft r11
by Steve Hanson 10 Jun '14

10 Jun '14

See WG call agenda for today @ http://redmine.ogf.org/dmsf_files/13263?download=. In order to get the updated spec published we need to close out on the open comments in the spec and incorporate agreed errata (attached) since draft r11 (attached). Let's use today's call to close some of the comments. Errata changes since April: Errata changes up to April including all public comments: Spec draft r11 incorporating errata from gwde-dfdl-experience-1-v1.2 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 10/06/2014 11:23 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Date: 30/04/2014 18:26 Subject: Re: uploaded new draft r10 - reviewed - draft r11 attached Mike, Attached is a draft r11 with comments noting any changes needed, and responding to your comments. I have not made any non-comment changes to the document. I have not undertaken a full editorial check, just reviewed the errata updates and looked at your comments. One observation on way through - the citation numbers don't seem to be hyperlinks ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Date: 30/04/2014 14:10 Subject: uploaded new draft r10 http://redmine.ogf.org/dmsf_files/13245?download= This has a few corrections to formatting (grammar terminals weren't bold ital), and some OPEN comment bubbles added to the 12.3 section where I had questions. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU #### draft-gwdrp-dfdl-v1.0.4-r11.docx moved to MyAttachments Repository V3.8 () on 14 May 2014 by Steve Hanson. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0