- dfdl-wg - lists.ogf.org

DFDL binaryCalendarRep pattern limitations
by Steve Hanson 23 Oct '12

23 Oct '12

For discussion at next DFDL WG call. See summary below but key points are: - DFDL uses a calendar pattern to convert binary calendar 'packed', 'bcd' and 'ibm4690Packed' reps to a schema calendar type - No trimming/padding of binary reps takes place, the parser uses what was extracted from the data - A 'packed' rep will always present an odd number of digits (because of sign nibble) - A 'bcd' rep will always present an even number of digits - ICU gives an error if the number of digits presented to it exceeds the length of the calendar pattern - Therefore the onus is on the user to ensure that the calendar pattern matches the number of digits, eg, by adding leading zeros to the pattern Are we happy with that behaviour? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/10/2012 10:49 ----- From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Date: 22/10/2012 15:36 Subject: DFDL binaryCalendarRep pattern limitations Hi Steve - a summary of the problem is below While adding support for IBM4690 packed representation for binaryNumberRep and binaryCalendarRep, it has become apparent that we may need to place a restriction on the calendarPattern property, depending on the choice of binaryCalendarRep. A problem surfaces in being able to reliably and reversibly distinguish a calendar value when the pattern length is incompatible to the packing type. If we use 'ibm4690packed' and a pattern that is of odd length, we end up matching an even-length string against an odd-length pattern and there isn't necessarily a well-defined defined answer. This is best understood with an example: - Consider a pattern with an odd number of characters, such as calendarPattern=yyyyMMddDDD - A value will require a bytestream of length 6 bytes, which would be serialised as 0x0{y}{y}{y}{y}{M}{M}{d}{d}{D}{D}{D} - For example, 2012-10-22 (day295) would be represented as 0x020121022295. If a parser is represented with this value and pattern, then it will try to match the string "020121022295" against "yyyyMMddDDD". ICU returns an error because the string is longer than the pattern and I can't say I blame it. Should it ignore the zero at the start, or the 5 at the end? Without understanding the pattern, a DFDL parser cannot know. ICU can't resolve the value as it has more than one group in the pattern so it can't resolve to one single solution. This becomes more problematic when we consider behaviour for a pattern of "DDD" and of "SSS" as they expect padding at different ends by default (a value of "0100" resolves as "DDD"=100days and "SSS"=0.010 seconds). For packed representation, the opposite problem occurs: There is always a sign nibble in the packed form, so we will always have a value made up of an odd number of digits. This can't match against all patterns of even length. The solution that we discussed was to require the calendar pattern to have a certain digit count depending on the choice of binaryCalendarRep, and allow the pattern to include number characters as a form of "padding". So for the pattern in the example above, this would have to be changed so that calendarPattern='0'yyyyMMddDDD to force it to have an even digit count. If that fully explains the problem, do you want to take it to the DFDL workgroup and check what the consensus opinion is? Cheers, Andy Andy Edwards - WebSphere Message Broker - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE2 U20 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Extra Call 2012-10-22 - updated time
by Steve Hanson 22 Oct '12

22 Oct '12

As we didn't complete all the action and agenda items on Tuesday's call, I have scheduled another for today (Monday) at the 15:15 UK (10:15 Eastern) - see separate calendar invite. Agenda has been updated in GridForge: https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (continued) (22 Oct 15:15 GDT)
by Steve Hanson 22 Oct '12

22 Oct '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Starting at 15:15

1 0

OGF DFDL WG Extra Call 2012-10-22
by Steve Hanson 22 Oct '12

22 Oct '12

As we didn't complete all the action and agenda items on Tuesday's call, I have scheduled another for today (Monday) at the same time - see separate calendar invite. Agenda has been updated in GridForge: https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (continued) (22 Oct 15:00 GDT)
by smh＠uk.ibm.com 19 Oct '12

19 Oct '12

Rescheduled: OGF DFDL Working Group weekly call (continued) 22/10/2012 - Chair: Steve Hanson/UK/IBM Sent By: Steve_Hanson/UK/IBM%IBMGB Steve_Hanson Steve Hanson has rescheduled this meeting. You have not added this entry to your calendar. Required: mbeckerle.dfdl(a)gmail.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0

Spec: Leading skip applied to every repeat of a repeating element [s]
by Steve Hanson 17 Oct '12

17 Oct '12

Need confirmation from the DFDL WG as to the purpose of the leadingSkip property. The grammar says that leadingSkip applies to all occurrences of an array element. Suman has indicated (below) that his intent for the property was to apply to the array as a whole. The DFDL approach for 'array as a whole' properties is to carry them on a wrapper sequence, not the element itself. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/10/2012 09:02 ----- From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB Cc: Alex Wood1/UK/IBM@IBMGB Date: 15/10/2012 22:58 Subject: Re: [Work Item 9791] Spec: Leading skip applied to every repeat of a repeating element [s] I intended the behavior of leading and trailing skip counts for repeats to be same as in MRM. I can't say for sure if this would break anything in PL/1 as we have not implemented but certainly this behavior is required for COBOL.. We can discuss this in workgroup and if everyone agrees add it to Errata.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: Alex Wood1/UK/IBM@IBMGB Date: 10/15/2012 01:24 PM Subject: Re: [Work Item 9791] Spec: Leading skip applied to every repeat of a repeating element [s] Does that mean in MRM that we have asymmetric behaviour between leadingSkip and trailingSkip ? I'm reading that for MRM leading skip was just for first element, but trailing skip is for every repeat. That's not how the DFDL grammar says things work - it applies leading skip to every repeat. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB Date: 15/10/2012 15:18 Subject: Re: [Work Item 9791] Spec: Leading skip applied to every repeat of a repeating element [s] I am travelling and offsite and cannot access the RTC.. Yes trailing skip count get applied to every repeating element.. this is the same semantic we have in MRM.. In COBOL terms , these are slack bytes which are added at the end of the structure if required. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 4

minor errata? "out-of-band" handling
by Mike Beckerle 17 Oct '12

17 Oct '12

The spec mentions DFDL has "out-of-band" handling in section 1. A long time ago we dropped out-of-band nil handling from DFDL v1.0. This comment in section 1 feels like it was not excised at that time and should be dropped. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

question/clarification - asserts using test patterns
by Mike Beckerle 16 Oct '12

16 Oct '12

Question: Is an assertion using a regular expression pattern allowed on (a) xs:string type elements (b) any data with text representation (c) any data with text or binary representation and, does the regular expression apply to the representation or the logical data value? (a) is the only case that is not ambiguous, because the representation and the logical value are the same thing. For everything else, there’s the question of whether the test is on the representation or the logical value. If it's the logical value, then how is a regex made meaningful on a logical value that is, for example, a number, without defining a canonical representation to which the logical value is converted? If it's to apply to the representation, then exactly what data? (Eg., what grammar region) is subject to the regex? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 2

OGF DFDL WG Extra Call 2012-10-19
by Steve Hanson 16 Oct '12

16 Oct '12

As we didn't complete all the action and agenda items on Tuesday's call, I have scheduled another for Friday at 2pm UK (9am Eastern) - see separate calendar invite Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/10/2012 16:37 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 16/10/2012 13:20 Subject: OGF DFDL WG Call Agenda 2012-10-16 Please find agenda for call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Broadcast: OGF DFDL Working Group weekly call (continued) (19 Oct 14:00 GDT)
by Steve Hanson 16 Oct '12

16 Oct '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0