November 2012 - dfdl-wg - lists.ogf.org

Unparsing - Pad to "big enough length to fit..." - can it be achieved in DFDL?
by Mike Beckerle 11 Dec '12

11 Dec '12

I have a format where fields are supposed to be padded so they all have the length of the longest one. I.e., something like this with fields terminated by | chars FIELD1|FIELD2 |ST|PH | ABC |CAPE COD|MA|888-999-0000| DEFG |BOSTON |MA|- | I believe I can warn if all the participants in a column of this tabular thing don't have the same length. I would just set a variable with the length of the first row, then on subsequent rows (parsed with a different element in the schema) I would verify that the length equals what is stored in the variable by way of an assertion. I believe this is what recoverableError kind is for actually, when you have redundant information you want to warn about. The problem I have is unparsing. How in DFDL I could determine how wide to pad each field before unparsing it. The spec requires me to measure the width of the fields (including that quasi-header row which is first, and measure how long each field is, then pad them to that maximum. I believe there is no way to do this currently in DFDL, because there is no way that an expression can refer to more than one record, nor any way to iterate or operate over sets of data. Instead an application must be written which can use DFDL v1.0 but the application has to provide some of the smarts. In this case, the application would have to walk the infoset and determine the maximum length for each such field, then provide these length maximums to the DFDL schema by way of externally set variables. Is there any other way? Second issue: In addition, the spec of my format requires the total maximum length for all the fields of a row to be at most 72 characters long. When unparsing, we don't evaluate assertions. So how can I test this? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

Clarification: can you escape a pad char with an escape char?
by Mike Beckerle 04 Dec '12

04 Dec '12

Did we decide whether an escape character can precede a pad character hence, making it part of the content, not the padding? E.g., suppose pad character is P, escape character is E. then can 'abcdEPPPP' be interpreted as 'abcdP' then 3 pad characters, or did we decide that padding ignores escape chars? I think we decided that padding is outside of the escape blocks, so that if [ and ] are my escape block start and end, then PPP[PabcP]PPP would be 'PabcP' as content. but I don't think we commented on regular single escape character characters and pad char. It seems consistent to me that 'abcdEPPPP' is abcdP as content. but if not, then 'abcdEPPP' stripping off the trailing padding ends with the escape character, so the escape character is escaping nothing hence the content would be 'abcd' I believe. Comments? ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

Puzzle: unparsing format involving lengthKind='pattern'
by Mike Beckerle 03 Dec '12

03 Dec '12

I have a file format where strings have this unusual discipline for termination. The string is either 64 characters long or, it is from 0 to 63 characters long, with a NUL terminator. I can parse this like so: <xs:complexType name="myStringType"> <xs:sequence> <xs:element name="s" type="xs:string" dfdl:lengthKind="pattern"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/">  <dfdl:element> <dfdl:property name="lengthPattern"><![CDATA[([^\x00]{0,63})(?=\x00)|[^\x00]{64}]]></dfdl:property> </dfdl:element> </xs:appinfo> </xs:annotation> </xs:element> <xs:element name="term" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="0" dfdl:initiator="&NUL;" dfdl:outputValueCalc="{ '' }" minOccurs="0" maxOccurs="1" dfdl:occursCountKind="expression" dfdl:occursCount="{ if(fn:string-length(../tns:s) lt 64) then 1 else 0 }" /> </xs:sequence> </xs:complexType> What I did is use a lengthKind pattern to pick off the content, excluding the NUL, and then model the NUL explicitly as the initiator of an empty element which is optionally occurring depending on the length of the string. That will work. I'd like to hide the "term" element in a hidden group, but other than that it will work for parsing. Question: How can I unparse this? I want the schema and DFDL processor to put the "term" element into the infoset by itself depending on the length of the 's' element that I will place into the infoset. So I put an outputValueCalc on there to assign it a value of empty string, but that will only happen if the occursCount expression causes the optional element to exist at all. Will this work for unparsing? Spec currently says that occursCount expression is used only when parsing, otherwise the number in the infoset are used. But I don't want to have to put this syntax-modeling element into the infoset from the application. I just want the application to create a string, and then based on whether it is 0-63 long, I want the Nul added or not. Other ways I've tried to model these strings include as a choice of two different elements. That has a similar issue that I then have to assemble my infoset for output using a length dependent element. I can't just put a string into the infoset and have it decide when unparsing which of the two choice branches is the right one. (asserts and discriminators are parse-only also.) Anyone have better ideas? ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 3

OGF DFDL WG Call Minutes 2012-11-27
by Steve Hanson 27 Nov '12

27 Nov '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12968?download= Errata document is now at v011 (draft) here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Second draft of DFDL Errata v011
by Steve Hanson 27 Nov '12

27 Nov '12

A few corrections made and added errata from Nov 20th call. Please review for next call. Also now on Redmine. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/11/2012 17:40 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 20/11/2012 14:23 Subject: First draft of DFDL Errata v011 Please find attached the first draft of Errata v011. This incorporates all DFDL spec errata raised since v010 (June 22 2012) until the last DFDL WG call (November 13 2012). Change tracking: I accepted all changes from v010 so the changes you see are all due to v011. I would appreciate a review for correctness and omissions. Please note that some of the new errata have been best handled by updating existing related errata, so please read from start. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

OGF DFDL WG Call Agenda 2012-11-27
by Steve Hanson 26 Nov '12

26 Nov '12

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12968?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Fw: setVariable with no value set - supported?
by Steve Hanson 26 Nov '12

26 Nov '12

Updated below to clarify defineVariable and newVariableInstance. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Mark Frost/UK/IBM, Tim Kimber/UK/IBM@IBMGB, Shane Cartledge/Toronto/IBM@IBMCA, Anita Leung/Toronto/IBM@IBMCA Date: 22/11/2012 12:20 Subject: Re: [DFDL-WG] Fw: setVariable with no value set - supported? Proposal to allow empty string to be used as a value for DFDL variables of type xs:string and xs:hexBinary (or derived from them) without resorting to using an expression. To set a default value or value to empty string use the attribute syntax, eg: <dfdl:defineVariable name="myVar" type="xs:string" defaultValue=""/> <dfdl:newVariableInstance ref="myVar" defaultValue=""/> <dfdl:setVariable ref="myVar" value=""/> The following are legal and leave the variable without a default value. <dfdl:defineVariable name="myVar" type="xs:string" /> <dfdl:defineVariable name="myVar" type="xs:string"></ dfdl:defineVariable> The following are legal and the default value is obtained from the defineVariable. <dfdl:newVariableInstance ref="myVar" /> <dfdl:newVariableInstance ref="myVar"></ dfdl:newVariableInstance> The following are schema definition errors - they do not set the value to anything, which is the purpose of setVariable. This allows us to apply a semantic to these syntaxes in a future DFDL release if we need to (eg, unsets the value). <dfdl:setVariable ref="myVar"></dfdl:setVariable> <dfdl:setVariable ref="myVar"/> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Mark Frost/UK/IBM, Tim Kimber/UK/IBM@IBMGB Date: 21/11/2012 09:48 Subject: Re: [DFDL-WG] Fw: setVariable with no value set - supported? Here's what Sandy had to say on this. He raised a good point about the future possibility of unsetting a variable using the empty syntax. Steve Hanson Hi Sandy - please can you ping back when free ? Ta! Sandy (S.) Gao hi steve Steve Hanson Hi ◄ From DFDL WG call today... ◄ 2. What is correct syntax for setting the value of a variable to empty string? DFDL variables allow both XSDL lexical form and DFDL expressions as values, so it seems plausible that <dfdl:setVariable ref="x" value=""/> or even <dfdl:setVariable ref="x"/> is an allowed syntax for setting the value of a string or hexBinary variable to empty string ◄ What's your opinion here? ◄ Syntax for setting a DFDL variable to empty string. Sandy (S.) Gao the first looks reasonable. if you want to distinguish between an unset value / null value vs. empty value, then you may not want to use the second. in xml, there's a difference between an attribute that's not present and an attribute with empty value. ◄ unless you are using element content as the value for the second case. Steve Hanson That's the issue - DFDL treats the attr form and element form as equivalent Sandy (S.) Gao so for non-empty value, you could <setV ... value="abc"/> or <setV ...>abc</setV>? Steve Hanson correct Sandy (S.) Gao ic... then i suppose you need to be consistent and allow both of those forms. ◄ but can someone specify a null value? or unset a variable? Steve Hanson null not allowed ◄ I don't think we allow unsetting either ◄ Good points though Sandy (S.) Gao if a value is required, then i don't see any issue in this approach. Steve Hanson ok - I will raise the unset issue, as we don't want to box ourselves in for future ◄ thanks Sandy Sandy (S.) Gao np... i guess you'll need to add a new element e.g. <unsetValue> or a special value <setV...>##unset</setV> to support it. Steve Hanson yes - if we don't allow the empty element + missing attr to mean that Sandy (S.) Gao exactly Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Date: 19/11/2012 15:11 Subject: Re: [DFDL-WG] Fw: setVariable with no value set - supported? Here is the way I would set a variable to the empty string. <dfdl:setVariable ref="someQName">{ "" }</dfdl:setVariable> Note that the value is an expression and so must have the {...} surround, the expression returns an empty string as its value. If you don't mind quoting hell, then <dfdl:setVariable ref="someQName" value='{ "" }'/> would also work. The DFDL validator doesn't catch the error about the value attribute being missing because both the above forms are accepted and XML Schema's validation can't enforce an OR of "you have to have an element child value OR the value attribute". This check has to be done by the DFDL implementation itself, not just XSD validation of the DFDL annotations. On Mon, Nov 19, 2012 at 9:34 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: See below... Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/11/2012 14:23 ----- From: Steve Hanson/UK/IBM To: Mark Frost/UK/IBM, Cc: Shane Cartledge/Toronto/IBM@IBMCA, Anita Leung/Toronto/IBM@IBMCA Date: 16/11/2012 11:46 Subject: Re: setVariable with no value set - supported? The specific example that Shane gives is invalid anyway as you can't set the value of one of the pre-defined DFDL variables. Assuming a realistic example like: <dfdl:setVariable ref="myVar"/> I'm wondering how you would set empty string for an element of type xs:string or xs:hexBinary. <dfdl:setVariable ref="myVar" value=""/> But why isn't that the same as <dfdl:setVariable ref="myVar"></dfdl:setVariable> which in XML rules is equivalent to: <dfdl:setVariable ref="myVar"/> I will raise with the DFDL WG. Thanks Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mark Frost/UK/IBM To: Shane Cartledge/Toronto/IBM@IBMCA, Steve Hanson/UK/IBM@IBMGB, Cc: Anita Leung/Toronto/IBM@IBMCA Date: 16/11/2012 11:19 Subject: Re: setVariable with no value set - supported? Hi Shane, While the spec isn't super-specific, I'd interpret it to mean "you must give exactly one of the value attribute or element value" Spec Section 7.9 : "The syntax supports both a dfdl:value attribute and the 'value' being specified by the element value. Only one or the other may be present. (Schema definition error otherwise.)" Steve: do you think altering the spec wording above to something like " Exactly one of the dfdl:value or the element value must be present. (Schema definition error otherwise.)" Regards, Mark Frost _____________________________________ MP 211, IBM Hursley, Winchester, SO21 2JN Phone: (01962) 817009 or x247009 Email: frostmar(a)uk.ibm.com From: Shane Cartledge/Toronto/IBM@IBMCA To: Mark Frost/UK/IBM@IBMGB, Cc: Anita Leung/Toronto/IBM@IBMCA Date: 15/11/2012 22:56 Subject: setVariable with no value set - supported? Hi Mark, Do you know if DFDL should supports a setVariable with no value set? Thought this may have already come up with your runtime testing. For example: <dfdl:setVariable ref="dfdl:binaryFloatRep"/> The DFDL specification is not clear on this point, and the DFDL validator does not flag it as an error. Thanks. Shane Cartledge Business Integration Tooling Phone: 1-905-413-5171, Tie Line: 313-5171 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Broadcast: OGF DFDL WG Call (8 Jan 15:00 GMT in DE3S09)
by Steve Hanson 26 Nov '12

26 Nov '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

AUTO: Sunil Dandamudi is out of the office (returning 11/27/2012)
by Sunil Dandamudi 23 Nov '12

23 Nov '12

I am out of the office until 11/27/2012. Note: This is an automated response to your message "dfdl-wg Digest, Vol 77, Issue 34" sent on 11/23/2012 10:42:15. This is the only notification you will receive while this person is away.

1 0

OGF DFDL WG Call Minutes 2012-11-20
by Steve Hanson 23 Nov '12

23 Nov '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12967?download= Errata document is now at v011 (draft) here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0