April 2012 - dfdl-wg - lists.ogf.org

OGF DFDL WG Call Agenda 2012-05-02
by Steve Hanson 30 Apr '12

30 Apr '12

Please find agenda for the above call on GridForge at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… Note this call is on Wed not Tues and is an hour earlier. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (2 May 14:00 GDT)
by Steve Hanson 24 Apr '12

24 Apr '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Changing the day of the call for the week commencing 30th April, so that Mike and Steve can attend. Also allocated an extra half hour as we have a few things to cover.

1 0

Cancelled: OGF DFDL Working Group weekly call
by smh＠uk.ibm.com 20 Apr '12

20 Apr '12

Cancelled: OGF DFDL Working Group weekly call 24/04/2012 - (Repeats) Chair: Steve Hanson/UK/IBM Sent By: Steve_Hanson/UK/IBM%IBMGB Location: IBM Hursley DE2K09 Steve_Hanson Steve Hanson has cancelled this meeting. Your calendar will be updated to reflect this change. Repeats This entry repeats Required: mbeckerle.dfdl(a)gmail.com, Stephanie Fetzer/Charlotte/IBM@IBMUS, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: adam.fox(a)nrl.navy.mil, a.mumfor(a)radium.ncsc.mil, boyd.fletcher(a)us.army.mil, bradley.r.sexton(a)gmail.com, dfdl-wg(a)ogf.org, michele.zundo(a)esa.int, remcgrat(a)illinois.edu

1 0

OGF DFDL WG Call Minutes 2012-04-17
by Steve Hanson 20 Apr '12

20 Apr '12

Please find minutes from the above call on GridForge at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… The minutes cover the call on 17th plus follow-on call on 24th. Errata document is still at version 8 and may be found via https://forge.ogf.org/sf/projects/dfdl-wg Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Next DFDL WG meetings
by Steve Hanson 20 Apr '12

20 Apr '12

Tues 24th April - cancelled Tues 1st May - moved to Wed 2nd May, time & agenda to follow Regards Steve Hanson Strategy, Common Transformation & DFDL Co-Chair, OGF DFDL WG IBM SWG, Hursley, UK, smh(a)uk.ibm.com, tel +44-(0)1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof
by Mike Beckerle 20 Apr '12

20 Apr '12

What is the DFDL string literal syntax for a hexBinary type value? E.g., I want a hex binary whose value is the 4 bytes described by this hex: DE AD BE EF. <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ ... }"/> So, what can one syntactically put, for literal constant values, in the input value calculation expression? Note that this is legal pure (non-DFDL) XSD (I think) <element name="aHexBin" type="xs:hexBinary" fixed="DeadBeef"/> That is, the fixed/default are allowed and one specifies these values as just strings of hex digits. Notice no special escaping or anything. You just use a string that just so happens to contain hex digits. I think there are three possibilites (a) we allow "DEADBEEF" i.e., because the type of the expression is hexBinary, a string is cast to hexBinary by interpreting it as hex nibbles. (b) we require a special kind of string literal - a bytes-only string literal, so for example: "%#rDE;%#rAD;%#rBE;%#rEF;" is the way you create 4 bytes. If you just put characters, then that's a processing error - like a cast failure. Only raw-bytes allowed. (c) Anything you return from the expression is converted to a hexBinary by unparsing it to bytes (using current properties), then using the bytes as the hexBinary data. So you could have an expression that returns a double, and that would create 8 bytes if representation="binary". In this case the decimal number 3735928559 (hex 0xdeadbeef) as a binary bigEndian int would produce the 4 bytes I want.

3 6

Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof
by Mike Beckerle 20 Apr '12

20 Apr '12

I don't think % by itself requires any escaping. There is only a need for escaping when the characters after the % match the syntax for one of our entities. I don't expect Dfdl:terminator="%done%" to require any escaping. On Apr 19, 2012 7:16 AM, "Steve Hanson" <smh(a)uk.ibm.com> wrote: > Tim, thinking some more on this: > > - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. > In this case, the returned value is an xs:string that conforms to the DFDL > String Literal syntax > > That is indeed how properties like initiator, terminator and separator > behave today. But there is a problem. Let's say I have dynamically defined > a separator at the start of my data. The value in the data is %. My > dfdl:separator expression therefore returns %. That will give an error as a > badly formed DFDL entity. DFDL string literal rules say that you must use > %% to represent a single % character. The expression itself can work around > this by checking for % and if so substituting %%, but that's a bit > unfriendly especially as fn:replace() is not in the DFDL XPath subset - I > think this is because it comes under this category > http://www.w3.org/TR/xpath-functions/#string.match and not > http://www.w3.org/TR/xpath-functions/#substring.functions. Perhaps we > should include fn:replace() or provide a DFDL function that handles %? > > I started wondering whether these properties' expression should return a > list of String. I can envisage no format that has in its data a value that > contains DFDL entity syntax and intends it to mean a DFDL entity! That is > too contrived to be real. However I can certainly envisage an expression > like this: > > dfdl:separator = "{if ../version eq 1 then %CR;%LF; else %LF;}" > > So I think String is not sufficient and DFDL String Literal must be > allowed. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, OGF DFDL Working Group > IBM SWG, Hursley, UK > smh(a)uk.ibm.com > tel:+44-1962-815848 > > > > From: Tim Kimber/UK/IBM > To: Steve Hanson/UK/IBM@IBMGB > Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org > Date: 19/04/2012 11:32 > Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: > String literals - various usage patterns thereof > > > I agree with all of that. One refinement, though. I don't think it's > necessary to *require* an implementation to auto-cast the result of a DFDL > Expression into the target type. If an implementation wants to be picky > about the return type AND issue a clearly-worded Schema Definition Error > stating what the problem is then I think we should allow it. > > Arguably, this would reduce the portability of DFDL schemas, but there is > precedent for defining a portable subset of a language while allowing > conveniences for users who don't need portability ( e.g. ANSI 'C' ). We > already take that line for the regular expression syntax, so there is > precedent for this in the DFDL specification too. > > regards, > > Tim Kimber, Common Transformation Team, > Hursley, UK > Internet: kimbert(a)uk.ibm.com > Tel. 01962-816742 > Internal tel. 246742 > > > > > > From: Steve Hanson/UK/IBM > To: Tim Kimber/UK/IBM > Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org > Date: 19/04/2012 11:04 > Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: > String literals - various usage patterns thereof > > > Hi Tim > > Firstly, both your bulleted assertions are correct, but your conclusion is > not. > > Secondly, let me flesh out my earlier reply about constructor functions. > > The last paragraph of Section 23 says: "The result of evaluating the > expression must be a single atomic value of the type expected by the > context, and it is a schema definition error otherwise". > > This is where the XPath constructors come into play. Eg: <element > name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ xs:hexBinary > (...) }"/> > These xs: constructors, plus the special fn:dateTime() constructor that > DFDL adds, allow the correct types to be created. > > Note that you don't always need the constructors. An expression that > returns a quoted value is returning an XPath string literal so that is > automatically xs:string. An expression that returns an unquoted number is > returning an XPath number literal so that can be xs:decimal, xs:integer, > xs:double (depends whether the number contains a '.' or 'e' or 'E'). > This is described here: http://www.w3.org/TR/xpath20/#id-literals > > So simply returning the literal 'DEADBEEF' will return an xs:string and if > the context requires xs:hexBinary that is a schema definition error > according to DFDL spec. > > A clarification is worth while though. Take the following expression: > {if ../type eq 'A' then 10000 else 20000}. That returns xs:integer. > - What if my context was xs:decimal? xs:integer is a restriction of > xs:decimal so the value will always be in range, so is that 'auto-cast' > allowed? > - What if my context was xs:long or another restriction of xs:integer? The > value may or may not be in range, so is that 'auto-cast' iff value in > range? > I think that we should auto-cast when type restrictoions are involved, and > clarify that in the spec. > > We *could* change the spec to say that the result of the expression is > always automatically cast to the type expected by the context. That takes > some of the burden off the modeler and makes it much more likely that > expressions written by XPath novices will return the correct results. But > it could also hide accidental errors. Note this proposal I shall call (d) > as it not the same as Mike's (c). If we made this change, then returning > the literal 'DEADBEEF' for xs:hexBinary would succeed. I don't think it > affects the desire for expressions to be statically type checkable - > because it is known whether type X can be cast to type Y, so a cast > mismatch can be statically detected. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, OGF DFDL Working Group > IBM SWG, Hursley, UK > smh(a)uk.ibm.com > tel:+44-1962-815848 > > > > > From: Tim Kimber/UK/IBM > To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Steve > Hanson/UK/IBM@IBMGB > Date: 19/04/2012 09:35 > Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: > String literals - various usage patterns thereof > > > I'm pretty sure that the rules are: > - DFDL expressions must not *contain* DFDL String Literals. They must be > valid XPath 2.0 expressions except that the list of allowable function > names includes the DFDL extension functions. > - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. > In this case, the returned value is an xs:string that conforms to the DFDL > String Literal syntax. But that does not apply to your example because the > dfdl:inputValueCalc must return a value ( an XML value ) that is valid for > the type of the element. > > I think that corresponds to your answer a) ; 'DEADBEEF' is a valid > xs:hexBinary lexical value. > > regards, > > Tim Kimber, Common Transformation Team, > Hursley, UK > Internet: kimbert(a)uk.ibm.com > Tel. 01962-816742 > Internal tel. 246742 > > > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Steve Hanson/UK/IBM@IBMGB > Cc: dfdl-wg(a)ogf.org > Date: 19/04/2012 07:42 > Subject: [DFDL-WG] String literal syntax for hexBinary ?? - Re: > String > literals - various usage patterns thereof > Sent by: dfdl-wg-bounces(a)ogf.org > > > > What is the DFDL string literal syntax for a hexBinary type value? > > E.g., I want a hex binary whose value is the 4 bytes described by this > hex: DE AD BE EF. > > <element name="myHexBin" type="xs:hexBinary" > dfdl:inputValueCalc="{ ... }"/> > > So, what can one syntactically put, for literal constant values, in the > input value calculation expression? > > Note that this is legal pure (non-DFDL) XSD (I think) > > <element name="aHexBin" type="xs:hexBinary" fixed="DeadBeef"/> > > That is, the fixed/default are allowed and one specifies these values as > just strings of hex digits. Notice no special escaping or anything. You > just use a string that just so happens to contain hex digits. > > I think there are three possibilites > (a) we allow "DEADBEEF" i.e., because the type of the expression is > hexBinary, a string is cast to hexBinary by interpreting it as hex nibbles. > > (b) we require a special kind of string literal - a bytes-only string > literal, so for example: "%#rDE;%#rAD;%#rBE;%#rEF;" is the way you create 4 > bytes. If you just put characters, then that's a processing error - like a > cast failure. Only raw-bytes allowed. > (c) Anything you return from the expression is converted to a hexBinary by > unparsing it to bytes (using current properties), then using the bytes as > the hexBinary data. So you could have an expression that returns a double, > and that would create 8 bytes if representation="binary". In this case the > decimal number 3735928559 (hex 0xdeadbeef) as a binary bigEndian int would > produce the 4 bytes I want. > > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > >

2 3

Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof
by Tim Kimber 19 Apr '12

19 Apr '12

It's a fair point. We could allow fn:replace(), although it does have a 'pattern' (regex) parameter which will make implementation of the function a little tricky for those implementing their own DFDL Expression processor. I don't think we should define a new DFDL extension function for escaping the % sign. Checking for '%' is annoying but not difficult and probably will only be done once per parse operation ( because the result can be cached in a global variable ). regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org Date: 19/04/2012 12:16 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Tim, thinking some more on this: - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. In this case, the returned value is an xs:string that conforms to the DFDL String Literal syntax That is indeed how properties like initiator, terminator and separator behave today. But there is a problem. Let's say I have dynamically defined a separator at the start of my data. The value in the data is %. My dfdl:separator expression therefore returns %. That will give an error as a badly formed DFDL entity. DFDL string literal rules say that you must use %% to represent a single % character. The expression itself can work around this by checking for % and if so substituting %%, but that's a bit unfriendly especially as fn:replace() is not in the DFDL XPath subset - I think this is because it comes under this category http://www.w3.org/TR/xpath-functions/#string.match and not http://www.w3.org/TR/xpath-functions/#substring.functions. Perhaps we should include fn:replace() or provide a DFDL function that handles %? I started wondering whether these properties' expression should return a list of String. I can envisage no format that has in its data a value that contains DFDL entity syntax and intends it to mean a DFDL entity! That is too contrived to be real. However I can certainly envisage an expression like this: dfdl:separator = "{if ../version eq 1 then %CR;%LF; else %LF;}" So I think String is not sufficient and DFDL String Literal must be allowed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org Date: 19/04/2012 11:32 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof I agree with all of that. One refinement, though. I don't think it's necessary to *require* an implementation to auto-cast the result of a DFDL Expression into the target type. If an implementation wants to be picky about the return type AND issue a clearly-worded Schema Definition Error stating what the problem is then I think we should allow it. Arguably, this would reduce the portability of DFDL schemas, but there is precedent for defining a portable subset of a language while allowing conveniences for users who don't need portability ( e.g. ANSI 'C' ). We already take that line for the regular expression syntax, so there is precedent for this in the DFDL specification too. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org Date: 19/04/2012 11:04 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Hi Tim Firstly, both your bulleted assertions are correct, but your conclusion is not. Secondly, let me flesh out my earlier reply about constructor functions. The last paragraph of Section 23 says: "The result of evaluating the expression must be a single atomic value of the type expected by the context, and it is a schema definition error otherwise". This is where the XPath constructors come into play. Eg: <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ xs:hexBinary (...) }"/> These xs: constructors, plus the special fn:dateTime() constructor that DFDL adds, allow the correct types to be created. Note that you don't always need the constructors. An expression that returns a quoted value is returning an XPath string literal so that is automatically xs:string. An expression that returns an unquoted number is returning an XPath number literal so that can be xs:decimal, xs:integer, xs:double (depends whether the number contains a '.' or 'e' or 'E'). This is described here: http://www.w3.org/TR/xpath20/#id-literals So simply returning the literal 'DEADBEEF' will return an xs:string and if the context requires xs:hexBinary that is a schema definition error according to DFDL spec. A clarification is worth while though. Take the following expression: {if ../type eq 'A' then 10000 else 20000}. That returns xs:integer. - What if my context was xs:decimal? xs:integer is a restriction of xs:decimal so the value will always be in range, so is that 'auto-cast' allowed? - What if my context was xs:long or another restriction of xs:integer? The value may or may not be in range, so is that 'auto-cast' iff value in range? I think that we should auto-cast when type restrictoions are involved, and clarify that in the spec. We *could* change the spec to say that the result of the expression is always automatically cast to the type expected by the context. That takes some of the burden off the modeler and makes it much more likely that expressions written by XPath novices will return the correct results. But it could also hide accidental errors. Note this proposal I shall call (d) as it not the same as Mike's (c). If we made this change, then returning the literal 'DEADBEEF' for xs:hexBinary would succeed. I don't think it affects the desire for expressions to be statically type checkable - because it is known whether type X can be cast to type Y, so a cast mismatch can be statically detected. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Steve Hanson/UK/IBM@IBMGB Date: 19/04/2012 09:35 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof I'm pretty sure that the rules are: - DFDL expressions must not *contain* DFDL String Literals. They must be valid XPath 2.0 expressions except that the list of allowable function names includes the DFDL extension functions. - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. In this case, the returned value is an xs:string that conforms to the DFDL String Literal syntax. But that does not apply to your example because the dfdl:inputValueCalc must return a value ( an XML value ) that is valid for the type of the element. I think that corresponds to your answer a) ; 'DEADBEEF' is a valid xs:hexBinary lexical value. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 19/04/2012 07:42 Subject: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Sent by: dfdl-wg-bounces(a)ogf.org What is the DFDL string literal syntax for a hexBinary type value? E.g., I want a hex binary whose value is the 4 bytes described by this hex: DE AD BE EF. <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ ... }"/> So, what can one syntactically put, for literal constant values, in the input value calculation expression? Note that this is legal pure (non-DFDL) XSD (I think) <element name="aHexBin" type="xs:hexBinary" fixed="DeadBeef"/> That is, the fixed/default are allowed and one specifies these values as just strings of hex digits. Notice no special escaping or anything. You just use a string that just so happens to contain hex digits. I think there are three possibilites (a) we allow "DEADBEEF" i.e., because the type of the expression is hexBinary, a string is cast to hexBinary by interpreting it as hex nibbles. (b) we require a special kind of string literal - a bytes-only string literal, so for example: "%#rDE;%#rAD;%#rBE;%#rEF;" is the way you create 4 bytes. If you just put characters, then that's a processing error - like a cast failure. Only raw-bytes allowed. (c) Anything you return from the expression is converted to a hexBinary by unparsing it to bytes (using current properties), then using the bytes as the hexBinary data. So you could have an expression that returns a double, and that would create 8 bytes if representation="binary". In this case the decimal number 3735928559 (hex 0xdeadbeef) as a binary bigEndian int would produce the 4 bytes I want. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg

1 0

Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof
by Steve Hanson 19 Apr '12

19 Apr '12

Tim, thinking some more on this: - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. In this case, the returned value is an xs:string that conforms to the DFDL String Literal syntax That is indeed how properties like initiator, terminator and separator behave today. But there is a problem. Let's say I have dynamically defined a separator at the start of my data. The value in the data is %. My dfdl:separator expression therefore returns %. That will give an error as a badly formed DFDL entity. DFDL string literal rules say that you must use %% to represent a single % character. The expression itself can work around this by checking for % and if so substituting %%, but that's a bit unfriendly especially as fn:replace() is not in the DFDL XPath subset - I think this is because it comes under this category http://www.w3.org/TR/xpath-functions/#string.match and not http://www.w3.org/TR/xpath-functions/#substring.functions. Perhaps we should include fn:replace() or provide a DFDL function that handles %? I started wondering whether these properties' expression should return a list of String. I can envisage no format that has in its data a value that contains DFDL entity syntax and intends it to mean a DFDL entity! That is too contrived to be real. However I can certainly envisage an expression like this: dfdl:separator = "{if ../version eq 1 then %CR;%LF; else %LF;}" So I think String is not sufficient and DFDL String Literal must be allowed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org Date: 19/04/2012 11:32 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof I agree with all of that. One refinement, though. I don't think it's necessary to *require* an implementation to auto-cast the result of a DFDL Expression into the target type. If an implementation wants to be picky about the return type AND issue a clearly-worded Schema Definition Error stating what the problem is then I think we should allow it. Arguably, this would reduce the portability of DFDL schemas, but there is precedent for defining a portable subset of a language while allowing conveniences for users who don't need portability ( e.g. ANSI 'C' ). We already take that line for the regular expression syntax, so there is precedent for this in the DFDL specification too. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org Date: 19/04/2012 11:04 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Hi Tim Firstly, both your bulleted assertions are correct, but your conclusion is not. Secondly, let me flesh out my earlier reply about constructor functions. The last paragraph of Section 23 says: "The result of evaluating the expression must be a single atomic value of the type expected by the context, and it is a schema definition error otherwise". This is where the XPath constructors come into play. Eg: <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ xs:hexBinary(...) }"/> These xs: constructors, plus the special fn:dateTime() constructor that DFDL adds, allow the correct types to be created. Note that you don't always need the constructors. An expression that returns a quoted value is returning an XPath string literal so that is automatically xs:string. An expression that returns an unquoted number is returning an XPath number literal so that can be xs:decimal, xs:integer, xs:double (depends whether the number contains a '.' or 'e' or 'E'). This is described here: http://www.w3.org/TR/xpath20/#id-literals So simply returning the literal 'DEADBEEF' will return an xs:string and if the context requires xs:hexBinary that is a schema definition error according to DFDL spec. A clarification is worth while though. Take the following expression: {if ../type eq 'A' then 10000 else 20000}. That returns xs:integer. - What if my context was xs:decimal? xs:integer is a restriction of xs:decimal so the value will always be in range, so is that 'auto-cast' allowed? - What if my context was xs:long or another restriction of xs:integer? The value may or may not be in range, so is that 'auto-cast' iff value in range? I think that we should auto-cast when type restrictoions are involved, and clarify that in the spec. We *could* change the spec to say that the result of the expression is always automatically cast to the type expected by the context. That takes some of the burden off the modeler and makes it much more likely that expressions written by XPath novices will return the correct results. But it could also hide accidental errors. Note this proposal I shall call (d) as it not the same as Mike's (c). If we made this change, then returning the literal 'DEADBEEF' for xs:hexBinary would succeed. I don't think it affects the desire for expressions to be statically type checkable - because it is known whether type X can be cast to type Y, so a cast mismatch can be statically detected. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Steve Hanson/UK/IBM@IBMGB Date: 19/04/2012 09:35 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof I'm pretty sure that the rules are: - DFDL expressions must not *contain* DFDL String Literals. They must be valid XPath 2.0 expressions except that the list of allowable function names includes the DFDL extension functions. - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. In this case, the returned value is an xs:string that conforms to the DFDL String Literal syntax. But that does not apply to your example because the dfdl:inputValueCalc must return a value ( an XML value ) that is valid for the type of the element. I think that corresponds to your answer a) ; 'DEADBEEF' is a valid xs:hexBinary lexical value. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 19/04/2012 07:42 Subject: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Sent by: dfdl-wg-bounces(a)ogf.org What is the DFDL string literal syntax for a hexBinary type value? E.g., I want a hex binary whose value is the 4 bytes described by this hex: DE AD BE EF. <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ ... }"/> So, what can one syntactically put, for literal constant values, in the input value calculation expression? Note that this is legal pure (non-DFDL) XSD (I think) <element name="aHexBin" type="xs:hexBinary" fixed="DeadBeef"/> That is, the fixed/default are allowed and one specifies these values as just strings of hex digits. Notice no special escaping or anything. You just use a string that just so happens to contain hex digits. I think there are three possibilites (a) we allow "DEADBEEF" i.e., because the type of the expression is hexBinary, a string is cast to hexBinary by interpreting it as hex nibbles. (b) we require a special kind of string literal - a bytes-only string literal, so for example: "%#rDE;%#rAD;%#rBE;%#rEF;" is the way you create 4 bytes. If you just put characters, then that's a processing error - like a cast failure. Only raw-bytes allowed. (c) Anything you return from the expression is converted to a hexBinary by unparsing it to bytes (using current properties), then using the bytes as the hexBinary data. So you could have an expression that returns a double, and that would create 8 bytes if representation="binary". In this case the decimal number 3735928559 (hex 0xdeadbeef) as a binary bigEndian int would produce the 4 bytes I want. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof
by Tim Kimber 19 Apr '12

19 Apr '12

I agree with all of that. One refinement, though. I don't think it's necessary to *require* an implementation to auto-cast the result of a DFDL Expression into the target type. If an implementation wants to be picky about the return type AND issue a clearly-worded Schema Definition Error stating what the problem is then I think we should allow it. Arguably, this would reduce the portability of DFDL schemas, but there is precedent for defining a portable subset of a language while allowing conveniences for users who don't need portability ( e.g. ANSI 'C' ). We already take that line for the regular expression syntax, so there is precedent for this in the DFDL specification too. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org Date: 19/04/2012 11:04 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Hi Tim Firstly, both your bulleted assertions are correct, but your conclusion is not. Secondly, let me flesh out my earlier reply about constructor functions. The last paragraph of Section 23 says: "The result of evaluating the expression must be a single atomic value of the type expected by the context, and it is a schema definition error otherwise". This is where the XPath constructors come into play. Eg: <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ xs:hexBinary(...) }"/> These xs: constructors, plus the special fn:dateTime() constructor that DFDL adds, allow the correct types to be created. Note that you don't always need the constructors. An expression that returns a quoted value is returning an XPath string literal so that is automatically xs:string. An expression that returns an unquoted number is returning an XPath number literal so that can be xs:decimal, xs:integer, xs:double (depends whether the number contains a '.' or 'e' or 'E'). This is described here: http://www.w3.org/TR/xpath20/#id-literals So simply returning the literal 'DEADBEEF' will return an xs:string and if the context requires xs:hexBinary that is a schema definition error according to DFDL spec. A clarification is worth while though. Take the following expression: {if ../type eq 'A' then 10000 else 20000}. That returns xs:integer. - What if my context was xs:decimal? xs:integer is a restriction of xs:decimal so the value will always be in range, so is that 'auto-cast' allowed? - What if my context was xs:long or another restriction of xs:integer? The value may or may not be in range, so is that 'auto-cast' iff value in range? I think that we should auto-cast when type restrictoions are involved, and clarify that in the spec. We *could* change the spec to say that the result of the expression is always automatically cast to the type expected by the context. That takes some of the burden off the modeler and makes it much more likely that expressions written by XPath novices will return the correct results. But it could also hide accidental errors. Note this proposal I shall call (d) as it not the same as Mike's (c). If we made this change, then returning the literal 'DEADBEEF' for xs:hexBinary would succeed. I don't think it affects the desire for expressions to be statically type checkable - because it is known whether type X can be cast to type Y, so a cast mismatch can be statically detected. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Steve Hanson/UK/IBM@IBMGB Date: 19/04/2012 09:35 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof I'm pretty sure that the rules are: - DFDL expressions must not *contain* DFDL String Literals. They must be valid XPath 2.0 expressions except that the list of allowable function names includes the DFDL extension functions. - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. In this case, the returned value is an xs:string that conforms to the DFDL String Literal syntax. But that does not apply to your example because the dfdl:inputValueCalc must return a value ( an XML value ) that is valid for the type of the element. I think that corresponds to your answer a) ; 'DEADBEEF' is a valid xs:hexBinary lexical value. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 19/04/2012 07:42 Subject: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Sent by: dfdl-wg-bounces(a)ogf.org What is the DFDL string literal syntax for a hexBinary type value? E.g., I want a hex binary whose value is the 4 bytes described by this hex: DE AD BE EF. <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ ... }"/> So, what can one syntactically put, for literal constant values, in the input value calculation expression? Note that this is legal pure (non-DFDL) XSD (I think) <element name="aHexBin" type="xs:hexBinary" fixed="DeadBeef"/> That is, the fixed/default are allowed and one specifies these values as just strings of hex digits. Notice no special escaping or anything. You just use a string that just so happens to contain hex digits. I think there are three possibilites (a) we allow "DEADBEEF" i.e., because the type of the expression is hexBinary, a string is cast to hexBinary by interpreting it as hex nibbles. (b) we require a special kind of string literal - a bytes-only string literal, so for example: "%#rDE;%#rAD;%#rBE;%#rEF;" is the way you create 4 bytes. If you just put characters, then that's a processing error - like a cast failure. Only raw-bytes allowed. (c) Anything you return from the expression is converted to a hexBinary by unparsing it to bytes (using current properties), then using the bytes as the hexBinary data. So you could have an expression that returns a double, and that would create 8 bytes if representation="binary". In this case the decimal number 3735928559 (hex 0xdeadbeef) as a binary bigEndian int would produce the 4 bytes I want. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0