June 2013 - dfdl-wg - lists.ogf.org

Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language
by Tim Kimber 24 Jul '13

24 Jul '13

Interesting...that never occurred to me. I have always considered the # to be a qualifier on the XML character reference entity. In XML, the carriage-return character can be specified in any of the following ways: &cr; - A named entity  - A numeric entity ( starts with '#' )  - A numeric entity in base 16 ( the numeric part starts with 'x' ) So the '#' character is signalling that the entity is a numeric entity. But there's no need for that to be signalled in this case - the parameter for these integer constructors will always be a number. On that basis, the prefix should be 'x'. I would prefer that to '#x' on the grounds of readability. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 20/06/2013 14:01 Subject: Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language I was thinking along the same lines as Tim. I'd prefer the hex string to start with '#x' for consistency with DFDL hex entity syntax. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 20/06/2013 09:44 Subject: Re: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language Sent by: dfdl-wg-bounces(a)ogf.org As an alternative, we could - drop the 'h' prefix - use the same names as the XPath integer constructors, including the use of the full name for unsigned variants ( e.g. 'dfdl:unsignedInt' ) - behave exactly like xs:byte / xs:short etc when the string does not begin with '0x' The only difference would be the namespace of the function. So the modeller would use the dfdl variant if they needed the '0x' notation, and they could choose to use the dfdl variant always when constructing integers from string literals. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 19/06/2013 23:28 Subject: [DFDL-WG] proposal: functions to allow reasonable syntax for hex constants in DFDL expression language Sent by: dfdl-wg-bounces(a)ogf.org Rationale: in writing DFDL schemas for binary data formats, there is a strong need to express binary data constants in hexadecimal. Proposed new functions: dfdl:hByte dfdl:hUByte dfdl:hShort dfdl:hUShort dfdl:hInt dfdl:hUInt dfdl:hLong dfdl:hULong The prefix 'h' is a reminder that the function converts from hexadecimal. The prefix 'U' on the name denotes that the function creates an unsigned type result. Normally one might like to write this out fully, but in this case brevity is helpful given the expected usage of these functions to construct literal constants in DFDL expressions. All the functions take a single string argument. The string must begin with "0x" and contain at least one hex digit after that. The string must contain only hexadecimal digits, that is, the characters 0-9a-zA-Z. It is a schema definition error otherwise. The hex digits represent a big endian twos complement representation of a binary number. Each function has a limit on the number of hex digits, with no more digits than 2, 4, 8, or 16 for the byte, short, int, and long versions. That is to say for dfdl:byte and dfdl:uByte, there can be at most 2 hex digits. For dfdl:short and dfdl:uShort, there can be at most 4 hex digits, and so on. It is a schema definition error if more digits are encountered than are suitable for the type being created. Examples: dfdl:hUInt("0xa1b2c3d4") is the value 2712847316. Note that in the first byte 'a1' the most significant bit is set, but since this is an unsigned type, this is not interpreted as a sign bit. dfdl:hInt("0xFFFFFFFF") is the int value -1. The sign bit indicates that the number is negative, and this twos complement value represents -1. dfdl:hUByte("0xFF") is the unsigned byte value 255 dfdl:hByte("0xff") is the byte value -1. dfdl:hByte("0x7F") is the byte value 127 dfdl:hByte("0x80") is the byte value -128 dfdl:hUByte("0x80") is the unsigned byte value 128 dfdl:hByte("0x0A3") is a schema definition error as the leading zero is not allowed because at most 2 digits are allowed for byte types. dfdl:hShort("0x0A3") is short value 208. Leading zero causes no issue here because up to 4 digits are allowed. Example of usage in expressions: <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:setVariable ref="ex:bOrd"> { if (xs:unsignedInt(.) eq dfdl:hUInt('0xa1b2c3d4')) then 'bigEndian' else if (xs:unsignedInt(.) eq dfdl:hUInt('0xd4c3b2a1')) then 'littleEndian' else fn:error(ex:magic_number, "Magic number was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).") } </dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 6

proposal: add fn:error function and dfdl:hex function to DFDL
by Mike Beckerle 12 Jul '13

12 Jul '13

Proposed: add fn:error function and add dfdl:hex function. DFDL only needs one of the several variations of fn:error fn:error($error as xs:QName?, $description as xs:string) as none dfdl:hex($num as xs:integer) as string The dfdl:hex function inverts the behavior of the dfdl:byte, and related functions (currently also proposed) which take a string specification of a hex value. The argument to dfdl:hex can be a long, unsignedLong, or any subtype thereof. The number of hex digits in the resulting string is a function of the input type. If byte or unsigned byte, exactly 2 hex digits are produced, for short and unsignedShort, 4 hex digits, and so on. The resulting value contains a prefix 'x' character as its first character. So: dfdl:short(dfdl:hex(xs:short(208))) eq xs:short(208) is true, and a corresponding tautology holds for all the other DFDL functions that construct integers from hexadecimal, if you replace the 208 above with a value in range for the corresponding numeric type. Example: This allows a meaningful error message that includes the offending parts of data. fn:error(ex:magic_number, fn:concat("The magic number: 0", dfdl:hex(.), " was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).")) The $error argument (of type QName) is used in an implementation-dependent manner to provide the context of the error. So for example, if namespace prefix 'ex' is bound to 'http://example.com' then the above call might produce: "Error: {http://example.com}magic_number - The magic number: 0xFEFF0000 was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian)." Only the part of the above string after the hyphen is standardized. The hyphen and part before the hyphen is implementation dependent.

3 8

Implementation limits
by Steve Hanson 11 Jul '13

11 Jul '13

Just checking - did we close on this? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 21/01/2013 18:38 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 19/11/2012 21:08 Subject: Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Do we have a section where this goes? There are lots of implied implementation limits. Maximum string length? Maximum hexbinary length? Maximum number of packed digits in various numbers. Maximum length of a delimiter? Maximum length of a fixed attribute? Maximum length of a pattern regex expression? Maximum length of a regular expression? There's also limits on buffering of data for unparsing, etc. There's also limits on lookahead/speculation. I.e., how far an implementation is willing to speculate forward. etc. For some types these are clear: e.g., binary xs:int is 32 bits max. Do we need a numbered section on implementation specific limits. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 4

regex free-spacing mode
by Mike Beckerle 27 Jun '13

27 Jun '13

I wrote this complicated regex today and it works in Daffodil. Question is this. Is the (?x) which turns on regex free-spacing mode, officially supported in DFDL? You can see from below that it is VERY desirable that it works..... <xs:simpleType name="frontMatterType"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:simpleType lengthKind="pattern" terminator="%FF;"> <dfdl:property name="lengthPattern"><![CDATA[(?x) # regex free spacing mode # # match the front matter of the document # .{1,8192}? # up to 8K of front matter content # # front matter ends at the first message description page # (?= # lookahead (followed by but not including...) \f # a formfeed character (?> \s | \x08 ){1,100}? # whitespace or backspace (x08) MESSAGE\ DESCRIPTION\r # this literal text \s{1,100}? # up to 100 whitespaces -{19}\r # exactly 19 hyphens and a CR ) # end lookahead ]]></dfdl:property> </dfdl:simpleType> </xs:appinfo> </xs:annotation> <xs:restriction base="xs:string" /> </xs:simpleType> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 2

dfdl:outputValueCalc referencing itself
by Steve Hanson 26 Jun '13

26 Jun '13

The spec currently says that the expression in dfdl:outputValueCalc must not refer to itself. I am wondering if that is too restrictive. For example, what if I wanted to apply a simple transformation to the data before it is serialized, example: <xs:element name="f1" type="xs:string" dfdl:outputValueCalc="{fn:upper-case(.)}" /> Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

XSD for DFDL?
by Garriss Jr., James P. 26 Jun '13

26 Jun '13

DFDL is XML. Is there an XML Schema (XSD) for DFDL? TIA.

2 1

small spec issues
by Mike Beckerle 25 Jun '13

25 Jun '13

Jonathan Cranford reported two issues in the spec: 1. Pages 11-12 of the draft include a sample DFDL schema for a comma-separated format. I asked you about this example before, and you told me that there's no ambiguity between having a comma (,) as a separator and as the textStandardGroupingSeparator. I think I understand why (there's a question on that below). Regardless of why, though, I think that having the same character as both the separator and textStandardGroupingSeparator is a potential point of confusion, especially this early in the document. So I recommend reworking the example to avoid the appearance of ambiguity and avoid relying on the technical workings of textNumberPattern, which hasn’t been introduced yet. Is this example not ambiguous because none of the textNumberPattern attributes contain a grouping separator? If so, couldn’t the textStandardGroupingSeparator attribute just be removed from the example so that others don't get confused on it like I did? 2. Minor editorial comment - Page 15 – I think 3rd bullet should end in period instead of a comma, for consistency with other bullets.

2 1

Can I concatenate a variable?
by Garriss Jr., James P. 25 Jun '13

25 Jun '13

Suppose I define a variable like this: <dfdl:defineVariable name="boundary" defaultValue="some_boundary" type="xsd:string"/> Can I later concatenate it with other values? For example, can I concatenate it as part of a pattern? <xsd:element name="Preamble" type="xsd:string" dfdl:lengthKind="pattern" dfdl:lengthPattern="(.+?)(?=--{$some_boundary})"/> Can I concatenate it as part of a separator? <xsd:sequence dfdl:separator="--{$some_boundary}" dfdl:separatorSuppressionPolicy="anyEmpty" dfdl:terminator="--%CR;%LF;"> TIA!

3 5

Can I specify a terminator for the last element in a sequence?
by Garriss Jr., James P. 24 Jun '13

24 Jun '13

I have a repeating element, one that can occur 1 to unbounded times. The elements are separated by a prefixed separator, like so: <xsd:sequence dfdl:separator="some-separator-here" dfdl:separatorSuppressionPolicy="anyEmpty" dfdl:separatorPosition="prefix"> <xsd:element name="BodyPart" dfdl:lengthKind="delimited" minOccurs="1" maxOccurs="unbounded" type="xsd:string" dfdl:occursCountKind="implicit"/> </xsd:sequence> The last element (and only the last element) has a terminator. Is there a way in DFDL to specify that final terminator?

2 1

Re: [DFDL-WG] Can DFDL parse TLV data?
by Andre Merzky 21 Jun '13

21 Jun '13

Hello, I forwarded your mail to the DFDL-WG mailing list -- they will be able to give a qualified answer. Best regards, Andre. On Fri, Jun 21, 2013 at 10:59 AM, Xin TZ Zhang <xzhangbj(a)cn.ibm.com> wrote: > Dear OFGer: > Can DFDL parse TLV data? TLV(tag-length-value) data is binary, and variable > length... > (sorry for my poor English) > Best Regards zx. -- Nothing is really difficult.

2 1