July 2014 - dfdl-wg - lists.ogf.org

missing XPath functions?
by Mike Beckerle 28 Jul '14

28 Jul '14

Did we omit these by mistake? They seem quite important even though in DFDL they could only be used for a single character (because we only allow single node return sequences). I have a data format I am modeling where there is a sort of ad-hoc encoding using 5 bits. The codepoints are values 0 - 31 assigned to the letters 0-7 A-H J-N P-Z (that's characters 0-7 and A-Z without I and O). The best I can do without functions that create characters from codepoint integers is a 32-leaf if-then-else tree statement on an inputValueCalc. This is ok, but it seems we should have a function to convert from a character to its codepoint as an integer, and back. 7.2 Functions to Assemble and Disassemble Strings Function Meaning fn:codepoints-to-string <http://www.w3.org/TR/xpath-functions/#func-codepoints-to-string> Creates an xs:string from a sequence of Unicode code points. fn:string-to-codepoints <http://www.w3.org/TR/xpath-functions/#func-string-to-codepoints> Returns the sequence of Unicode code points that constitute an xs:string. 7.2.1 fn:codepoints-to-string fn:codepoints-to-string($arg as xs:integer*) as xs:string Summary: Creates an xs:string from a sequence of [The Unicode Standard] <http://www.w3.org/TR/xpath-functions/#Unicode4> code points. Returns the zero-length string if $arg is the empty sequence. If any of the code points in $arg is not a legal XML character, an error is raised [err:FOCH0001 <http://www.w3.org/TR/xpath-functions/#ERRFOCH0001>]. 7.2.1.1 Examples - fn:codepoints-to-string((2309, 2358, 2378, 2325)) returns "अशॊक" 7.2.2 fn:string-to-codepoints fn:string-to-codepoints($arg as xs:string?) as xs:integer* Summary: Returns the sequence of [The Unicode Standard] <http://www.w3.org/TR/xpath-functions/#Unicode4> code points that constitute an xs:string. If $arg is a zero-length string or the empty sequence, the empty sequence is returned. 7.2.2.1 Examples - fn:string-to-codepoints("Thérèse") returns the sequence (84, 104, 233, 114, 232, 115, 101) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

daffodil-calabash-extension now open-sourced
by Cranford, Jonathan W. 26 Jul '14

26 Jul '14

DFDL WG, Announcing the first open-source release of the daffodil-calabash extension. https://opensource.ncsa.illinois.edu/stash/projects/DFDL/repos/daffodil-cal… This project contain extension XProc steps so that data can be parsed into XML using DFDL and then manipulated in XProc. It integrates Daffodil into the Calabash implementation of XProc (http://xmlcalabash.com/) Here's an example to whet your appetite, from the README.txt file: <p:declare-step xmlns:p="http://www.w3.org/ns/xproc" xmlns:dfdl="urn:daffodil:calabash" version="1.0"> <p:output port="result"/> <p:import href="../../etc/daffodil-library.xpl"/> <dfdl:parse-file file="simpleCSV" schema="csv.dfdl.xsd" root="ex:file" xmlns:ex="http://example.com"/> </p:declare-step> Here, a CSV is parsed using a sample CSV DFDL schema. Of course, the real power of this extension is in manipulating the XML after the data is parsed. More examples to come on that front. More details are available at the above URL in the README.txt file. Feedback is welcome and encouraged. Please direct your comments to the daffodil-users(a)oss.tresys.com mailing list. With best regards, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org)

1 0

thread on bit order and byte order in mil-std-2045 and related formats
by Mike Beckerle 25 Jul '14

25 Jul '14

I've included material here from a separate email thread discussing the bit order and byte order requirements in order to handle mil-std-2045 and related tactical data link (TDL) standards (one is known as link16). These TDL standards are not publicly available though limited information about them (their names, and purposes) is available on Wikipedia. Much of the discussion thread revolves around link16 and the additional requirements it has above and beyond what the (publicly-available) mil-std-2045 headers require. What Russ Hawkins (link16 SME) describes in the thread below as: "first byte-level bit order reversal over the entire (element's data), then individual parsing of each field and performing a bit reversal on the post-parse field value; or the equivalent functionality of these two cascaded bit order operations (byte and field level bit order)." This is in fact equivalent to what dfdl:bitOrder='leastSignificantBitFirst' means, and is how it is implemented inside Daffodil. This transformation is required for all the fields in MIL-STD-2045, as well as the code-points of the 7-bit characters and 6-bit characters. The primary conclusions from this thread: (a) no new byteOrder enum is needed - the byte/word swapping isn't a byte-order thing, it's a pre-processing pass. (b) 6-bit chars are needed. Like all email threads this is in reverse chronological order, so you may need to peruse from the bottom. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> ________________________________________ From: Mike Beckerle Sent: Thursday, July 24, 2014 5:22 PM To: Hawkins, Russ C Subject: RE: Bit-order - Please review sections 1 to 5 ? Thanks Russ, I did have the word-swap notion wrong. However, as a preprocessing pass this doesn't need to be represented in a DFDL property quite yet. And yes, if you have values that have to be mathematically combined with small-ish calculations that is supported in both parsing and unparsing directions. ...mikeb ________________________________________ From: Hawkins, Russ C [russ.c.hawkins(a)lmco.com] Sent: Thursday, July 24, 2014 5:10 PM To: Mike Beckerle Subject: RE: Bit-order - Please review sections 1 to 5 ? Mike, It is only a preprocessing step optionally performed over the entire body content. The reordering of bytes from byte order 2143 to byte order 1234 is only a preprocessing pre-parse operation (or post-processing after unparse). Based on wording in your email below, I wanted to be sure that this occasional need does NOT swap 16-bit word-pairs (byte order 3412 to byte order 1234) but each pair of bytes are changing places with each other (byte order 2143 to byte order 1234), since the below sounded like the former. I took a quick look through the ... spec and came up with several fields well over 32 bits. None appear to require integer type. Some spares are much greater than 32 bits, off-byte bounds, and must be zeroed for unparse. I assume that is possible. I don't think any fields are true integers and I found no float/double reals. Many fields exist that exceed 24 bits and many others must be concatenated ... from different places in the binary to exceed 24 bits. I assume the latter can be accommodated with expressions that calculate the final value by multiplying and adding. Russ -----Original Message----- From: Mike Beckerle [mailto:mbeckerle@tresys.com <https://exchange10.columbia.tresys.com/owa/redir.aspx?C=011791ecd2a549079e4…> ] Sent: Wednesday, July 23, 2014 8:33 PM To: Hawkins, Russ C Cc: Riechel, Jay P; Christensen, Craig M Subject: EXTERNAL: RE: Bit-order - Please review sections 1 to 5 ? One more question. Assuming I have correctly pre-processed the msg payloads to swap the 16-bit words, then I am now ready to extract fields one by one. Are some of the fields now going to be 32-bit integers in this mixed endian byte order (I was calling it littleEndian16Bit) where the first 16 bits are the least-significant 16 bits, and the second 16 bits are the most significant 16 bits ? Or is the reordering that involves swapping the 16-bit words really ONLY needed by the pre-processing of the message payload? Thanks again ...mike ________________________________________ From: Hawkins, Russ C [russ.c.hawkins(a)lmco.com] Sent: Wednesday, July 23, 2014 12:58 PM To: Mike Beckerle Cc: Riechel, Jay P; Christensen, Craig M Subject: RE: Bit-order - Please review sections 1 to 5 ? Mike, I think your email correctly characterizes the situation - as long as in Step 2 'using the required least significant-bit first bit order" includes both: first byte-level bit order reversal over the entire body, then individual parsing of each field and performing a bit reversal on the post-parse field value; or the equivalent functionality of these two cascaded bit order operations (byte and field level bit order). Since you are considering performing this using a multi-pass prototype, I I will say how our parser performs all these cascaded operations we've been talking about, but does so with a single parse or unparse pass: (Here Russ describes a multi-pass scheme that they have implemented in some data parsing system. This is interesting enough in its own right that I've excerpted it into a separate dfdl-wg email item.) Instead of using multiple passes, ... -----Original Message----- From: Mike Beckerle [mailto:mbeckerle@tresys.com <https://exchange10.columbia.tresys.com/owa/redir.aspx?C=011791ecd2a549079e4…> ] Sent: Wednesday, July 23, 2014 8:41 AM To: Hawkins, Russ C Subject: EXTERNAL: RE: Bit-order - Please review sections 1 to 5 ? So I would characterize this in this way: Step 1: A source-site specific pre-processing pass that re-orders 16-bit words. Only done for some sites. Step 2: Parsing the fields of the message using the required least-significant-bit first bit order & little-endian byte order. If I have this right, then this requires a small application, not just the DFDL parser. The application would call the DFDL parser to parse whatever header data, and get a byte array for the payload data that must be (sometimes) pre-processed. The application would then decide whether to pre-process, i.e., swap the 16-bit words of the byte array. The application would then call the DFDL parser again to parse the actual message fields. This is typical of processing when data is encoded and must be decoded before being parsed. E.g., we have the exact same issue with email messages, where the message may have sections encoded via the base64 scheme. This must be decoded, and then the data parsed. (Another format) has the same issue in that mil-std-2045 header can say the body is gzipped. Ungzipping it isn't something we do (today) in the DFDL parser. Some day we hope to add this sort of multi-pass capability into DFDL - probably DFDL v2.0 however. In the mean time this sort of thing requires that the multiple passes are managed by a small application. We plan to use the Daffodil implementation to experiment with features that allow this sort of multi-pass capability. ________________________________________ From: Hawkins, Russ C [russ.c.hawkins(a)lmco.com] Sent: Tuesday, July 22, 2014 7:18 PM To: Mike Beckerle Subject: RE: Bit-order - Please review sections 1 to 5 ? Mike, All msg types exhibit the same behavior. For a given site, if any msg types require the littleEndian16Bit byte-pair swapping, they all do. It is not unique to any one msg type. This particular byte-pair swap is unique to the site and its supplying feed source. Then after the above site-dependent byte-pair swap, every site does require the byte-level bitOrder reversal though for our parser to parse (extract) the fields left-to-right - this provides fields that cross byte boundaries with a contiguous bit order that has all bits, including Character fields, in the correct relative position, although bit-reversed. Then because of that field level reversal that we introduced, every field, including Characters (including 6-bit Characters) must have its bits reversed to get the intended value. I have looked over our ... msg type parse tables and verified that it exhibits this behavior in its entire body, including its 6-bit Character fields. Russ Hawkins -----Original Message----- From: Mike Beckerle [mailto:mbeckerle@tresys.com <https://exchange10.columbia.tresys.com/owa/redir.aspx?C=011791ecd2a549079e4…> ] Sent: Tuesday, July 22, 2014 4:26 PM To: Hawkins, Russ C; Stephen Lawrence; Taylor Wise Cc: Christensen, Craig M; Riechel, Jay P Subject: EXTERNAL: RE: Bit-order - Please review sections 1 to 5 ? Thanks Russ, I looked over your comments. I've got the 6-bit charset encoding. That we can add easily. I understand from your comments that for some message types the whole message payload (but not header) has word and byte swapping that must be done as a pre-processing pass before the payload can be parsed into its multiple fields. Is there an example specific message type that has this characteristic? If I correctly recall prior email threads, I think you said there is no clear indicator when this pre-processing is required, but I wanted to make sure I'm understanding this correctly. ...mikeb ____________________________ From: Hawkins, Russ C [russ.c.hawkins(a)lmco.com] Sent: Tuesday, July 22, 2014 1:12 PM To: Stephen Lawrence; Mike Beckerle; Taylor Wise Cc: Hawkins, Russ C; Christensen, Craig M; Riechel, Jay P Subject: RE: Bit-order - Please review sections 1 to 5 ? Mike, Some preliminary comments embedded with initials RH. Obviously a lot of thought and work went into this, however some things were confusing to me. Noted need for 6-bit Packed ASCII encoding and possible need for bitOrder and 16bit byte order reversals for 6-bit ASCII and other non-numeric fields. Russ Hawkins -----Original Message----- From: Stephen Lawrence [mailto:slawrence@tresys.com <https://exchange10.columbia.tresys.com/owa/redir.aspx?C=011791ecd2a549079e4…> ] Sent: Thursday, July 17, 2014 9:24 AM To: Mike Beckerle; Hawkins, Russ C; Taylor Wise Subject: EXTERNAL: RE: Bit-order - Please review sections 1 to 5 ? Made a couple of comments (attached). Some things in section 5 were a little unclear to me. - Steve ________________________________________ From: Mike Beckerle Sent: Wednesday, July 16, 2014 4:25 PM To: 'russ.c.hawkins(a)lmco.com'; Stephen Lawrence; Taylor Wise Subject: Bit-order - Please review sections 1 to 5 ? Russ, Steve, Taylor, Attached is an internal DFDL Workgroup working document about bit order. It is my personal internal draft as I'm the author. I believe I now understand why bit order has been so hard to convey to others in the DFDL workgroup. It is because there are two different formats both of which can be considered to be "LSB first". This document teases them apart. All review comments gratefully accepted. If it is easiest, you can edit the document directly and put in MS Word Comment bubbles. Then I can merge them. Or however you prefer really. ...mike beckerle

2 1

lengthUnits bits not allowed for strings, binary floats, hexBinary
by Mike Beckerle 25 Jul '14

25 Jul '14

DFDL spec currently says this w.r.t. lengthUnits property: - 'bits' may only be used for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt, and xs:unsignedLong simple types with binary representation. This feels like a hold over from when we only had strings made up of 8-bit byte code-units. Now that we have 7-bit and 6-bit characters, this restriction seems unnecessary, and is in fact awkward because many specs specify the lengths of strings in bits (which are used universally for the length of everything in these formats). This is a real concern. In many cases DFDL Schemas will be generated from other specifications by programs. Having to conditionally convert the length as specified into different units for strings is just one more place to have to test, one more way the DFDL schema doesn't obviously match the specification from which it was derived, etc. Similarly: - 'bytes' must be used for type xs:hexBinary. - 'bytes' must be used for types xs:float and xs:double with binary representation. These are to prevent the user misunderstanding the limitations of these types. I.e., that we dont support hexbinary that is not a multiple of 8-bits in size, and float and double that are not exactly 4 and 8 bytes respectively. But now this restriction just seems annoying. If my data format specification base has all these values in bits, then it is painful when creating a DFDL schema to have to transform the values for just those element declarations that are of these types. I'm not suggesting we lift the actual restrictions. I'm good with hexBinary requiring whole bytes, and that float and double are exactly 32-bits and 64-bits respectively. I just think having to use bytes as the length units is just arbitrary. We thought it would be preventing people from making mistakes, but in fact it is likely to have the opposite effect, forcing them to have to interpret the length differently based on a type that might not even be defined in the same file where they see the dfdl:length property. Consider: <element name="x" type="foo:xType" dfdl:length="448"/> Is that 448 correct? It depends on the definition of foo:xType. If it's a simple type derived from string, then length units has to be characters or bytes, but in all the formats where I see these 448's. They are measured in bits. This is 56 bytes, holding 64 characters. But when I write out this element I don't have the information right there to know whether to divide by 8 or 7 or not without knowledge of the type. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

draft r18 on redmine
by Mike Beckerle 25 Jul '14

25 Jul '14

I posted spec r18 to redmine. This incorporates all the errata as of the 1.5 experience doc 1. - One exception - the errata about implementation-defined and implementation-dependent. I went through and updated all the places where this was not correctly stated per Jonathan's document. - However, I did not create an appendix also listing all these things. That turns out to be hard to do, and redundant (you end up copying text blocks - then they're in two places....) - I think if we want an index to everyplace that "implementation-defined" and "implementation-dependent" appear in the document, that's what an index is for, so we should generate one and put those in as indexed terms. I added the appendix of DFDL Standard Character Set Encodings, which includes both 7-bit and 6-bit flavors now. I believe the last thing to do is bit order, but that should be reviewed as a separate document first. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2014-07-22
by Steve Hanson 23 Jul '14

23 Jul '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13301?download= Please review spec draft r17 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

new documents
by Mike Beckerle 22 Jul '14

22 Jul '14

I pushed two new documents: http://redmine.ogf.org/dmsf/dfdl-wg?folder_id=5485 They are draft-gwdrp-dfdl-v1.0.4-r17 of the spec - mostly this is SMH changes, but a few others. Also has the US-ASCII-7-Bit-Packed moved out into a separate spec. draft-gwdi-dfdl-standard-encodings-v01.docx - first cut at a 2nd spec document which specifies only DFDL Standard character set encodings. Only entry currently is x-dfdl-us-ascii-7-bit-packed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 2

Fw: errata on number of fractional second digits
by Steve Hanson 22 Jul '14

22 Jul '14

Erratum 2.68 updated to reflect below. Spec draft r17 contains the change. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2014 14:43 ----- From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 09/07/2014 18:21 Subject: Re: [DFDL-WG] errata on number of fractional second digits Jonathan The two statements apply to different pattern symbols. The first is for the S symbol, the second is for the I symbol. However, it is clearer if the implementer uses the same precision for each, so the I symbol should change to match the S symbol. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 08/07/2014 16:07 Subject: [DFDL-WG] errata on number of fractional second digits Sent by: dfdl-wg-bounces(a)ogf.org Not sure if this has been addressed by a subsequent errata or not, but I just noticed something that should be corrected in the spec. Section 13.11.1 is on the dfdl:calendarPattern property. The following two statements are made about the number of fractional second digits. - "Any number of fractional seconds "S" may by specified in the pattern and accepted by implementations, but an implementation is free to represent a limited number of fractional seconds internally. Excess fractional seconds are truncated, not rounded up. At least millisecond accuracy must be implemented." (Section 13.11.1) - "The number of fractional second digits supported is implementation dependent but must be at least one." (Section 13.11.1) These two statements are not consistent. I believe the second one should be corrected to say that the minimum # of fractional second digits is three (to support a minimum of millisecond accuracy). I'm working through the last version that was put out for public comment (v1.0.4), so my apologies if this has already been corrected. FYI, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org) -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2014-07-22
by Steve Hanson 22 Jul '14

22 Jul '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13301?download= Please review spec draft r17 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Please review - New version of bit order/MIL-STD-2045 document
by Mike Beckerle 18 Jul '14

18 Jul '14

I posted this to our documents. http://redmine.ogf.org/dmsf_files/13300 This version is quite different. This version proposes that there are actually two different variations on the least-significant-bit first format. One is where the bits are actually reversed (logical 0xE2 becomes 0x47 when unparsed to a file) The other is where the bits are simply interpreted as LSB first, but the bytes are stored 'normally'. (0xE2 is in the file, but the 'first' 3 bits of it are 010, not 111.) Turns out it is this latter form that MIL-STD-2045 and related standards need, and that the Daffodil code implements today. I hope this version clarifies some things. You'll see there is a need to come up with useful names for these concepts when you read the document. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2