- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Possible DFDL spec errata - errata taken
by Steve Hanson 20 Jun '12

20 Jun '12

Errata taken for both this issues in this email thread: 2. Is parent/child alignment rule too strict? Yes, remove rule altogether. 3. Does DFDL need to support signed integer types with lengthUnits 'bits'? Yes, add support for signed integer types. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org Date: 29/05/2012 19:08 Subject: Re: [DFDL-WG] Possible DFDL spec errata "Only the first child (aka "first anything" in the structure) gets aligned and can have the conflict of its local alignment specification, and that of some enclosing structure. " Yes that's what I was calling the first child rule. In DFDL terms, would that also mean that the first child can not have any leading skip, as that would throw off the alignment? We also need to bear in mind that we have complex element, sequence, first child element, all of which have alignment. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 29/05/2012 18:51 Subject: Re: [DFDL-WG] Possible DFDL spec errata I don't get it. Only the first child (aka "first anything" in the structure) gets aligned and can have the conflict of its local alignment specification, and that of some enclosing structure. If some later field gets aligned, then that's not relevant at all, as you treat that normally as another instance of alignment recursively (it could, afterall, also be a structure with sub fields having their own alignments, etc.) Is that the "first child rule" ...mike On Tue, May 29, 2012 at 1:45 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: Hi Mike The one rule that C does apply is the first child rule. C does not allow padding before the first child in a struct, so the alignment of the first child must not be greater than the alignment of the parent. But it won't complain if the third element's alignment is greater than the parent's, for example. So the rule needs to be relaxed, I think the only question is whether we drop the rule altogether or go with the first child rule. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 29/05/2012 18:08 Subject: Re: [DFDL-WG] Possible DFDL spec errata On Tue, May 29, 2012 at 12:03 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: These were on the agenda for today's cancelled call. 2. Is parent/child alignment rule too strict? Spec property description for alignment says "The alignment of a child component must be less than or equal the alignment of its parent element, sequence or choice". Experiments with creating DFDL schema from C structures have shown that this rule is violated. It seems unnecessary. Specifically, C ensures that the alignment of objects in a repeating structure is preserved by rounding up the length of the structure, rather than aligning it. I think the point of this restriction was to insure that we didin't have conflicting alignments. Where the begining of the structure was aligned, say 64, but the first element was aligned at some incompatible boundary. Like 128. It would almost certainly be an error if the first element in a structure has a larger alignment constraint than the overall structure, and the user should pick where to place the constraint, and not constrain both, except when the constraints on one are implied by the constraints of another. Example: I have a 4k page aligned structure. The first element is some type of double float coming from a global simple type def. The global simple type definition that says my specific kind of double floats are always 8-byte aligned. The 4k page is obviously 4096 byte aligned. There should be no conflict here. The page alignment "Wins" and the begining of the structure is 4k aligned, and that satisifies the 8-byte alignment of my specialized double float. The point of having any restriction at all, is to catch conflicts. Eg., what if I had something that was 64-byte aligned, and I put it inside a structure that has only 2-byte alignment....., this is probably an error. If this isn't going to catch any errors, then this restriction is unnecessary. 3. Does DFDL need to support signed integer types with lengthUnits 'bits'? Spec only allows unsigned integers to have lengthUnits 'bits', but it is possible in C structures to delare bit fields for signed integers. They behave like two's complement. It looks like the spec is being too restrictive in its types for 'bits'. I missed this over restriction. Signed are definitely needed. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] String literal syntax for hexBinary ?? - need DFDL string literal constructor
by Steve Hanson 19 Jun '12

19 Jun '12

Agreed today that such a function is needed, errata taken. Proposals for function name welcome. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org Date: 20/04/2012 09:04 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - need DFDL string literal constructor I think what we need is a function that takes a string and returns the equivalent DFDL string literal. It looks for the characters that need cause problems (%, space) and replaces them (with '%%' and '%SP;' respectively), and if the string is the empty string replaces it with '%ES;'. I think it's just those that are problematic, as all other characters will be interpreted correctly (the entity syntax is primarily just for convenience of data entry). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 19/04/2012 14:27 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Ok, these are very good reasons. So we need the function(s) in our xpath library to make doing this same substitution easy (e.g., the replace function discussed elsewhere in this thread would do it I believe because all we have to do is replace "%" with "%%".) On Thu, Apr 19, 2012 at 9:20 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: The reason that % needs escaping is that most entities start with just % - eg %NUL; - and it means we have simple rules - you want to use a literal % then escape it. and if you use just % then we expect to see an entity next. If we don't do it this way, then we will not be able to extend the list of entities in the future without breaking existing expressions, and we won't detect the very common error of leaving off the trailing semi-colon by mistake.. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: Tim Kimber/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 19/04/2012 14:01 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof I don't think % by itself requires any escaping. There is only a need for escaping when the characters after the % match the syntax for one of our entities. I don't expect Dfdl:terminator="%done%" to require any escaping. On Apr 19, 2012 7:16 AM, "Steve Hanson" <smh(a)uk.ibm.com> wrote: Tim, thinking some more on this: - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. In this case, the returned value is an xs:string that conforms to the DFDL String Literal syntax That is indeed how properties like initiator, terminator and separator behave today. But there is a problem. Let's say I have dynamically defined a separator at the start of my data. The value in the data is %. My dfdl:separator expression therefore returns %. That will give an error as a badly formed DFDL entity. DFDL string literal rules say that you must use %% to represent a single % character. The expression itself can work around this by checking for % and if so substituting %%, but that's a bit unfriendly especially as fn:replace() is not in the DFDL XPath subset - I think this is because it comes under this category http://www.w3.org/TR/xpath-functions/#string.match and not http://www.w3.org/TR/xpath-functions/#substring.functions. Perhaps we should include fn:replace() or provide a DFDL function that handles %? I started wondering whether these properties' expression should return a list of String. I can envisage no format that has in its data a value that contains DFDL entity syntax and intends it to mean a DFDL entity! That is too contrived to be real. However I can certainly envisage an expression like this: dfdl:separator = "{if ../version eq 1 then %CR;%LF; else %LF;}" So I think String is not sufficient and DFDL String Literal must be allowed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org Date: 19/04/2012 11:32 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof I agree with all of that. One refinement, though. I don't think it's necessary to *require* an implementation to auto-cast the result of a DFDL Expression into the target type. If an implementation wants to be picky about the return type AND issue a clearly-worded Schema Definition Error stating what the problem is then I think we should allow it. Arguably, this would reduce the portability of DFDL schemas, but there is precedent for defining a portable subset of a language while allowing conveniences for users who don't need portability ( e.g. ANSI 'C' ). We already take that line for the regular expression syntax, so there is precedent for this in the DFDL specification too. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, dfdl-wg(a)ogf.org Date: 19/04/2012 11:04 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Hi Tim Firstly, both your bulleted assertions are correct, but your conclusion is not. Secondly, let me flesh out my earlier reply about constructor functions. The last paragraph of Section 23 says: "The result of evaluating the expression must be a single atomic value of the type expected by the context, and it is a schema definition error otherwise". This is where the XPath constructors come into play. Eg: <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ xs:hexBinary (...) }"/> These xs: constructors, plus the special fn:dateTime() constructor that DFDL adds, allow the correct types to be created. Note that you don't always need the constructors. An expression that returns a quoted value is returning an XPath string literal so that is automatically xs:string. An expression that returns an unquoted number is returning an XPath number literal so that can be xs:decimal, xs:integer, xs:double (depends whether the number contains a '.' or 'e' or 'E'). This is described here: http://www.w3.org/TR/xpath20/#id-literals So simply returning the literal 'DEADBEEF' will return an xs:string and if the context requires xs:hexBinary that is a schema definition error according to DFDL spec. A clarification is worth while though. Take the following expression: {if ../type eq 'A' then 10000 else 20000}. That returns xs:integer. - What if my context was xs:decimal? xs:integer is a restriction of xs:decimal so the value will always be in range, so is that 'auto-cast' allowed? - What if my context was xs:long or another restriction of xs:integer? The value may or may not be in range, so is that 'auto-cast' iff value in range? I think that we should auto-cast when type restrictoions are involved, and clarify that in the spec. We *could* change the spec to say that the result of the expression is always automatically cast to the type expected by the context. That takes some of the burden off the modeler and makes it much more likely that expressions written by XPath novices will return the correct results. But it could also hide accidental errors. Note this proposal I shall call (d) as it not the same as Mike's (c). If we made this change, then returning the literal 'DEADBEEF' for xs:hexBinary would succeed. I don't think it affects the desire for expressions to be statically type checkable - because it is known whether type X can be cast to type Y, so a cast mismatch can be statically detected. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Steve Hanson/UK/IBM@IBMGB Date: 19/04/2012 09:35 Subject: Re: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof I'm pretty sure that the rules are: - DFDL expressions must not *contain* DFDL String Literals. They must be valid XPath 2.0 expressions except that the list of allowable function names includes the DFDL extension functions. - A DFDL expression is sometimes allowed to *return* a DFDL String Literal. In this case, the returned value is an xs:string that conforms to the DFDL String Literal syntax. But that does not apply to your example because the dfdl:inputValueCalc must return a value ( an XML value ) that is valid for the type of the element. I think that corresponds to your answer a) ; 'DEADBEEF' is a valid xs:hexBinary lexical value. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 19/04/2012 07:42 Subject: [DFDL-WG] String literal syntax for hexBinary ?? - Re: String literals - various usage patterns thereof Sent by: dfdl-wg-bounces(a)ogf.org What is the DFDL string literal syntax for a hexBinary type value? E.g., I want a hex binary whose value is the 4 bytes described by this hex: DE AD BE EF. <element name="myHexBin" type="xs:hexBinary" dfdl:inputValueCalc="{ ... }"/> So, what can one syntactically put, for literal constant values, in the input value calculation expression? Note that this is legal pure (non-DFDL) XSD (I think) <element name="aHexBin" type="xs:hexBinary" fixed="DeadBeef"/> That is, the fixed/default are allowed and one specifies these values as just strings of hex digits. Notice no special escaping or anything. You just use a string that just so happens to contain hex digits. I think there are three possibilites (a) we allow "DEADBEEF" i.e., because the type of the expression is hexBinary, a string is cast to hexBinary by interpreting it as hex nibbles. (b) we require a special kind of string literal - a bytes-only string literal, so for example: "%#rDE;%#rAD;%#rBE;%#rEF;" is the way you create 4 bytes. If you just put characters, then that's a processing error - like a cast failure. Only raw-bytes allowed. (c) Anything you return from the expression is converted to a hexBinary by unparsing it to bytes (using current properties), then using the bytes as the hexBinary data. So you could have an expression that returns a double, and that would create 8 bytes if representation="binary". In this case the decimal number 3735928559 (hex 0xdeadbeef) as a binary bigEndian int would produce the 4 bytes I want. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 15

OGF DFDL WG Call Agenda 2012-06-18/19 - updated
by Steve Hanson 18 Jun '12

18 Jun '12

Please find agenda for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… We have quite a few items building up, so there are two calls this week, with a single agenda as per link. Mon 18th @ 16:00 UK (extra call) Tues 19th @ 15:00 UK (regular time) The document has been updated with the results of the 18th call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Actiion 173: DFDL String Literals Analysis for review
by Steve Hanson 18 Jun '12

18 Jun '12

I have reviewed and updated Mike's spreadsheet at the link below. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/06/2012 15:59 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 23/05/2012 18:11 Subject: Actiion 173: DFDL String Literals Analysis for review ----- Forwarded by Steve Hanson/UK/IBM on 23/05/2012 18:10 ----- From: "Mike Beckerle (Google Docs)" <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: Tim Kimber/UK/IBM@IBMGB Date: 21/04/2012 17:40 Subject: String Literals Analysis (smh(a)uk.ibm.com) I've shared String Literals Analysis Message from mbeckerle.dfdl(a)gmail.com: Online editable spreadsheet of all the various string-literal situations. This is better than that UML mixin diagram. Click to open: String Literals Analysis Google Docs makes it easy to create, store and share online documents, spreadsheets and presentations. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL spec & error codes (was Fw: TDML question)
by Steve Hanson 18 Jun '12

18 Jun '12

Added to WG agenda. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 18/06/2012 09:38 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 14/06/2012 09:19 Subject: Re: TDML question We should discuss this at the WG when we have got our current backlog and action 140 out of the way. It's all part of the deferred conformance suite action 166. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Steve Hanson/UK/IBM@IBMGB Date: 13/06/2012 21:42 Subject: Re: TDML question Hi Mike, Fair question. IBM requires all diagnostic messages from its software products to be identified by a unique error code. In the IBM test driver program, it is this code that we check for. The XML Schema specification actually assigns unique strings to the various types of error that can occur ( e.g. cvc-* ). If the DFDL specification did the same then the TDML format would be able to specify that the content of the <errors> tag is a list of defined error codes. For DFDL v1.0, I think implementers are free to use it in a way that fits their own requirements. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 13/06/2012 20:45 Subject: TDML question I'm enhancing the daffodil TDML runner. I want to keep it compatible with the IBM one so that we can use TDML files as an interchange medium for discussing bugs/semantics/etc. I added the error feature where one can put expected errors into the TDML file. I have a question about the <error>...</error> element. Which is this: How is the string contents of these error elements used? I tentatively just have it search the error messages created by the parse for these error strings. If any actual error message contains the error string, then that error "passes". Here's my example, which is parsing a 2 character integer, which will fail because the text is AA, and this is base 10. <ts:testSuite xmlns:ts={ tdml } suiteName="theSuiteName"> <ts:parserTestCase ID="some identifier" name="firstUnitTest" root="data"> <ts:document>AA</ts:document> <ts:errors> <ts:error>convert</ts:error>  <ts:error>xs:int</ts:error>  </ts:errors> </ts:parserTestCase> </ts:testSuite> So my test passes so long as the words "convert" and "xs:int" are found in the error message that is generated. Is this consistent with your TDML file usage? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-06-18/19
by Steve Hanson 18 Jun '12

18 Jun '12

Please find agenda for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… We have quite a few items building up, so there are two calls this week, with a single agenda as per link. Mon 18th @ 16:00 UK (extra call) Tues 19th @ 15:00 UK (regular time) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Minutes 2012-06-12
by Steve Hanson 12 Jun '12

12 Jun '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Extra call next Monday to progress agenda backlog. Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Invitation: OGF DFDL Working Group weekly extra call (18 Jun 16:00 GDT in IBM Hursley DE3S09)
by Steve Hanson 12 Jun '12

12 Jun '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/

1 0

Re: [DFDL-WG] Action 174: Making DFDL implementations easier
by Steve Hanson 12 Jun '12

12 Jun '12

Interesting, but there's a problem - variables are currently an optional feature! You could swap variables for paths, which is what I think Mike proposed in an earlier mail? exprSubset = / | / exprList atom = . | .. | identifier exprList = atom | atom / exprList That's essentially equivalent to what MRM provides for its 'repeat reference' and 'length reference' properties. It doesn't handle things like an array count being 0 based, where you need the ability to add 1 to the result. And you can't do a simple comparison, which would make it easy to add choices and discriminators to an implementation. The trouble is that once you add in operators and literals you've pulled in a lot of the language. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Steve Hanson/UK/IBM@IBMGB Date: 25/05/2012 14:29 Subject: Re: [DFDL-WG] Action 174: Making DFDL implementations easier My current thinking is: - Minimum expression language is a simple variable reference. e.g. {$myLength} or {$myParentStructureLength}. - That would require the ability to declare DFDL variables, set them, read them and put them into and out of scope. And restore their value after backtracking if backtracking is supported in the implementation. This would make all the XPath functions optional, and would also make it unnecessary to use an XPath query engine. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: Tim Kimber/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 25/05/2012 13:58 Subject: Re: [DFDL-WG] Action 174: Making DFDL implementations easier But the expression language is highly restricted in the subset. On Fri, May 25, 2012 at 7:59 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Tim I tend to agree with endOfParent as optional. Expression language was kept in the core to handle occursCountKind 'expression' which is also in the core. The examples of binary data we have seen from NSA and ESA both have occurs counts in the data, in the same way as COBOL does. When you have untagged binary data, it's typically the way to provide an array size. It was felt that dropping this from the core did left us with too little capability. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 25/05/2012 10:07 Subject: Re: [DFDL-WG] Action 174: Making DFDL implementations easier 1) I would make endOfParent an optional feature - there are not many formats that require it. 3) There are many formats that do not require the expression language - it is only required when a property value or an assert/discriminator needs to query already-parsed data. On that basis, I think the entire expression language feature should be optional. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 25/05/2012 00:08 Subject: [DFDL-WG] Action 174: Making DFDL implementations easier Sent by: dfdl-wg-bounces(a)ogf.org Agreed on list, just need to answer questions 1) and 3) below. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/05/2012 18:21 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 15/05/2012 09:55 Subject: Making DFDL implementations easier Please see below for a proposal to make an additional set of DFDL features optional. The goal is to make it considerably easier to create a minimal conforming DFDL processor for binary data. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/05/2012 12:50 ----- Feature Detection Text representation for types other than String dfdl:representation="text" for Number, Calendar or Boolean types Delimiters dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or dfdl:lengthKind="delimited" BCD calendars dfdl:binaryCalendarRep="bcd" Multiple schemas xs:include or xs:import in xsd Named Formats dfdl:defineFormat or dfdl:ref Choices xs:choice in xsd ** Arrays where size not known in advance dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' ** Advanced expressions Advanced features of the DFDL expression language (tbd) ** Including one of these features mean that speculative parsing is needed. Remaining questions: 1) What about lengthKind 'endOfParent' ? 2) Is leaving out choices too restrictive? 3) Expression language subset The result is that a minimal conformant DFDL implementation just needs to support the following annotations and properties, and does not need speculative parsing. dfdl:element dfdl:sequence dfdl:format byteOrder encoding utf16width alignment alignmentUnits (bytes) fillByte leadingSkip trailingSkip lengthKind (explicit, implicit) length lengthUnits (bytes, characters) representation (binary) textPadKind textTrimKind textStringJustification textStringPadCharacter truncateSpecifiedLengthString decimalSigned binaryNumberRep binaryVirtualDecimalPoint binaryFloatRep (ieee) binaryBooleanTrueRep binaryBooleanFalseRep binaryCalendarRep (binarySeconds, binaryMilliseconds) binaryCalendarEpoch sequenceKind (ordered) occursCountKind (fixed, expression) occursCount Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 3

OGF DFDL WG Call Agenda 2012-06-12
by Steve Hanson 11 Jun '12

11 Jun '12

Please find agenda for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… We have quite a few items building up, and could do with a second call this week. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0