September 2014 - dfdl-wg

Catch 22.for EVDP & NVDP ?
by Steve Hanson 17 Sep '14

17 Sep '14

<xs:element name="choiceGroup" dfdl:lengthKind="implicit"> <xs:complexType> <xs:choice dfdl:choiceLengthKind="implicit" dfdl:initiatedContent="yes"> <xs:element name="myNum1" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:textNumberPattern="#0" dfdl:initiator="num:" dfdl:terminator="~" /> > <xs:element name="myText1" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:initiator="text:" dfdl:terminator="~" /> </xs:choice> </xs:complexType> </xs:element> The above model is a choice of two elements, each of which is fixed length and has an initiator and terminator. Because the elements are fixed length, I must set dfdl:emptyValueDelimiterPolicy (EVDP) to be 'none' **. I also want to discriminate using the initiator, so I set dfdl:initiatedContent 'yes' on the choice, so I must set EVDP to 'initiator' or 'both' ++. (Same for NVDP if nillable). Catch 22. ** 'none' because the only valid empty representation is that there is nothing there at all. The element is fixed length X, so if the delimiters are present then the content must be length X and can't be zero length. ++ 'initiator' or 'both' because an initiator must always be present when dfdl:initiatedContent 'yes'. The implication is that I can't use dfdl:initiatedContent for this choice. While I can switch off nillable to avoid the error with NVDP, there is no other way of avoiding the EVDP error. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

referencing local element declarations from path expressions between two schema documents
by Mike Beckerle 17 Sep '14

17 Sep '14

I am looking for clarification about how path steps are to be interpreted with respect to the issue of qualified/unqualified element names, and default or no-default namespaces. If there is a precedent we can follow set by other tools which embed paths into XSD (e.g., schematron?) or even just the selector/paths things in XSD key/uniqueness constraints then I'd like to understand it. These are some of the conundrums: Two Schema Files: A, and B Schema file A has target namespace AA, and binds the AA namespace to prefix aa, and the BB namespace to prefix bb Schema file B has target namespace BB, and the same prefix bindings. Schema B has an element named b1 which has two local element declarations inside it named c1 and c2. Schema A has an element named a1 which contains an element reference to bb:b1. a1 has a DFDL annotation (e.g., dfdl:assert) which has a path expression in it. this path contains bb:b1/c1 When is the path correct? 1a) when schema B has elementFormDefault='unqualified' (that's the default), and schema A does NOT have a default namespace. Why? the default namespace - if it existed for schema A - would implicitly qualify the c1 step of the path, but since c1 is a local element in schema B with element form unqualified, it does not require (and cannot have) a namespace specified in path steps for c1. 1b) when schema B has elementFormDefault='qualified', and schema A has xmlns='BB'. Why? Because in this case the path bb:b1/c1 is equivalent to bb:b1/bb:c1, and since c1 will have a namespace-qualified name in schema B, that path will properly describe the path to the c1 element. 1c) Variation: when schema B has no target namespace (which I believe makes elementFormDefault irrelevant), and schema A has no default namespace. Because then c1 can't be qualfied as it lives in no namespace, and schema A has no implied qualifiers. When is the path incorrect? 2a) when schema A has xmlns="BB". Why? Because the default namespace means the bb:b1/c1 is equivalent to {BB}b1/{BB}c1, but there is no {BB}/c1 since c1 is just a local element declaration, and B has elementFormDefault unqualified. 2b) when schema B has elementFormDefault='qualified', and schema A has no default namespace. In that case the step c1 needs a namespace qualifier either by default, or explicitly via a prefix, but it doesn't have one. So I am interested in opinions on whether the above analysis is correct or not, and especially in precedent for how this stuff works in other tools. I have made a big assumption above which is that if xmlns="BB", then any path step inside something carrying that default binding is interpreted as qualified by that namespace. I picked this because it seemed reasonable, and it seems one can always modify one's schema to properly address something in another schema without having to change that other schema. I.e., it seems to have good composition properties. Thanks for any thoughts. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

OR operator in DFDL Expressions
by Mike Beckerle 17 Sep '14

17 Sep '14

This is really a question about how IBM's DFDL works, but I have to introduce the topic: XPath 2.0 says that implementations have freedom about whether these expressions cause errors or not: true() or error() = true() Let's call this the sequential semantics true() or error() = error Let's call this the parallel semantics error() or true() = true Let's call this the bizarre semantics There are analogous cases for 'and' with false() Saxon, which dates from XPath 1.0, implements the sequential semantics, which is required by XPath 1.0, and so that's what the various TDML tests and such that we have in the Daffodil project have come to depend on. Quite often we have things like dfdl:occursIndex() = 1 or ../r[dfdl:occursIndex() - 1]/flag where that 2nd operand is effectively an error if the first operand is true. Strictly by the new XPath 2.0 rules, the only portable way to write this expression is with an if-then-else. But I am reluctant to change all these tests we have. Part of me says the most conservative thing is the parallel semantics - because it prevents you from writing an OR statement like the one I have above that depends on the sequentiality. It is probably more important that the initial implementations are consistent so that schemas are more likely to interoperate. What does IBM DFDL do for the semantics of the OR expressions? Thanks Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2014-09-16
by Steve Hanson 16 Sep '14

16 Sep '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13340?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2014-09-16
by Steve Hanson 16 Sep '14

16 Sep '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13340?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Updated DFDL 1.0 Specification submitted for publishing
by Steve Hanson 10 Sep '14

10 Sep '14

See news item here: https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function
by Steve Hanson 09 Sep '14

09 Sep '14

Mark Yes, that makes more sense. The existing wording says: dfdl:occursIndex() Returns the position of the current item within an array as an xs:long. The first element is at position 1. The function may be used on non-array elements. So I blindly used xs:long for the argument type as well. On consideration the argument type should be xs:nonNegativeInteger and so should the return type. Could be xs:positiveInteger, but as you say it's not used anywhere in the spec (no XPath constructor, etc). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mark Frost/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 03/09/2014 13:35 Subject: Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function Would a parameter type xs:positiveInteger (right value space) or xs:NonNegativeInteger (less wrong value space, and already used in DFDL) be more appropriate? Regards, Mark Mark Frost IBM United Kingdom Software Engineer Hursley Park IBM DFDL, IBM Integration Bus Winchester SO21 2JN Phone: +44 (0)1962 817009 England e-mail: frostmar(a)uk.ibm.com From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 03/09/2014 13:16 Subject: Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function Sent by: dfdl-wg-bounces(a)ogf.org Closed. After discussion, settled on an xs:long argument being a 1 based count up the parent axis. It is a processing error if the argument reaches beyond the root. It is a schema definition error if the argument is <= 0. Erratum 2.167 created. To be included in draft r23. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 02/09/2014 15:21 Subject: Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function Sent by: dfdl-wg-bounces(a)ogf.org I am currently implementing functionality in the area where dfdl:occursIndex( path ) will be in the Daffodil implementation. It doesn't look very hard to put in a specific argument that is restricted to be upward only and which identifies exactly which array. This function is a special case in any DFDL runtime no matter what, and will require a fair bit of test attention for any implementation so making it simpler by allowing only the innermost array to be examined isn't saving a lot of work really. Question: is a crazy path like ../foo/../../bar/.././.. (which just so happens to be equivalent to ../../..), a schema definition error? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 2, 2014 at 9:28 AM, Mike Beckerle <mbeckerle(a)tresys.com> wrote: ________________________________ From: Steve Hanson [smh(a)uk.ibm.com] Sent: Monday, September 01, 2014 11:08 AM To: Mike Beckerle Subject: Fw: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function 269 dfdl:occursIndex() function (Steve) 29/7: Clarify behaviour and decide whether an argument is needed to make it context independent. Noted that fn:position() also does not take an argument, but it returns position in current sequence and not position in array. 4/8: With Steve but consensus is that an argument is needed. 26/8: Mike to review. If an argument is needed, we should make sure this is in next published spec as the function is used in MIL-STD-2045 schemas. Regards Steve Hanson Architect, IBM DFDL< http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 01/09/2014 16:03 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Cc: dfdl-wg-bounces(a)ogf.org Date: 31/07/2014 10:57 Subject: Re: [DFDL-WG] clarification on dfdl:occursIndex() function ________________________________ Note: The reason why fn:position() takes no argument is because it gives position in the current sequence. Which is also why we have a separate DFDL specific function dfdl:occursIndex() which is intended for giving position in enclosing array. I think we need an argument that targets the specific enclosing array. Regards Steve Hanson Architect, IBM DFDL< http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 28/07/2014 23:51 Subject: Re: [DFDL-WG] clarification on dfdl:occursIndex() function Sent by: dfdl-wg-bounces(a)ogf.org ________________________________ I agree that it is inconvenient for the 'nearest array parent' to be inaccessible. However, experience with discriminators makes me fearful of any rule that includes the phrase 'nearest enclosing' :-) I think at least one other DFDL function allows the target of the function to be specified as an argument, but insists that the argument must be in the dynamic scope of the element ( i.e. its parent/grandparent etc ). I would be much happier with that solution for occursCountIndex(). I can think of a use case where it may be useful to get a consistent behaviour for occursCountIndex(). If a DFDL schema is generated from some other data format description then the model generation code may want to refer to the nth occurrence of something else in the model, where n is the occurs index of the current element - regardless of whether this particular element is an array. regards, Tim Kimber, From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 28/07/2014 23:27 Subject: [DFDL-WG] clarification on dfdl:occursIndex() function Sent by: dfdl-wg-bounces(a)ogf.org ________________________________ This function says it can be called on non-array elements. However, it does not say what the result is. If called when "." is not itself an array element there are only two possible behaviors consistent with the fact that it is explicitly allowed on non-array elements. The result has to be either (a) 1 (b) the occursIndex of the nearest enclosing array parent, or 1 if there is no enclosing array parent. I claim (a) is fairly pointless. You will just end up having to create newVariableInstances to carry the array current index downward into expressions. I cannot think of a use case where one would want to call occursIndex() polymorphically, i.e., where you want a number in the case of an array, but 1 otherwise. So (b) is the preferable behavior. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com<http://www.tresys.com/> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy< http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 4

Sept 2014 revision is on redmine
by Mike Beckerle 09 Sep '14

09 Sep '14

I completed the editing of "r23", accepted the changes to remove the change bars, and published the MS-WORD, PDF, and Zipped HTML versions to the DFDL redmine current documents list. I removed "draft-" prefix and "-r23" suffix from the titles and file names. https://redmine.ogf.org/dmsf/dfdl-wg?folder_id=5485 I still have "r23" with changes-tracked, in case there is a need. As usual, there have sometimes been glitches in MS-Word's save-to-PDF and save-to-HTML versions. I have only given those light review. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2014-09-09
by Steve Hanson 09 Sep '14

09 Sep '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13332?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

A gap in the behaviour for when escapeKind=escapeBlock
by Andrew Edwards 09 Sep '14

09 Sep '14

While working in the escapeBlock area, I seem to have found a gap in the definition for DFDL's behaviour when using escape blocks. >From section 13.2.1, relevant extracts are: When 'escapeBlock': On unparsing the entire data are escaped by adding dfdl:escapeBlockStart to the beginning and dfdl:escapeBlockEnd to the end of the data. The data is either always escaped or escaped when needed as specified by dfdl:generateEscapeBlock. If the data is escaped and contains the dfdl:escapeBlockEnd then first character of each appearance of the dfdl:escapeBlockEnd is escaped by the dfdl:escapeEscapeCharacter. and On parsing the dfdl:escapeBlockStart string must be the first characters in the (trimmed) data in order to activate the escape scheme. The dfdl:escapeBlockStart string is removed from the beginning of the data. Until a matching dfdl:escapeBlockEnd string (that is, one not preceded by the dfdl:escapeEscapeCharacter) is found in the data, any in-scope terminating delimiter encountered in the data is not interpreted as such, and any dfdl:escapeEscapeCharacters are removed when they precede an dfdl:escapeBlockEnd string. Now consider a a model where: escapeBlockStart="start" escapeBlockEnd="end" escapeEscapeCharacter="#" Then take a logical value of: A hash is a # When we serialize the value, we wrap the value with the escapeBlockStart and escapeBlockEnd, and we preceed any instance of the escapeBlockEnd within the data with an escapeEscapeCharacter. This then gives us the physical value "startA hash is a #end". If we were to parse that data, we see the "#end" as an escaped escapeBlockEnd and report that there is no escapeBlockEnd. The gap in the behavioural definition seems to be that the specification makes no claim to do anything to escape an instance of an escapeEscapeCharacter when serializing; There is nothing to catch the case of an escapeEscapeCharacter that isn't escaping an escapeBlockEnd but ends up doing it by circumstance. Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2