November 2014 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

Fw: Fw: Action 248 (was Thoughts on a discriminator scenario)
by Steve Hanson 26 Nov '14

26 Nov '14

This action was raised because of concern with the behaviour of the discriminator in the following example. Because OCK is 'implicit' the 1st occurrence is not an actual PoU but the other 9 occurrences are. This means that for 1st occurrence, the discriminator actually acts on a higher PoU if one exists. <xs:element name="Type1" maxOccurs="10" dfdl:occursCountKind="implicit"> <dfdl:discriminator test="{fn:exists(A)}" /> <xs:complexType> <xs:sequence> <xs:element name="A" dfdl:initiator="A:" ... /> <xs:element name="B" dfdl:initiator="B:" ... /> <xs:element name="C" dfdl:initiator="C:"... /> </xs:sequence> </xs:complexType> This led to the suggestion that a discriminator should not 'leak' beyond a potential PoU, regardless of whether it is an actual PoU. The argument for this is contained in the thread below, and on re-reading I still think it is the best solution to this, so that is what I propose. There were also issues about the wording in section 9.3.3. Sections 9.3.3 and 7.4 are reproduced below, and updated to address the wording and leaking issues. ------------------------------------------------- 9.3.3 Points of Uncertainty A point of uncertainty occurs when parsing a schema component when an occurrence of that schema component might not be the next item encountered in the data stream. Points of uncertainty can be nested. Any one of the following schema constructs is a potential point of uncertainty: · A branch of xs:choice · All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') · An optional xs:element · An array xs:element. · All xs:elements in an xs:sequence containing one or more floating xs:elements. The parser resolves these points of uncertainty by way of a set of construct-specific rules given below along with determining whether schema components are known-to-exist or known-not-to-exist. For some of these constructs, whether there is an actual point of uncertainty depends on the representation of the constructs in the data. A branch of xs:choice is always a point of uncertainty. A choice is resolved sequentially, or by direct dispatch. Sequential choice resolution occurs by parsing each choice branch in schema definition order until one is known-to-exist. It is a processing error if none of the choice branches are known-to-exist. Direct-dispatch choice resolution occurs by matching the value of the dfdl:choiceDispatchKey property to the value of the dfdl:choiceChoiceBranchKey property of one of the choice branches. It is a processing error if none of the choice branches have a matching value in their dfdl:choiceChoiceBranchKey property. An element in an unordered xs:sequence is always a point of uncertainty. It is resolved by parsing for the child components of the sequence in schema definition order at each point in the data stream where a component can exist until the required number of occurrences of each child component is known- to-exist or the sequence is terminated by delimiters or specified length. An element in a sequence with one or more floating elements is always a point of uncertainty. It is resolved by parsing for the expected element at that point in the data stream. If the expected element is known-not-to-exist then an occurrence of each floating element is parsed in schema definition order. When parsing an array, points of uncertainty only occur for certain values of occursCountKind, as follows: occursCountKind Details of Point of Uncertainty fixed No point of uncertainty (maxOccurs occurrences expected). implicit A point of uncertainty exists after minOccurs occurrences found and until maxOccurs found. parsed A point of uncertainty exists for all occurrences expression No point of uncertainty (occursCount occurrences expected) stopValue No point of uncertainty (stopValue must always be present, even when minOccurs is 0). Table 11: Points of Uncertainty and dfdl:occursCountKind An optional element point of uncertainty is resolved by parsing the element until it is either known-to-exist or known-not-to-exist. Whether an optional element is an actual point of uncertainty depends on property dfdl:occursCountKind as described above. (Property dfdl:occursCountKind is defined in Section 16.1 dfdl:occursCountKind property.) For an array element, the point of uncertainty is resolved for each occurrence separately by parsing the occurrence until it is either known-to-exist or known-not-to-exist. Note that a discriminator defined on, or contained by, a schema construct that is a potential point of uncertainty, will only ever resolve that point of uncertainty. For example, if a discriminator is defined on an array element which is contained within the branch of a choice, the discriminator will only resolve the existence of occurrences of the array element, and never the existence of the occurrence of the choice branch. This is true even when a potential point of uncertainty is not an actual point of uncertainty. ------------------------------------ 7.4 The dfdl:discriminator Statement Annotation Element DFDL discriminators are used during parsing to resolve points of uncertainty that cannot be resolved by speculative parsing. Discriminators are not used during unparsing. They can also be used to force a resolution earlier during the parsing of a group so that subsequent parsing errors are treated as processing errors of a known component rather than a failure to find a component. A discriminator determines the existence or non-existence of a component. If the discriminator is successful then the component is known to exist and any subsequent errors will not cause backtracking at points of uncertainty. If a discriminator is unsuccessful then the component is known not to exist and backtracking occurs immediately. If the complex type of an element contains a sequence group as its content model then if the sequence group is known not to exist, then the element is known not to exist. Examples of dfdl:discriminator annotation are below : <dfdl:discriminator> { ../recType eq 0 } </dfdl:discriminator> <dfdl:discriminator test="{ ../recType eq 0}" /> When the discriminator's expression evaluates to "false", then it causes a processing error, and the discriminator is said to fail. A discriminator defined on, or contained by, a schema construct that is a potential point of uncertainty, will only ever resolve that point of uncertainty. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 17/11/2014 11:54 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 15/05/2014 10:48 Subject: Re: [DFDL-WG] Fw: Action 248 (was Thoughts on a discriminator scenario) Tim - I've responded to your specific comments below in blue font. All - You will see that I have some concerns over the words used in the definition of a PoU, as we seem to be unclear as to whether a PoU is a point in the data stream or a point in the model. I am wondering whether the concepts of 'potential PoU' and 'actual PoU' can be better expressed as 'PoU in the model' and 'PoU in the data'. I want to mull this over for a while. I'm not changing the rules by this, just how we express them. So please let me run with this before replying. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 14/05/2014 23:31 Subject: Re: [DFDL-WG] Fw: Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org I agree that the wording is not easy to get right. However, I think the current wording needs some adjustment so I'm going to make some suggestions and see where it leads. "A point of uncertainty occurs in the data stream when there is more than one schema component that might occur at that point." I don't think this is precise enough. SMH: Agree. I need to think about this sentence. There are several things potentially wrong. It is defining a PoU as occurring in the data stream, whereas elsewhere PoU is equated to a position in the model. It says 'more than one schema component that might occur' - maybe it should say 'a schema component may or may not occur'. And schema components don't occur in the data stream anyway - occurrences of them do. - if an optional element occurs at the end of the input data then there is only *one* schema component that might occur at that point. The end of the data stream might occur instead. SMH: Yes but I I raised some similar arguments earlier in the thread, about the last branch of a choice not being a PoU, or the last element in an unordered sequence when all the others had been found not being a PoU. We agreed that these are still all treated as PoUs for clarity. This is another example. - if an optional element occurs before the last required element in a sequence AND the separatorSuppressionPolicy is not 'anyEmpty' then there is exactly one schema component that can occur at that point in the data stream. But it might be 'empty', in which case it will not be put into the info set. This is not pedantry. The parser will never need to backtrack in either of these cases and in the second case it is obvious in advance which schema component the parser should select for parsing. SMH: We have agreed in the past that the presence of a separator is not enough to infer 'known-to-exist', so separators should not be brought into this definition. You are right that in a positional sequence the parser is looking for an occurrence of a component or its empty rep, and never an occurrence of the next schema component, so the parser can certainly optimise here. Let's take any discussion of separators out of this for the moment, and raise a separate action if needed. Points of uncertainty can be nested. Any one of the following constructs is a potential point of uncertainty: 1. An xs:choice 2. All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') 3. An optional xs:element 4. An array xs:element. 5. All xs:elements in an xs:sequence containing one or more floating xs:elements. 1. should say 'A member of an xs:choice' because it is the member, not the group itself, that is the point of uncertainty. I think the confusion has arisen because only one member of a choice group can exist in the data. So if any member exists, it automatically ends any speculation about the content of the choice group. But I insist that the real point of uncertainty is the member. A choice group is always 'known to exist' because according to DFDL rules it must have minOccurs=maxOccurs=1. FWIW, I have no problem with talking about 'resolving a choice', provided that we define that as 'Determining which member of a choice group ( if any ) is known to exist in the data'. SMH: I agree that it should say member. 2. Should say 'All members of an unordered xs:sequence' to keep the language consistent with 1. The section on unordered groups clearly restricts members to elements only. SMH: No. Using 'xs:element' is consistent with optionals & arrays in 3 and 4, which are also always elements. so xs:element is more consistent. 3. See above - an optional elements is not always a 'point of uncertainty' according to the literal definition that we are currently using. SMH; Right, but the bullets are defining potential PoUs, so it is correct as it stands. 4. Should say 'An optional occurrence of an array element, unless the separator properties make it a positional array and the occurrence is required in the data' SMH: No. All occurrences can be PoUs, it depends on OCK. And separators do not resolve PoUs as noted. This definition 4 is the one that is key for Action 248, which is ultimately what led to this discussion and what needs to be resolved. The question is whether 4 should say a) all arrays are potential PoUs as it does now, or b) just some arrays are potential PoUs depending on OCK. Whatever we choose, a discriminator within that array must not leak beyond the array as explained below in bold red font. I think a) is clearer and we can then make a general statement about discriminators not leaking outside of any potebtial PoU. If we adopt b) then we need a separate statement about discriminators and arrays, which seems more bitty. 5. Should say 'All members...' for consistency. SMH: See 3. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: , Date: 13/05/2014 10:28 Subject: [DFDL-WG] Fw: Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org This will be discussed on today's call. Please have a position on the paragraph below that ends 'What do others think?' Thanks Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/05/2014 10:19 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 30/04/2014 12:25 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Tim Responses below. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/04/2014 14:03 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org "2. If a potential point of uncertainty is sometimes an actual point of uncertainty (ock 'implicit') then a discriminator that applies it will only ever resolve, or have no effect on, that point of uncertainty. It never has an effect on any enclosing point of uncertainty." This could be misinterpreted. The discriminator could evaluate to 'false' and thus cause the POI to be resolved negatively ( the component would be 'known not to exist' ) SMH: Agree, and I can improve the words here. 1. and 3. will both apply if an element with ock='fixed' appears as a choice branch. Is the POI always an actual POI or never? SMH: No. There are two independent points of uncertainty, the choice branch and the array. The wording of 3. reads very strangely. 'If a potential point of uncertainty is never an actual point of uncertainty' begs the question 'why is it even a potential point of uncertainty?'. The current wording follows from our definition of the term 'point of uncertainty': "A point of uncertainty occurs in the data stream when there is more than one schema component that might occur at that point." Points of uncertainty can be nested. Any one of the following constructs is a potential point of uncertainty: 1. An xs:choice 2. All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') 3. An optional xs:element 4. An array xs:element. 5. All xs:elements in an xs:sequence containing one or more floating xs:elements. I think this definition is too broad. It forces us to discuss potential POUs that will never be actual POUs according to the first sentence. SMH: Yes it does read a bit strangely, but there's a reason for this. If we said that ock 'fixed', 'expression' or 'stopValue' are never POUs then what does it mean if a discriminator is placed on such an element? A discriminator gets evaluated for each occurrence of an array. For that reason we can not let a discriminator within an array leak beyond the array - regardless of whether it is a POU or not - otherwise what does that mean to enclosing POUs? So even if we said that ock 'fixed', 'expression' or 'stopValue' are never POUs we would still need the spec to state that a discriminator never leaks beyond them. I think it is clearer to say that a discriminator never leaks beyond a potential POU and keep the existing definition. What do others think? regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/04/2014 11:44 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org 248 Discriminators and potential points of uncertainty (Steve) 28/1: Steve to write up a proposal to prevent a discriminator from behaving in a non-obvious manner when used with a potential point of uncertainty that turns out not to be an actual point of uncertainty. 5/2: Steve sent an email to check whether choice branches, unordered elements and floating elements should always be actual points of uncertainty, as there are times when there is no uncertainty, eg, last choice branch; all floating elements found. It was decided that they are always actual points of uncertainty. To do otherwise will complicate implementations and result in fragile schemas. Steve will proceed with the proposal on that basis. Based on the above, which reflects the email discussion below, here is what I propose to resolve this action. 1. If a potential point of uncertainty is always an actual point of uncertainty (choice branch, element in unordered sequence, floating element, ock 'parsed') then a discriminator that applies to it will only ever resolve that point of uncertainty. It never has an effect on any enclosing point of uncertainty. 2. If a potential point of uncertainty is sometimes an actual point of uncertainty (ock 'implicit') then a discriminator that applies it will only ever resolve, or have no effect on, that point of uncertainty. It never has an effect on any enclosing point of uncertainty. 3. If a potential point of uncertainty is never an actual point of uncertainty (ock 'fixed', 'expression', 'stopValue') then a discriminator that applies to it will never have an effect on that point of uncertainty. Nor does it ever have an effect on any enclosing point of uncertainty. I think 1 and 2 are not controversial, but there is an alternative for 3: 3. If a potential point of uncertainty is never an actual point of uncertainty (ock 'fixed', 'expression', 'stopValue') then a discriminator that applies to it will never have an effect on that point of uncertainty. Instead the discriminator is applied to any enclosing point of uncertainty. The alternative means that changing an element from (say) ock 'parsed' to ock 'expression' has the same effect on a discriminator as changing the element to (1,1). The discriminator that applied to it now applies to any enclosing pou. SMH: Afternote: The alternative does not work for the reason given in my reply to Tim above. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 05/02/2014 12:04 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Thanks Tim, all good points. Comments to your comments. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 05/02/2014 11:01 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) A couple of comments below. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 05/02/2014 10:50 Subject: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org 248 Discriminators and potential points of uncertainty (Steve) 28/1: Steve to write up a proposal to prevent a discriminator from behaving in a non-obvious manner when used with a potential point of uncertainty that turns out not to be an actual point of uncertainty. 5/2: With Steve I started on this by reading section 9.3.3 on points of uncertainty, which lists the potential PoUs. Here's the list to save getting the spec out. 1. An xs:choice branch 2. All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') 3. An optional xs:element 4. An array xs:element 5. All xs:elements in an xs:sequence containing one or more floating xs:elements. The section then looks at each in turn and gives the circumstances when it is an actual PoU or not. As currently written, it is only 3 and 4 where a potential PoU might not be an actual PoU. For 1, 2 and 5 it says they are always actual PoUs. But I'm not sure that's correct. A deeper analysis of what is actually going on with 1, 2 and 5 says to me that there are times when there might not be an actual PoU. 1. Given that there is no concept in DFDL of optional choice branches, then if the last branch is reached then there is no longer a PoU. It must be that branch else it is a processing error. TK: I think of it slightly differently. It is a PoU, even if the branch is the only remaining branch. If we say that the final choice branch is not a PoU then diagnostics become confused - the parser reports the error code as 'error while parsing root/choice/lastBranch/field1' when the correct error code would be 'none of the branches of root/choice were found in the data'. SMH: I see your point. My thinking was that choices have finite branches and a choice is (1,1). If I have got to the last branch then I am not one of the other branches so I must be this one. If there is any other possibility then the model is missing a branch, even if it is just one that contains an empty sequence with an assert {fn:false()}. In practice of course users forget to add that last branch (there's no XSDL equivalent to the 'default' branch of a switch/case statement), so yes they could end up with an unclear diagnostic. 2. There can come a point in an unordered sequence when all that can be encountered is one element, and if that is (1,1) then there is no longer a PoU. TK: It's still a PoU. The specification says that occursCountKind is 'parsed' for all members of an unordered group, so min/maxOccurs do not come into play. SMH: Interesting. The spec says that if a member is optional or an array then it must be 'parsed'. If it is (1,1) though it does not have an occursCountKind. The specific case I was thinking of is when all members are (1,1), so when you have one element to go there is no PoU. However, the rewrite into a repeating choice has the effect of making everything 'parsed', which is really the point you are making. So I agree with you, it is easier to say that everything is an actual PoU else it complicates the rewrite semantic. 5. If all floating elements are (1,1) and all are encountered, then from that point on there are no longer any PoUs due to floating elements. TK: I suspect that floating elements are somewhat like unordered branches - most users will not want min/maxOccurs to affect the parsing of the group. Schema validation ( or more complex validation applied in the receiving application ) will deal with non-conformances. SMH: Possibly yes. With something like X12 NTE segments, that is the case. But we don't express the floating semantic as a rewrite of the whole sequence like we do for unordered, it's more of a per element thing. And if that is done dynamically as we go through the sequence, having no PoU can result. I'd like us to get straight on this before I proceed with the action proper. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/02/2014 10:12 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 27/01/2014 17:39 Subject: Fw: Thoughts on a discriminator scenario Been thinking some more on the discriminator scenario below that I mailed out before xmas, and discussing it with the IBM DFDL team. The 'confusing' aspect of the behaviour is that a discriminator within a potential PoU will act on a higher level PoU if the potential PoU is not an actual PoU. In the example, the array element 'Type1' is not an actual PoU for occurrence 1, only for occurrences 2+. So when the discriminator fires for occurrence 1 it will resolve a higher level unresolved PoU if one exists. Perhaps the spec should say that a discriminator can't 'leak' beyond the potential PoU that encloses it ? If so, then for occurrence 1 the discriminator has no effect, and only has an effect for occurrences 2+. This makes for more predictable and robust schemas. We'd need to go through spec section 9.3.3 carefully to see if this does not break any of the potential PoUs that are listed. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/01/2014 09:55 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 20/12/2013 13:20 Subject: Thoughts on a discriminator scenario Take the following schema (simplified) for element Type1 (1,10) being a loop for elements A,B,C. Type 1 does not have an initiator so I need to use a discriminator to establish the existence of an occurrence of Type1 so that incorrect backtracking does not occur after an error. Because occursCountKind is 'implicit', the 1st occurrence is not a point of uncertainty so the discriminator acts instead on any enclosing point of uncertainty, but for 2nd and subsequent occurrences it acts on Type1. That is all working as designed, but I think users find will the 1st occurrence behaviour a bit confusing. There are workarounds to avoid the problem, eg, use occursCountKind 'parsed' or split Type1 into two as (1,1) and (0,9). I think this is worth documenting in a tutorial as this is quite subtle stuff. <xs:element name="Type1" maxOccurs="10" dfdl:occursCountKind="implicit"> <dfdl:discriminator test="{fn:exists(A)}" /> <xs:complexType> <xs:sequence> <xs:element name="A" dfdl:initiator="A:" ... /> <xs:element name="B" dfdl:initiator="B:" ... /> <xs:element name="C" dfdl:initiator="C:"... /> </xs:sequence> </xs:complexType> Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 4

glossary entry for "fixed length" needed.
by Mike Beckerle 26 Nov '14

26 Nov '14

Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call Minutes 2014-11-25
by Steve Hanson 26 Nov '14

26 Nov '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13377?download= ** Updated DFDL 1.0 specification published at http://www.ogf.org/documents/GFD.207.pdf. ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 272: Catch 22.for EVDP & NVDP ?
by Steve Hanson 26 Nov '14

26 Nov '14

Spec updates in this tracker http://redmine.ogf.org/issues/245. Note that a fixed length element with length 0 can have an empty representation, it's only those that have non-zero length that can't. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 26/11/2014 10:28 ----- From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 18/11/2014 17:49 Subject: Re: [DFDL-WG] Action 272: Catch 22.for EVDP & NVDP ? Sent by: dfdl-wg-bounces(a)ogf.org Review for next call please. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 28/10/2014 14:48 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Thinking through the implications of having EVDP not apply if content length can not be zero: 1) Cases are dfdl:lengthKind 'explicit', dfdl:length not an expression and not zero, and (simple only) dfdl:lengthKind 'implicit' and maxLength facet not zero. 2) Does it mean a) that there is no concept of an empty representation, or b) that there is but it is not subject to EVDP and 'effective' EVDP is 'none' ? - If b) that does not fix the issue around initiatedContent, - if a) it means not finding the element must be taken as missing which implies no default value can be applied. Hmm 3) What about NVDP ? Should we be disallowing the use of %ES; as a nil value for these cases as well ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/09/2014 15:44 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Having thought this through I think that you are correct. If the content length can never be 0 then EVDP does not apply. I think IBM DFDL is issuing an error when it should not be. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 09/09/2014 15:37 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Interesting. To me if lengthKind is explicit, and non-variable (i.e., is fixed length), then EVDP doesn't apply, because by definition, such a representation is not ever empty. I would assume that EVDP would be ignored in such a case, and an implementation might even generate warnings if dfdl:emptyValueDelimiterPolicy appeared directly on such an element or simpleType where a fixed length is also expressed. This is one of the challenges that having lengthKind 'explicit' which handles both the fixed length, and variable-length using an expression, creates. Even though we don't have a lengthKind 'fixed', we still have fixed-length as a concept, and some other features interact with fixed length, like empty values. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 2, 2014 at 10:31 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: <xs:element name="choiceGroup" dfdl:lengthKind="implicit"> <xs:complexType> <xs:choice dfdl:choiceLengthKind="implicit" dfdl:initiatedContent="yes"> <xs:element name="myNum1" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:textNumberPattern="#0" dfdl:initiator="num:" dfdl:terminator="~" /> > <xs:element name="myText1" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:initiator="text:" dfdl:terminator="~" /> </xs:choice> </xs:complexType> </xs:element> The above model is a choice of two elements, each of which is fixed length and has an initiator and terminator. Because the elements are fixed length, I must set dfdl:emptyValueDelimiterPolicy (EVDP) to be 'none' **. I also want to discriminate using the initiator, so I set dfdl:initiatedContent 'yes' on the choice, so I must set EVDP to 'initiator' or 'both' ++. (Same for NVDP if nillable). Catch 22. ** 'none' because the only valid empty representation is that there is nothing there at all. The element is fixed length X, so if the delimiters are present then the content must be length X and can't be zero length. ++ 'initiator' or 'both' because an initiator must always be present when dfdl:initiatedContent 'yes'. The implication is that I can't use dfdl:initiatedContent for this choice. While I can switch off nillable to avoid the error with NVDP, there is no other way of avoiding the EVDP error. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Fw: Fw: Action 260
by Steve Hanson 25 Nov '14

25 Nov '14

http://redmine.ogf.org/issues/243 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 19/11/2014 10:55 Subject: Re: [DFDL-WG] Fw: Fw: Action 260 Agreed on the WG call that the behaviour in the table below will be adopted including the restrictions for OCK 'parsed'. List of spec changes as a result of this action are below. Note that the behaviour for OCK 'stopValue' in the table was not correct for unparsing. SSP 'anyEmpty' has a different behaviour than the others - it causes zero-length occurrences not to be output - which is fine. Please review so we can close on next call. 1) 14.2 - updates to paragraphs that describe positional and non-positional sequences Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. In DFDL, a sequence is considered positional if it contains only required elements and/or optional and array elements that have dfdl:occursCountKind 'implicit', 'fixed' or 'expression', and it has dfdl:separatorSuppressionPolicy 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Often the components of such a sequence have an initiator. Such sequences sometimes allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing might need to be employed by to identify each occurrence. In DFDL, a sequence is non-positional if it contains any optional or array elements that have dfdl:occursCountKind 'parsed' or 'stopValue', and/or it has dfdl:separatorSuppressionPolicy 'anyEmpty'. 2) 14.2.2 - updates to the last sentences of the 'When dfdlk:occursCountKind is ...' paragraphs to match the table and notes below. When an element is required and is not an array then one occurrence is always expected along with its separator. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). Otherwise the behaviour is dependent on dfdl:occursCountKind. When dfdl:occursCountKind is 'fixed' then XSDL minOccurs must equal maxOccurs and that many occurrences are always expected along with their separators. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). When dfdl:occursCountKind is 'expression' the number of occurrences is given by dfdl:occursCount and exactly that many occurrences are always expected along with their separators. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). When dfdl:occursCountKind is 'parsed' any number of occurrences and their separators are expected. The dfdl:separatorSuppressionPolicy of the sequence must be 'anyEmpty' and it is a schema definition error otherwise. When dfdl:occursCountKind is 'stopValue', any number of occurrences and their separators are expected followed by the stop value and its separator. The dfdl:separatorSuppressionPolicy of the sequence has no effect. When dfdl:occursCountKind is 'implicit', between XSDL minOccurs and maxOccurs (inclusive) occurrences and their separators are expected, according to the dfdl:separatorSuppressionPolicy of the sequence. 3) 14.2.3 - updates to the last sentences of the 'When dfdlk:occursCountKind is ...' paragraphs to match the table and notes below. When an element is required and is not an array then one occurrence is always output along with its separator. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). Otherwise the behaviour is dependent on dfdl:occursCountKind. When dfdl:occursCountKind is 'fixed' or 'expression' the occurrences in the augmented Infoset are always output along with their separators. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). When dfdl:occursCountKind is 'parsed' non zero-length occurrences in the augmented Infoset are output along with their separators. The dfdl:separatorSuppressionPolicy of the sequence must be 'anyEmpty' and it is a schema definition error otherwise. When dfdl:occursCountKind is 'stopValue' the occurrences in the augmented Infoset are output along with their separators followed by the stop value and its separator, according to the dfdl:separatorSuppressionPolicy of the sequence. When dfdl:occursCountKind is 'implicit' the occurrences in the augmented Infoset are output along with their separators, according to the dfdl:separatorSuppressionPolicy of the sequence. 4) 16 - update to occursStopValue property description The property is a list of logical values, so need to add: "The dfdl:stopValue property must not be empty string." Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 18/11/2014 12:23 Subject: Re: [DFDL-WG] Fw: Fw: Action 260 Mike Comments in-line. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/11/2014 23:11 Subject: Re: [DFDL-WG] Fw: Fw: Action 260 So I looked into what would be changed in the spec to make the adjustment suggested in this email thread, in intent. Rewriting the section is very undesirable at this stage for DFDL, so I was looking for incremental changes. What I came up with is this. In section 14.2.2 simply drop the sentences that mention this idea of an implied SSP, because they suggest the need to reconcile conflicting behaviors in the mixed case. I.e., drop all "The dfdl:separatorSuppressionPolicy is not applicable and the implied behaviour is '....'." With the omission of these sentences, the notion of an implied separator suppression policy that conflicts with one on the sequence goes away. There is only the SSP property of the sequence, and it is either relevant to the decision of whether to expect an item and its separator, or it isn't. The current descriptions in 14.2.2 of the behavior around suppression for each occursCountKind seem to be correct and match the table in this email thread, except for parsed. I'm not sure I agree with the assertion in this email thread that ock parsed only makes sense with "anyEmpty" behavior. We currently say in the spec that it has anyEmpty behavior, but if we drop that sentence (per suggestion above), then we would be loosening the behavior to allow empty elements in some cases. SMH: The intent of the table is that it is a schema definition error if 'parsed' and not 'anyEmpty' Suppose: <sequence dfdl:separatorSuppressionPolicy="trailingEmpty" dfdl:separator="|" dfdl:separatorPosition="infix" dfdl:terminator="%NL;"> <element name="a" type="xs:string" maxOccurs="unbounded" dfdl:occursCountKind='parsed'/> </sequence> In this case, the array is declared last in its sequence. The occurrences will all be element a, so this is positional. The number of them is determined by OCK parsed, and if enabled, validation will check, in this case, that at least 1 occurrence (the default for minOccurs) appears. In this case, empty string is a legal value, and we're not strict about trailing separators, so data like: 5|6|7|| is fine isn't it? The 'parsed' means you'll get 2 more empty string elements for the array when parsing that will not be re-created when unparsing, as they would be suppressed. I believe that is ok. There are many formats that can have that sort of asymmetry. SMH: To be clear, 'anyEmpty' already allows empty content when parsing, it has a lax semantic Change the above example to SSP trailingEmptyStrict, and now: 5,6,7||9 Now makes sense and you get one empty string in position 4. On unparsing this empty string would even get written out. I agree 'parsed' and SSP 'never' don't make sense together (as they don't for OCK implicit - SMH: only if maxOccurs 'unbounded' - see table 18 in 14.2.2), but the other 3 SSPs seem ok to me for declared-last elements. OCK parsed behaves (w.r.t. suppression, and ignoring defaulting and validation) just like OCK implicit with minOccurs 0 and maxOccurs 'unbounded'. (SMH: To be clear, defaulting and validation are independent of OCK). If we want to preserve the current restriction that parsed behaves like 'anyEmpty', then we can stipulate that when OCK is 'parsed', any number of non-empty occurrences and their separators are expected. (I would not be in favor of this.) SMH: So what you are saying is that 'parsed' is same as 'implicit' (0..unbounded). There are circumstances when this combination gives a schema definition error, as per Table 18 in 14.2.2 - this must therefore be the same for 'parsed'. It turns out that you only end up adding one extra legal behaviour for 'parsed', namely where element is declared last and SSP is 'trailingEmpty/Strict'. That's the example you quote above, but it's the only one. So is it buying much? SMH: When IBM put the table below together and said 'parsed' and SSP <> 'anyEmpty' is an error, it was to prevent a subtle change in behaviour for existing schemas with 'trailingEmpty/Strict'. Today the behaviour is 'anyEmpty', in the future it would be 'trailingEmpty/Strict'. Hence making it an error. As already noted in this thread, we also should add a sentence to stopValue: "The dfdl:stopValue property must not include empty string." The net result of these changes still isn't all that great, but it does remove one source of confusion - the 'implied' SSP conflict. SMH: If your 'parsed' proposal is adopted, you would need to document the 'parsed' behaviour as some combinations are schema definition errors. Arguably you would need the equivalent of Tables 18 and 19. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Mon, Oct 13, 2014 at 10:08 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: IBM has discussed this issue at some length internally, and come to the conclusion that the optional and array elements in a sequence should follow the separator suppression policy (SSP), and not have an implied SSP which is at odds with the sequence's SSP. In the table below, cells marked with a cross imply a schema definition error, and cells marked ok imply that there is a behaviour of the element for that OCK which is in keeping with the SSP of the sequence. SSP (1) OCK fixed implicit expression parsed stopValue (2) never ok ok ok x (5) ok (6) trailingEmpty trailingEmptyStrict ok (3) ok ok (4) x (5) ok (6) anyEmpty ok (3) ok ok (4) ok ok Notes: (1) SSP property applies only to an ordered sequence. An unordered sequence assumes 'anyEmpty' (as all optional/array elements must be 'parsed') (2) Missing restriction - for 'stopValue' the dfdl:stopValue property must not include empty string. (3) maxOccurs provides count, so nothing is eligible for suppression, so SSP has no practical effect (same as a required element) (4) infoset provides count, so nothing is eligible for suppression, so SSP has no practical effect (same as a required element) (5) 'parsed' only makes sense with 'anyEmpty' (6) Because a stop value must appear, and from (2) empty string is not allowed, SSP has no practical effect. The issue of maxOccurs = '0' is discussed in a separate email. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/10/2014 14:40 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 29/08/2014 21:21 Subject: Re: [DFDL-WG] Fw: Action 260 I reviewed this. It looks good to me. The note at the bottom that we don't say what happens on a zero-trip I.e., a represented element, but where occursCount evaluates to 0, is a useful clarification also. Do we want to create an erratum for this? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Thu, Aug 28, 2014 at 10:13 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Please review for Tuesday's WG call ... Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2014 15:02 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 06/08/2014 13:50 Subject: Fw: [DFDL-WG] Action 260 So my suggestion below, to wrap the array in a sequence, does not work; it just moves the problem down into the new sequence. After much deliberation, we think that the definitions of Positional sequence and Non-positional sequence should not be viewed as driving the behaviour of a sequence, but simply as the resultant characteristics of a sequence that has certain properties. That leaves modellers free to mix occursCountKinds, as in Tim's example. No need for any new SDE scenarios. Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. In DFDL, a sequence is considered positional if it contains only required elements and/or optional and array elements that have dfdl:occursCountKind 'implicit', 'fixed' or 'expression', and it has dfdl:separatorSuppressionPolicy 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Often the components of such a sequence have an initiator. Such sequences sometimes allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing might need to be employed by to identify each occurrence. In DFDL, a sequence is non-positional if it contains any optional or array elements that have dfdl:occursCountKind 'parsed' or 'stopValue', and/or it has dfdl:separatorSuppressionPolicy 'anyEmpty'. See parallel email for action 261 that ensures 'expression' behaves itself. One behaviour that is missing from the spec. For a sequence with separators, what is expected in the data stream if occursCount = 'fixed' / 'implicit' and maxOccurs = '0', or occursCountKind = 'expression' and occursCount evaluates to 0 ? We believe that no separator should be expected when parsing and none output when unparsing (same behaviour as inputValueCalc). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/08/2014 12:42 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 30/06/2014 10:04 Subject: Re: [DFDL-WG] Action 260 You would wrap the array and it's count in a sequence. Then the 'count+array' is treated as a single entity as far as the parent sequence is concerned. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 26/06/2014 20:06 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org Before we settle one way or the other, I would like the following data format to be taken into consideration. chars,5,A,B,C,D,E,integers,1,2,3 chars,3,C,,,integers,2,10,11 I am assuming that the occursCountKind for the arrays is 'expression' and the occursCount refers to the integer field that precedes the array. In order to represent the empty strings on the second line it is essential to specify SSP as 'trailingEmpty' or 'never'. If we disallow the combination of ock='expression' and SSP='trailingEmpty' then how would this format be modelled? regards, Tim Kimber, Technical Lead for IBM Integration Bus Healthcare Pack Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 25/06/2014 16:25 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org I prefer choice (a) for two reasons * It is more restrictive and therefore more conservative (preserving freedom to change in future if needed) * If a user has a positional data format, you don't want them to even have to understand the concept of speculation in order to model their data. So choice (a) allows a simpler description that doesn't need to introduce the notion that the parser might be speculation. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Jun 25, 2014 at 5:20 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: 260 Positional and non-positional sequences (All) 10/6: Spec defines the above but also allows different occursCountKinds within the same sequence which may have different (implied) separatorSuppressionPolicy, which results in a sequence which is a mixture of both. Should this be allowed? If so what are the rules? Can certain combinations be disallowed? 17/6: IBM have discussed internally and will submit a proposal. In the spec we define Positional Sequence and Non-Positional Sequence: Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. A positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Typically the components of such a sequence have an initiator. Such sequences allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing is employed by the parser to identify each occurrence. A non-positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'anyEmpty'. The problem is that the setting of dfdl:separatorSuppressionPolicy is only examined for child elements with dfdl:occursCountKind 'implicit'. For other dfdl:occursCountKinds, there is the concept of an 'implied' dfdl:separatorSuppressionPolicy: When dfdl:occursCountKind is 'fixed' then ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'expression' ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'parsed' ... the implied behaviour is 'anyEmpty'. When dfdl:occursCountKind is 'stopValue' ...the implied behaviour is 'anyEmpty'. So if a Positional sequence as defined above contains children with dfdl:occursCountKind 'parsed' or 'stopValue' then surely it is no longer a Positional sequence. A solution to this is to prevent the appearance of certain values of dfdl:occursCountKind within a Positional sequence. However, precisely which values to outlaw is subject to interpretation of the phrase "Each occurrence in the sequence can be identified by its position in the data". Is this intended to mean: a) an observer of the raw data can identify an occurrence of an element in the sequence solely by counting separators => SDE if 'parsed', 'stopValue' or 'expression' ** appeared in a Positional sequence; ** Although 'expression' would appear to be like 'fixed' it actually breaks a) so must be included in the SDE list. or b) a parser does not have to speculate to identify an occurrence of an element in the sequence => SDE only if 'parsed' appeared in a Positional sequence. Note that it is possible to wrap a 'parsed' etc element in a local sequence or another element to avoid an SDE. But this could still be seen as a violation of a) if the separators of both are the same, as the observer can not count the separators. So should the rule be applied recursively, ie, a Positional sequence can not contain a non-Positional sequence unless the separators are different? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 271: semantic of maxoccurs '0'
by Steve Hanson 25 Nov '14

25 Nov '14

271 Use of maxOccurs '0' (All) 2/9: Legal in XSDL and DFDL. One use case in XSDL is when deriving a complex type by restriction, where it is used to indicate that an element in the base type must not appear. Another use case could be if a schema undergoes a version revision that removes elements; perhaps clearer if the removed element gets maxOccurs '0' rather than omitting it. Steve has seen an instance of its use in a DFDL schema. So if DFDL continues to support it, need to document the behaviour for the various occursCountKinds and what happens to separator. Also behaviour of occursCount expression that returns 0. Proposal needed. ... 23/9: No progress IBM has discussed this internally and has the following proposal. Infoset refers to the augmented infoset. OCK maxOccurs '0' Parsing Unparsing fixed No occurrences looked for in the data (1) No occurrences looked for in the infoset or written (2) implicit No occurrences looked for in the data (1) No occurrences looked for in the infoset or written (2) expression occursCount occurrences expected. If any are found it is a validation error. Any number of occurrences expected in the infoset and written. If any are found it is a validation error. parsed Any number of occurrences expected in the data. If any are found it is a validation error. Any number of occurrences expected in the infoset and written. If any are found it is a validation error. stopValue Any number of occurrences expected in the data. If any are found it is a validation error. Any number of occurrences expected in the infoset and written. If any are found it is a validation error. OCK Parsing and occursCount '0' Unparsing and no occurrences in infoset expression No occurrences looked for in the data. (1) No occurrences written. (2) Notes (1) If sequence has a separator, no separator is looked for in the data when parsing. (2) If sequence has a separator, no separator is output when unparsing. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 4

OGF DFDL WG Call Agenda 2014-11-25
by Steve Hanson 24 Nov '14

24 Nov '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13377?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

dfdl github - mil-std-2045 schema updated
by Mike Beckerle 19 Nov '14

19 Nov '14

I finally got around to enhancing the mil-std-2045 schema using the ideas that came out of review of the original work. It is vastly improved in terms of complexity of the schema, and other schemas that have to generate this sort of thing are greatly simplified by the techniques illustrated here which avoid the need to generate any top-level group definitions. I did have to put in the workaround of using "WSP*" in a terminator instead of "ES" because Daffodil doesn't support ES in terminators yet (there's a bug). I also left out the defaults, which are needed for unparsing, due to a bug in Daffodil. https://github.com/DFDLSchemas/mil-std-2045 Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2014-11-18
by Steve Hanson 19 Nov '14

19 Nov '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13374?download= ** Updated DFDL 1.0 specification published at http://www.ogf.org/documents/GFD.207.pdf. ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Fw: Fw: Action 260
by Steve Hanson 19 Nov '14

19 Nov '14

Agreed on the WG call that the behaviour in the table below will be adopted including the restrictions for OCK 'parsed'. List of spec changes as a result of this action are below. Note that the behaviour for OCK 'stopValue' in the table was not correct for unparsing. SSP 'anyEmpty' has a different behaviour than the others - it causes zero-length occurrences not to be output - which is fine. Please review so we can close on next call. 1) 14.2 - updates to paragraphs that describe positional and non-positional sequences Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. In DFDL, a sequence is considered positional if it contains only required elements and/or optional and array elements that have dfdl:occursCountKind 'implicit', 'fixed' or 'expression', and it has dfdl:separatorSuppressionPolicy 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Often the components of such a sequence have an initiator. Such sequences sometimes allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing might need to be employed by to identify each occurrence. In DFDL, a sequence is non-positional if it contains any optional or array elements that have dfdl:occursCountKind 'parsed' or 'stopValue', and/or it has dfdl:separatorSuppressionPolicy 'anyEmpty'. 2) 14.2.2 - updates to the last sentences of the 'When dfdlk:occursCountKind is ...' paragraphs to match the table and notes below. When an element is required and is not an array then one occurrence is always expected along with its separator. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). Otherwise the behaviour is dependent on dfdl:occursCountKind. When dfdl:occursCountKind is 'fixed' then XSDL minOccurs must equal maxOccurs and that many occurrences are always expected along with their separators. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). When dfdl:occursCountKind is 'expression' the number of occurrences is given by dfdl:occursCount and exactly that many occurrences are always expected along with their separators. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). When dfdl:occursCountKind is 'parsed' any number of occurrences and their separators are expected. The dfdl:separatorSuppressionPolicy of the sequence must be 'anyEmpty' and it is a schema definition error otherwise. When dfdl:occursCountKind is 'stopValue', any number of occurrences and their separators are expected followed by the stop value and its separator. The dfdl:separatorSuppressionPolicy of the sequence has no effect. When dfdl:occursCountKind is 'implicit', between XSDL minOccurs and maxOccurs (inclusive) occurrences and their separators are expected, according to the dfdl:separatorSuppressionPolicy of the sequence. 3) 14.2.3 - updates to the last sentences of the 'When dfdlk:occursCountKind is ...' paragraphs to match the table and notes below. When an element is required and is not an array then one occurrence is always output along with its separator. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). Otherwise the behaviour is dependent on dfdl:occursCountKind. When dfdl:occursCountKind is 'fixed' or 'expression' the occurrences in the augmented Infoset are always output along with their separators. The dfdl:separatorSuppressionPolicy of the sequence has no effect (nothing is eligible for suppression). When dfdl:occursCountKind is 'parsed' non zero-length occurrences in the augmented Infoset are output along with their separators. The dfdl:separatorSuppressionPolicy of the sequence must be 'anyEmpty' and it is a schema definition error otherwise. When dfdl:occursCountKind is 'stopValue' the occurrences in the augmented Infoset are output along with their separators followed by the stop value and its separator, according to the dfdl:separatorSuppressionPolicy of the sequence. When dfdl:occursCountKind is 'implicit' the occurrences in the augmented Infoset are output along with their separators, according to the dfdl:separatorSuppressionPolicy of the sequence. 4) 16 - update to occursStopValue property description The property is a list of logical values, so need to add: "The dfdl:stopValue property must not be empty string." Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 18/11/2014 12:23 Subject: Re: [DFDL-WG] Fw: Fw: Action 260 Mike Comments in-line. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/11/2014 23:11 Subject: Re: [DFDL-WG] Fw: Fw: Action 260 So I looked into what would be changed in the spec to make the adjustment suggested in this email thread, in intent. Rewriting the section is very undesirable at this stage for DFDL, so I was looking for incremental changes. What I came up with is this. In section 14.2.2 simply drop the sentences that mention this idea of an implied SSP, because they suggest the need to reconcile conflicting behaviors in the mixed case. I.e., drop all "The dfdl:separatorSuppressionPolicy is not applicable and the implied behaviour is '....'." With the omission of these sentences, the notion of an implied separator suppression policy that conflicts with one on the sequence goes away. There is only the SSP property of the sequence, and it is either relevant to the decision of whether to expect an item and its separator, or it isn't. The current descriptions in 14.2.2 of the behavior around suppression for each occursCountKind seem to be correct and match the table in this email thread, except for parsed. I'm not sure I agree with the assertion in this email thread that ock parsed only makes sense with "anyEmpty" behavior. We currently say in the spec that it has anyEmpty behavior, but if we drop that sentence (per suggestion above), then we would be loosening the behavior to allow empty elements in some cases. SMH: The intent of the table is that it is a schema definition error if 'parsed' and not 'anyEmpty' Suppose: <sequence dfdl:separatorSuppressionPolicy="trailingEmpty" dfdl:separator="|" dfdl:separatorPosition="infix" dfdl:terminator="%NL;"> <element name="a" type="xs:string" maxOccurs="unbounded" dfdl:occursCountKind='parsed'/> </sequence> In this case, the array is declared last in its sequence. The occurrences will all be element a, so this is positional. The number of them is determined by OCK parsed, and if enabled, validation will check, in this case, that at least 1 occurrence (the default for minOccurs) appears. In this case, empty string is a legal value, and we're not strict about trailing separators, so data like: 5|6|7|| is fine isn't it? The 'parsed' means you'll get 2 more empty string elements for the array when parsing that will not be re-created when unparsing, as they would be suppressed. I believe that is ok. There are many formats that can have that sort of asymmetry. SMH: To be clear, 'anyEmpty' already allows empty content when parsing, it has a lax semantic Change the above example to SSP trailingEmptyStrict, and now: 5,6,7||9 Now makes sense and you get one empty string in position 4. On unparsing this empty string would even get written out. I agree 'parsed' and SSP 'never' don't make sense together (as they don't for OCK implicit - SMH: only if maxOccurs 'unbounded' - see table 18 in 14.2.2), but the other 3 SSPs seem ok to me for declared-last elements. OCK parsed behaves (w.r.t. suppression, and ignoring defaulting and validation) just like OCK implicit with minOccurs 0 and maxOccurs 'unbounded'. (SMH: To be clear, defaulting and validation are independent of OCK). If we want to preserve the current restriction that parsed behaves like 'anyEmpty', then we can stipulate that when OCK is 'parsed', any number of non-empty occurrences and their separators are expected. (I would not be in favor of this.) SMH: So what you are saying is that 'parsed' is same as 'implicit' (0..unbounded). There are circumstances when this combination gives a schema definition error, as per Table 18 in 14.2.2 - this must therefore be the same for 'parsed'. It turns out that you only end up adding one extra legal behaviour for 'parsed', namely where element is declared last and SSP is 'trailingEmpty/Strict'. That's the example you quote above, but it's the only one. So is it buying much? SMH: When IBM put the table below together and said 'parsed' and SSP <> 'anyEmpty' is an error, it was to prevent a subtle change in behaviour for existing schemas with 'trailingEmpty/Strict'. Today the behaviour is 'anyEmpty', in the future it would be 'trailingEmpty/Strict'. Hence making it an error. As already noted in this thread, we also should add a sentence to stopValue: "The dfdl:stopValue property must not include empty string." The net result of these changes still isn't all that great, but it does remove one source of confusion - the 'implied' SSP conflict. SMH: If your 'parsed' proposal is adopted, you would need to document the 'parsed' behaviour as some combinations are schema definition errors. Arguably you would need the equivalent of Tables 18 and 19. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Mon, Oct 13, 2014 at 10:08 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: IBM has discussed this issue at some length internally, and come to the conclusion that the optional and array elements in a sequence should follow the separator suppression policy (SSP), and not have an implied SSP which is at odds with the sequence's SSP. In the table below, cells marked with a cross imply a schema definition error, and cells marked ok imply that there is a behaviour of the element for that OCK which is in keeping with the SSP of the sequence. SSP (1) OCK fixed implicit expression parsed stopValue (2) never ok ok ok x (5) ok (6) trailingEmpty trailingEmptyStrict ok (3) ok ok (4) x (5) ok (6) anyEmpty ok (3) ok ok (4) ok ok Notes: (1) SSP property applies only to an ordered sequence. An unordered sequence assumes 'anyEmpty' (as all optional/array elements must be 'parsed') (2) Missing restriction - for 'stopValue' the dfdl:stopValue property must not include empty string. (3) maxOccurs provides count, so nothing is eligible for suppression, so SSP has no practical effect (same as a required element) (4) infoset provides count, so nothing is eligible for suppression, so SSP has no practical effect (same as a required element) (5) 'parsed' only makes sense with 'anyEmpty' (6) Because a stop value must appear, and from (2) empty string is not allowed, SSP has no practical effect. The issue of maxOccurs = '0' is discussed in a separate email. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/10/2014 14:40 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 29/08/2014 21:21 Subject: Re: [DFDL-WG] Fw: Action 260 I reviewed this. It looks good to me. The note at the bottom that we don't say what happens on a zero-trip I.e., a represented element, but where occursCount evaluates to 0, is a useful clarification also. Do we want to create an erratum for this? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Thu, Aug 28, 2014 at 10:13 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Please review for Tuesday's WG call ... Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2014 15:02 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 06/08/2014 13:50 Subject: Fw: [DFDL-WG] Action 260 So my suggestion below, to wrap the array in a sequence, does not work; it just moves the problem down into the new sequence. After much deliberation, we think that the definitions of Positional sequence and Non-positional sequence should not be viewed as driving the behaviour of a sequence, but simply as the resultant characteristics of a sequence that has certain properties. That leaves modellers free to mix occursCountKinds, as in Tim's example. No need for any new SDE scenarios. Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. In DFDL, a sequence is considered positional if it contains only required elements and/or optional and array elements that have dfdl:occursCountKind 'implicit', 'fixed' or 'expression', and it has dfdl:separatorSuppressionPolicy 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Often the components of such a sequence have an initiator. Such sequences sometimes allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing might need to be employed by to identify each occurrence. In DFDL, a sequence is non-positional if it contains any optional or array elements that have dfdl:occursCountKind 'parsed' or 'stopValue', and/or it has dfdl:separatorSuppressionPolicy 'anyEmpty'. See parallel email for action 261 that ensures 'expression' behaves itself. One behaviour that is missing from the spec. For a sequence with separators, what is expected in the data stream if occursCount = 'fixed' / 'implicit' and maxOccurs = '0', or occursCountKind = 'expression' and occursCount evaluates to 0 ? We believe that no separator should be expected when parsing and none output when unparsing (same behaviour as inputValueCalc). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/08/2014 12:42 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 30/06/2014 10:04 Subject: Re: [DFDL-WG] Action 260 You would wrap the array and it's count in a sequence. Then the 'count+array' is treated as a single entity as far as the parent sequence is concerned. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 26/06/2014 20:06 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org Before we settle one way or the other, I would like the following data format to be taken into consideration. chars,5,A,B,C,D,E,integers,1,2,3 chars,3,C,,,integers,2,10,11 I am assuming that the occursCountKind for the arrays is 'expression' and the occursCount refers to the integer field that precedes the array. In order to represent the empty strings on the second line it is essential to specify SSP as 'trailingEmpty' or 'never'. If we disallow the combination of ock='expression' and SSP='trailingEmpty' then how would this format be modelled? regards, Tim Kimber, Technical Lead for IBM Integration Bus Healthcare Pack Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 25/06/2014 16:25 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org I prefer choice (a) for two reasons * It is more restrictive and therefore more conservative (preserving freedom to change in future if needed) * If a user has a positional data format, you don't want them to even have to understand the concept of speculation in order to model their data. So choice (a) allows a simpler description that doesn't need to introduce the notion that the parser might be speculation. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Jun 25, 2014 at 5:20 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: 260 Positional and non-positional sequences (All) 10/6: Spec defines the above but also allows different occursCountKinds within the same sequence which may have different (implied) separatorSuppressionPolicy, which results in a sequence which is a mixture of both. Should this be allowed? If so what are the rules? Can certain combinations be disallowed? 17/6: IBM have discussed internally and will submit a proposal. In the spec we define Positional Sequence and Non-Positional Sequence: Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. A positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Typically the components of such a sequence have an initiator. Such sequences allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing is employed by the parser to identify each occurrence. A non-positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'anyEmpty'. The problem is that the setting of dfdl:separatorSuppressionPolicy is only examined for child elements with dfdl:occursCountKind 'implicit'. For other dfdl:occursCountKinds, there is the concept of an 'implied' dfdl:separatorSuppressionPolicy: When dfdl:occursCountKind is 'fixed' then ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'expression' ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'parsed' ... the implied behaviour is 'anyEmpty'. When dfdl:occursCountKind is 'stopValue' ...the implied behaviour is 'anyEmpty'. So if a Positional sequence as defined above contains children with dfdl:occursCountKind 'parsed' or 'stopValue' then surely it is no longer a Positional sequence. A solution to this is to prevent the appearance of certain values of dfdl:occursCountKind within a Positional sequence. However, precisely which values to outlaw is subject to interpretation of the phrase "Each occurrence in the sequence can be identified by its position in the data". Is this intended to mean: a) an observer of the raw data can identify an occurrence of an element in the sequence solely by counting separators => SDE if 'parsed', 'stopValue' or 'expression' ** appeared in a Positional sequence; ** Although 'expression' would appear to be like 'fixed' it actually breaks a) so must be included in the SDE list. or b) a parser does not have to speculate to identify an occurrence of an element in the sequence => SDE only if 'parsed' appeared in a Positional sequence. Note that it is possible to wrap a 'parsed' etc element in a local sequence or another element to avoid an SDE. But this could still be seen as a violation of a) if the separators of both are the same, as the observer can not count the separators. So should the rule be applied recursively, ie, a Positional sequence can not contain a non-Positional sequence unless the separators are different? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 272: Catch 22.for EVDP & NVDP ?
by Steve Hanson 19 Nov '14

19 Nov '14

It goes beyond the applicability of that one property. That's what my point 2) below is exploring. If we say that an element can never have an empty rep then (for example) it can never be assigned a default value on parsing. I am asking the WG to check whether there are any gotchas here. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 18/11/2014 21:49 Subject: Re: [DFDL-WG] Action 272: Catch 22.for EVDP & NVDP ? Intuitively, if an element cannot have an empty value then a property with name 'Empty value delimiter policy' cannot apply to it. I must be missing something, but I don't know what it is. Can you supply some examples to illustrate the edge cases? regards, Tim Kimber, From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 18/11/2014 17:49 Subject: Re: [DFDL-WG] Action 272: Catch 22.for EVDP & NVDP ? Sent by: dfdl-wg-bounces(a)ogf.org Review for next call please. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 28/10/2014 14:48 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Thinking through the implications of having EVDP not apply if content length can not be zero: 1) Cases are dfdl:lengthKind 'explicit', dfdl:length not an expression and not zero, and (simple only) dfdl:lengthKind 'implicit' and maxLength facet not zero. 2) Does it mean a) that there is no concept of an empty representation, or b) that there is but it is not subject to EVDP and 'effective' EVDP is 'none' ? - If b) that does not fix the issue around initiatedContent, - if a) it means not finding the element must be taken as missing which implies no default value can be applied. Hmm 3) What about NVDP ? Should we be disallowing the use of %ES; as a nil value for these cases as well ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/09/2014 15:44 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Having thought this through I think that you are correct. If the content length can never be 0 then EVDP does not apply. I think IBM DFDL is issuing an error when it should not be. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 09/09/2014 15:37 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Interesting. To me if lengthKind is explicit, and non-variable (i.e., is fixed length), then EVDP doesn't apply, because by definition, such a representation is not ever empty. I would assume that EVDP would be ignored in such a case, and an implementation might even generate warnings if dfdl:emptyValueDelimiterPolicy appeared directly on such an element or simpleType where a fixed length is also expressed. This is one of the challenges that having lengthKind 'explicit' which handles both the fixed length, and variable-length using an expression, creates. Even though we don't have a lengthKind 'fixed', we still have fixed-length as a concept, and some other features interact with fixed length, like empty values. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 2, 2014 at 10:31 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: <xs:element name="choiceGroup" dfdl:lengthKind="implicit"> <xs:complexType> <xs:choice dfdl:choiceLengthKind="implicit" dfdl:initiatedContent="yes"> <xs:element name="myNum1" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:textNumberPattern="#0" dfdl:initiator="num:" dfdl:terminator="~" /> > <xs:element name="myText1" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:initiator="text:" dfdl:terminator="~" /> </xs:choice> </xs:complexType> </xs:element> The above model is a choice of two elements, each of which is fixed length and has an initiator and terminator. Because the elements are fixed length, I must set dfdl:emptyValueDelimiterPolicy (EVDP) to be 'none' **. I also want to discriminate using the initiator, so I set dfdl:initiatedContent 'yes' on the choice, so I must set EVDP to 'initiator' or 'both' ++. (Same for NVDP if nillable). Catch 22. ** 'none' because the only valid empty representation is that there is nothing there at all. The element is fixed length X, so if the delimiters are present then the content must be length X and can't be zero length. ++ 'initiator' or 'both' because an initiator must always be present when dfdl:initiatedContent 'yes'. The implication is that I can't use dfdl:initiatedContent for this choice. While I can switch off nillable to avoid the error with NVDP, there is no other way of avoiding the EVDP error. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 272: Catch 22.for EVDP & NVDP ?
by Steve Hanson 18 Nov '14

18 Nov '14

Review for next call please. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 28/10/2014 14:48 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Thinking through the implications of having EVDP not apply if content length can not be zero: 1) Cases are dfdl:lengthKind 'explicit', dfdl:length not an expression and not zero, and (simple only) dfdl:lengthKind 'implicit' and maxLength facet not zero. 2) Does it mean a) that there is no concept of an empty representation, or b) that there is but it is not subject to EVDP and 'effective' EVDP is 'none' ? - If b) that does not fix the issue around initiatedContent, - if a) it means not finding the element must be taken as missing which implies no default value can be applied. Hmm 3) What about NVDP ? Should we be disallowing the use of %ES; as a nil value for these cases as well ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 17/09/2014 15:44 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Having thought this through I think that you are correct. If the content length can never be 0 then EVDP does not apply. I think IBM DFDL is issuing an error when it should not be. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 09/09/2014 15:37 Subject: Re: [DFDL-WG] Catch 22.for EVDP & NVDP ? Interesting. To me if lengthKind is explicit, and non-variable (i.e., is fixed length), then EVDP doesn't apply, because by definition, such a representation is not ever empty. I would assume that EVDP would be ignored in such a case, and an implementation might even generate warnings if dfdl:emptyValueDelimiterPolicy appeared directly on such an element or simpleType where a fixed length is also expressed. This is one of the challenges that having lengthKind 'explicit' which handles both the fixed length, and variable-length using an expression, creates. Even though we don't have a lengthKind 'fixed', we still have fixed-length as a concept, and some other features interact with fixed length, like empty values. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 2, 2014 at 10:31 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: <xs:element name="choiceGroup" dfdl:lengthKind="implicit"> <xs:complexType> <xs:choice dfdl:choiceLengthKind="implicit" dfdl:initiatedContent="yes"> <xs:element name="myNum1" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:textNumberPattern="#0" dfdl:initiator="num:" dfdl:terminator="~" /> > <xs:element name="myText1" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:initiator="text:" dfdl:terminator="~" /> </xs:choice> </xs:complexType> </xs:element> The above model is a choice of two elements, each of which is fixed length and has an initiator and terminator. Because the elements are fixed length, I must set dfdl:emptyValueDelimiterPolicy (EVDP) to be 'none' **. I also want to discriminate using the initiator, so I set dfdl:initiatedContent 'yes' on the choice, so I must set EVDP to 'initiator' or 'both' ++. (Same for NVDP if nillable). Catch 22. ** 'none' because the only valid empty representation is that there is nothing there at all. The element is fixed length X, so if the delimiters are present then the content must be length X and can't be zero length. ++ 'initiator' or 'both' because an initiator must always be present when dfdl:initiatedContent 'yes'. The implication is that I can't use dfdl:initiatedContent for this choice. While I can switch off nillable to avoid the error with NVDP, there is no other way of avoiding the EVDP error. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Re: [DFDL-WG] Action 274 - on qnames in path expressions in DFDL
by Tim Kimber 18 Nov '14

18 Nov '14

You're right - I did not realize that this 'no undeclaring of a non-empty prefix' rule existed. http://www.w3.org/TR/REC-xml-names/#nsc-NoPrefixUndecl So this is not valid: <foo:root xmlns:foo="www.tk.com"> <foo:child xmlns:foo=""/> </foo:root> but this is valid: <root xmlns="www.tk.com"> <child xmlns=""/> </root> So there is a way to change the default namespace from "someURL" to "". And it should therefore be possible to re-assign the default namespace in an element reference so that it can refer to a global element declared in a noTargetNamespace xsd. regards, Tim Kimber, Technical Lead for IBM Integration Bus Healthcare Pack Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 03/11/2014 14:38 Subject: Re: [DFDL-WG] Action 274 - on qnames in path expressions in DFDL Tim, I'm not sure that you comparing apples with apples here. Mike is asking whether xmlns:foo="" is valid, whereas your stackoverflow link is talking about xmlns="" and it's specific use in an XML document to undeclare the namespace in scope. Are you sure that xmlns:foo="" is valid ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 17/10/2014 03:55 Subject: Re: [DFDL-WG] Action 274 - on qnames in path expressions in DFDL Sent by: dfdl-wg-bounces(a)ogf.org I think Eclipse must be doing something non-standard then: http://stackoverflow.com/questions/1587891/is-xmlns-a-valid-xml-namespace Which version of Eclipse / Eclipse plugin are you using? regards, Tim Kimber, Technical Lead for IBM Integration Bus Healthcare Pack Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Tim Kimber/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 16/10/2014 14:52 Subject: Re: [DFDL-WG] Action 274 - on qnames in path expressions in DFDL Tim, How do you do this? => "You define a non-empty prefix that points to an empty namespace, and use that prefix in the QName." specifically, what XML text corresponds to "an empty namespace". xmlns:foo="" isn't allowed (in eclipse anyway) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Thu, Oct 16, 2014 at 12:58 AM, Tim Kimber <KIMBERT(a)uk.ibm.com> wrote: My off-the-cuff reactions: - every conforming XPath processor allows a 'namespace context' to be defined. The context defines a prefix->namespace URI mapping for zero or more prefixes. - namespace prefixes ( including the empty prefix ) are then interpreted in exactly the same way as they would be in an XML document. This makes sense, because XPaths were explicitly designed to be included within XML documents. It would look rather strange if the prefixes within an XPath were to be interpreted differently by a) an XML processor and b) an XPath processor. I think that answers most of the queries, although it raises a question over whether the Eclipse implementation is conforming. re: the question about empty prefixes, there is a (non-obvious) way to refer to a noTargetNamespace global element from within a document that has a default namespace. You define a non-empty prefix that points to an empty namespace, and use that prefix in the QName. regards, Tim Kimber, Technical Lead for IBM Integration Bus Healthcare Pack Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 16/10/2014 01:41 Subject: [DFDL-WG] Action 274 - on qnames in path expressions in DFDL Sent by: dfdl-wg-bounces(a)ogf.org I did some experiments with the XML validation built into eclipse, and the key constraint, which has paths (in the selector and field attributes) that refer to other parts of the XML infoset/document and so involve the names of elements. I also have been reading the XPath 2.0 specs including the formal semantics. To my interpretation XPath 2.0 is pretty explicit: An element or type QName consisting only of a local part NCName expands to the default element/type namespace and the local part. They don't qualify this in any way that I can find for the names used in path steps. The formal semantics is pretty hard to understand, but this detail seems to be swept under the rug. There would need to be specific treatment of this in the naming environment, but I cannot find this. There is discussion of name resolution in some of their judgements, but nothing I can see that creates an exception for names in path steps. (Called name tests) What I was looking for was language that specifies something about a non-prefixed name in a path step and that it is NOT augmented by the default namespace. I could find no such exception. But if I am right and this detail isn't there, i.e., If there is no such exception, then the implementation of XPath in eclipse (which is xerces I believe) is incorrect in the way it resolves names that appear in path steps. Here's how it works according to eclipse: In xpath expressions in the selectors and fields of unique and key constraints, xmlns (the default namespace) is not considered as implicitly augmenting unqualified names that appear in path steps. An identifier in a path step that has no prefix (e.g., 'foo' in the path './p1:bar/foo/p2:baz') is always interpreted as identifying either 1) a child element defined by a local element declaration with form "unqualified". or 2) this corner case: a child element defined by an element reference to a global element declaration but only when that global element declaration has no effective target namespace. (Use of 'effective' here rules out an element decl inside a schema document that itself has no target namespace but is included into a schema that has one. Such an element declaration effectively has a target namespace, so this corner case doesn't apply.) Case (2) here is a rather obscure corner case, but there nevertheless. In no case (that I could construct) does the name in a path step get augmented by a schema's default namespace (defined by xmlns="...whatever..."). The above is true for names without prefix appearing in path steps of expressions. The QNames that are used in an XML schema to reference a global declaration or definition (in the ref of an element ref, type attribute in a type reference, base in a simple type derivation, etc.) are different. Those names DO pickup a default namespace if one is defined and they have no explicit prefix themselves. This creates a corner case for these references (to global decls/defs) which is if a definition/declaration appears in no effective target namespace, then there is no way to reference it at all from a schema that has a default namespace definition surrounding the point of reference. There is no syntactic way to turn-off use of a default namespace. (The semantics of xpath allows for a "none" default namespace definition, but there is no xml syntax for expressing explicitly this "none". It is simply what you have if there is no prefix and no default namespace). You can change which namespace the default is, but there's no way to say explicitly "I meant this QName reference to be to something not in any namespace. Unless/until we can find language in some XML Schema spec indicating how path step names (name tests) are specifically resolved, then the above is just "how one system does it." But we may end up having to follow precedent here from implementations. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Fw: Fw: Action 260
by Steve Hanson 18 Nov '14

18 Nov '14

IBM has discussed this issue at some length internally, and come to the conclusion that the optional and array elements in a sequence should follow the separator suppression policy (SSP), and not have an implied SSP which is at odds with the sequence's SSP. In the table below, cells marked with a cross imply a schema definition error, and cells marked ok imply that there is a behaviour of the element for that OCK which is in keeping with the SSP of the sequence. SSP (1) OCK fixed implicit expression parsed stopValue (2) never ok ok ok x (5) ok (6) trailingEmpty trailingEmptyStrict ok (3) ok ok (4) x (5) ok (6) anyEmpty ok (3) ok ok (4) ok ok Notes: (1) SSP property applies only to an ordered sequence. An unordered sequence assumes 'anyEmpty' (as all optional/array elements must be 'parsed') (2) Missing restriction - for 'stopValue' the dfdl:stopValue property must not include empty string. (3) maxOccurs provides count, so nothing is eligible for suppression, so SSP has no practical effect (same as a required element) (4) infoset provides count, so nothing is eligible for suppression, so SSP has no practical effect (same as a required element) (5) 'parsed' only makes sense with 'anyEmpty' (6) Because a stop value must appear, and from (2) empty string is not allowed, SSP has no practical effect. The issue of maxOccurs = '0' is discussed in a separate email. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 13/10/2014 14:40 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 29/08/2014 21:21 Subject: Re: [DFDL-WG] Fw: Action 260 I reviewed this. It looks good to me. The note at the bottom that we don't say what happens on a zero-trip I.e., a represented element, but where occursCount evaluates to 0, is a useful clarification also. Do we want to create an erratum for this? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Thu, Aug 28, 2014 at 10:13 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Please review for Tuesday's WG call ... Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2014 15:02 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 06/08/2014 13:50 Subject: Fw: [DFDL-WG] Action 260 So my suggestion below, to wrap the array in a sequence, does not work; it just moves the problem down into the new sequence. After much deliberation, we think that the definitions of Positional sequence and Non-positional sequence should not be viewed as driving the behaviour of a sequence, but simply as the resultant characteristics of a sequence that has certain properties. That leaves modellers free to mix occursCountKinds, as in Tim's example. No need for any new SDE scenarios. Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. In DFDL, a sequence is considered positional if it contains only required elements and/or optional and array elements that have dfdl:occursCountKind 'implicit', 'fixed' or 'expression', and it has dfdl:separatorSuppressionPolicy 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Often the components of such a sequence have an initiator. Such sequences sometimes allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing might need to be employed by to identify each occurrence. In DFDL, a sequence is non-positional if it contains any optional or array elements that have dfdl:occursCountKind 'parsed' or 'stopValue', and/or it has dfdl:separatorSuppressionPolicy 'anyEmpty'. See parallel email for action 261 that ensures 'expression' behaves itself. One behaviour that is missing from the spec. For a sequence with separators, what is expected in the data stream if occursCount = 'fixed' / 'implicit' and maxOccurs = '0', or occursCountKind = 'expression' and occursCount evaluates to 0 ? We believe that no separator should be expected when parsing and none output when unparsing (same behaviour as inputValueCalc). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/08/2014 12:42 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 30/06/2014 10:04 Subject: Re: [DFDL-WG] Action 260 You would wrap the array and it's count in a sequence. Then the 'count+array' is treated as a single entity as far as the parent sequence is concerned. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 26/06/2014 20:06 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org Before we settle one way or the other, I would like the following data format to be taken into consideration. chars,5,A,B,C,D,E,integers,1,2,3 chars,3,C,,,integers,2,10,11 I am assuming that the occursCountKind for the arrays is 'expression' and the occursCount refers to the integer field that precedes the array. In order to represent the empty strings on the second line it is essential to specify SSP as 'trailingEmpty' or 'never'. If we disallow the combination of ock='expression' and SSP='trailingEmpty' then how would this format be modelled? regards, Tim Kimber, Technical Lead for IBM Integration Bus Healthcare Pack Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 25/06/2014 16:25 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org I prefer choice (a) for two reasons * It is more restrictive and therefore more conservative (preserving freedom to change in future if needed) * If a user has a positional data format, you don't want them to even have to understand the concept of speculation in order to model their data. So choice (a) allows a simpler description that doesn't need to introduce the notion that the parser might be speculation. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Jun 25, 2014 at 5:20 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: 260 Positional and non-positional sequences (All) 10/6: Spec defines the above but also allows different occursCountKinds within the same sequence which may have different (implied) separatorSuppressionPolicy, which results in a sequence which is a mixture of both. Should this be allowed? If so what are the rules? Can certain combinations be disallowed? 17/6: IBM have discussed internally and will submit a proposal. In the spec we define Positional Sequence and Non-Positional Sequence: Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. A positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Typically the components of such a sequence have an initiator. Such sequences allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing is employed by the parser to identify each occurrence. A non-positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'anyEmpty'. The problem is that the setting of dfdl:separatorSuppressionPolicy is only examined for child elements with dfdl:occursCountKind 'implicit'. For other dfdl:occursCountKinds, there is the concept of an 'implied' dfdl:separatorSuppressionPolicy: When dfdl:occursCountKind is 'fixed' then ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'expression' ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'parsed' ... the implied behaviour is 'anyEmpty'. When dfdl:occursCountKind is 'stopValue' ...the implied behaviour is 'anyEmpty'. So if a Positional sequence as defined above contains children with dfdl:occursCountKind 'parsed' or 'stopValue' then surely it is no longer a Positional sequence. A solution to this is to prevent the appearance of certain values of dfdl:occursCountKind within a Positional sequence. However, precisely which values to outlaw is subject to interpretation of the phrase "Each occurrence in the sequence can be identified by its position in the data". Is this intended to mean: a) an observer of the raw data can identify an occurrence of an element in the sequence solely by counting separators => SDE if 'parsed', 'stopValue' or 'expression' ** appeared in a Positional sequence; ** Although 'expression' would appear to be like 'fixed' it actually breaks a) so must be included in the SDE list. or b) a parser does not have to speculate to identify an occurrence of an element in the sequence => SDE only if 'parsed' appeared in a Positional sequence. Note that it is possible to wrap a 'parsed' etc element in a local sequence or another element to avoid an SDE. But this could still be seen as a violation of a) if the separators of both are the same, as the observer can not count the separators. So should the rule be applied recursively, ie, a Positional sequence can not contain a non-Positional sequence unless the separators are different? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

do we allow indexing of non-array, non-optional
by Mike Beckerle 17 Nov '14

17 Nov '14

I don't recall whether we decided this matter or not. I would search for it myself, but I tried and failed to find anything. This is a hard topic to do searching on... no good keywords. If element 'e' has maxOccurs = 1, minOccurs = 1 (or neither are mentioned), is e[1] a valid expression? what about e[../some/pathExp/here]? If element 'e' has minOccurs= 0 maxOccurs = 1, is e[1] a valid expression, or do you access an optional element just by ../e ? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 4

Re: [DFDL-WG] referencing local element declarations from path expressions between two schema documents
by Mike Beckerle 17 Nov '14

17 Nov '14

Roger, I am struggling to defend the semantics you suggest below, because I cannot find anything in the XSD spec that specifically says how path steps are interpreted. In the absence of that, other members of the DFDL workgroup argue that path steps are just references to elements, and the default xmlns definition should work exactly as it works for any QName. So if the default namespace is defined, then a name with no prefix is by default interpreted relative to that definition. Hence, you have to do this: <xs:schema xmlns="BB"... > .... <xs:key name="myKey"> <xs:selector path="a:foo/bar" xmlns=""/> ... That is to say, if you wish to refer to locally declared elements with form unqualified, you cannot have a default namespace in effect, and the above provides the way you work around this, which is you explicitly reset the default namespace to "" meaning no-namespace. So is there anything you know of in the XSD spec that clarifies this, or is this only clarified based on precedent of how it works in Xerces XSD validator, but also Schematron, XSLT, etc. I have struggled to interpret the formal semantics document about XSD. My read of it did not find clarification of this point, but I confess to not understanding it sufficiently to be sure. Thanks in advance ...mike beckerle Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Wed, Sep 17, 2014 at 8:30 AM, Costello, Roger L. <costello(a)mitre.org> wrote: > Hi Mike, > > > > 1. In XSD only the global stuff can be referenced. So this is not allowed: > > > > <xs:element ref="bb:b1/c1"> > > > > 2. When using xs:key the XPath expression in @path must have a namespace > prefix even if the targetNamespace is the > > default namespace. Eg. *bk:Book. *If the schema had instead set > elementFormDefault="unqualified" then the XPath expressions would not be > namespace-qualified. > > > > 3. If you embed Schematron in an XSD file, during processing the > Schematron processor will extract out the Schematron and generate an XSLT > program. At that point you are in the world of XSLT and ordinary XPath > rules apply. So if BB.xsd has declared a global element {BB}b1 which > contains {}c1 then the XPath expression to reference c1 in an instance > document is this: bb:b1/c1 (where bb has been bound to BB in the > Schematron/XSLT code). It’s irrelevant to Schematron/XSLT whether the XML > Schemas have/don’t have a default namespace declaration. > > > > /Roger > > > > *From:* Mike Beckerle [mailto:mbeckerle.dfdl@gmail.com] > *Sent:* Wednesday, September 17, 2014 4:37 AM > *To:* dfdl-wg(a)ogf.org; Costello, Roger L. > *Subject:* referencing local element declarations from path expressions > between two schema documents > > > > > > I am looking for clarification about how path steps are to be interpreted > with respect to the issue of qualified/unqualified element names, and > default or no-default namespaces. > > If there is a precedent we can follow set by other tools which embed paths > into XSD (e.g., schematron?) or even just the selector/paths things in XSD > key/uniqueness constraints then I'd like to understand it. > > These are some of the conundrums: > > > Two Schema Files: A, and B > > Schema file A has target namespace AA, and binds the AA namespace to > prefix aa, and the BB namespace to prefix bb > > Schema file B has target namespace BB, and the same prefix bindings. > > Schema B has an element named b1 which has two local element declarations > inside it named c1 and c2. > > Schema A has an element named a1 which contains an element reference to > bb:b1. > > a1 has a DFDL annotation (e.g., dfdl:assert) which has a path expression > in it. this path contains bb:b1/c1 > > When is the path correct? > > 1a) when schema B has elementFormDefault='unqualified' (that's the > default), and schema A does NOT have a default namespace. Why? the default > namespace - if it existed for schema A - would implicitly qualify the c1 > step of the path, but since c1 is a local element in schema B with element > form unqualified, it does not require (and cannot have) a namespace > specified in path steps for c1. > > 1b) when schema B has elementFormDefault='qualified', and schema A has > xmlns='BB'. Why? Because in this case the path bb:b1/c1 is equivalent to > bb:b1/bb:c1, and since c1 will have a namespace-qualified name in schema B, > that path will properly describe the path to the c1 element. > > 1c) Variation: when schema B has no target namespace (which I believe > makes elementFormDefault irrelevant), and schema A has no default > namespace. Because then c1 can't be qualfied as it lives in no namespace, > and schema A has no implied qualifiers. > > > > When is the path incorrect? > > 2a) when schema A has xmlns="BB". Why? Because the default namespace means > the bb:b1/c1 is equivalent to {BB}b1/{BB}c1, but there is no {BB}/c1 since > c1 is just a local element declaration, and B has elementFormDefault > unqualified. > > 2b) when schema B has elementFormDefault='qualified', and schema A has no > default namespace. In that case the step c1 needs a namespace qualifier > either by default, or explicitly via a prefix, but it doesn't have one. > > So I am interested in opinions on whether the above analysis is correct or > not, and especially in precedent for how this stuff works in other tools. > > I have made a big assumption above which is that if xmlns="BB", then any > path step inside something carrying that default binding is interpreted as > qualified by that namespace. I picked this because it seemed reasonable, > and it seems one can always modify one's schema to properly address > something in another schema without having to change that other schema. > I.e., it seems to have good composition properties. > > Thanks for any thoughts. > > > > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > www.tresys.com > > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the OGF Intellectual Property Policy > <http://www.ogf.org/About/abt_policies.php> > > >

1 0

OGF DFDL WG Call Agenda 2014-11-18
by Steve Hanson 17 Nov '14

17 Nov '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13374?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Minutes 2014-11-11
by Steve Hanson 12 Nov '14

12 Nov '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13373?download= ** Updated DFDL 1.0 specification published at http://www.ogf.org/documents/GFD.207.pdf. ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] elementFormDefault and the form attribute on element declarations
by Steve Hanson 12 Nov '14

12 Nov '14

WG agreed that elementFormDefault and form attributes will be mentioned explicitly in section 5.1. http://redmine.ogf.org/issues/241 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 03/11/2014 14:57 Subject: Re: [DFDL-WG] elementFormDefault and the form attribute on element declarations Section 5.1 of the spec has 'Standard XSDL namespace management' in its list of what a DFDL schema consists of. Section 5.1 does not go on explicitly to exclude 'elementFormDefault' attribute or 'form' attribute, so my reading is that both these are therefore allowed. We could expand what is meant by 'Standard XSDL namespace management'. But if we do that we should make sure that we do so in any other areas that are a bit vague. For example, we disallow complex type derivation and substitution groups so does this imply that 'final' and 'block' on complex types and elements are prohibited regardless of setting? For example, we disallow identity constraints but does that mean that 'id' can never be used on schema objects? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 15/10/2014 23:25 Subject: [DFDL-WG] elementFormDefault and the form attribute on element declarations Sent by: dfdl-wg-bounces(a)ogf.org Neither the term "elementFormDefault" nor the term "unqualified" appear in the DFDL Spec. So we're silent about whether these are allowed or not in DFDL schemas. We should specify what of this XML Schema behavior is allowed, which I believe means add a line to each of the lists in section 5.1 of things used and not-used/reserved from XML Schema. FYI: Today the Daffodil implementation implements elementFormDefault on the xs:schema but not the form attribute on element declarations. We have tests that exercise elementFormDefault in qualified and unqualified and not specified at all (which defaults to "unqualified" behavior in XSD, so we're consistent with that.) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: minor/typographical - fn prefix namespace not in spec
by Steve Hanson 12 Nov '14

12 Nov '14

WG agreed that the namespace URI needs adding to section 23 and linking to the fn: prefix. http://redmine.ogf.org/issues/240 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 12/11/2014 12:04 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 28/10/2014 17:41 Subject: [DFDL-WG] minor/typographical - fn prefix namespace not in spec Sent by: dfdl-wg-bounces(a)ogf.org The URI corresponding to the xpath functions namespace http://www.w3.org/2005/xpath-functions is not in the DFDL spec. I am assuming where we are borrowing the exact XPath function behavior the namespace is the xpath functions URI, not some DFDL-specific URI. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] clarification: fn:exactly-one function
by Steve Hanson 12 Nov '14

12 Nov '14

WG agreed that fn:exactly-one() would have its description updated to match XPath 2.0 spec. Perhaps also worth updating fn:exists() and fn:empty() so their descriptions match XPath 2.0 spec. http://redmine.ogf.org/issues/239 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 03/11/2014 14:08 Subject: Re: [DFDL-WG] clarification: fn:exactly-one function The main purpose envisaged for fn:exactly-one() was as a guard around off-the-shelf XPath processors. In that scenario, throwing an error is probably required, as what would you do otherwise? If you wanted to return a true/false value could you just use fn:count($arg) eq 1 instead of fn:exactly-one() ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 30/10/2014 10:00 Subject: [DFDL-WG] clarification: fn:exactly-one function Sent by: dfdl-wg-bounces(a)ogf.org According to the DFDL spec, fn:exactly-one returns "True if the provided sequence contains exactly one node/value," which suggests that the function would return false if the sequence contains > or < 1 node/value. The XPath description states that fn:exactly-one "returns $arg if it contains exactly one item. Otherwise, raises an error." It seems of value to have a true/false result, because something that raises an error like this can only be used as an assertion effectively. On the other hand, that's not what XPath does. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end
by Steve Hanson 12 Nov '14

12 Nov '14

WG accepted the new wording. Tracked by http://redmine.ogf.org/issues/238. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 11/11/2014 16:53 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end One further modification as we are missing a case when there is nothing but zeroes in the content region... When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, or in all of the SimpleContent region being trimmed, then the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 18:10 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Hi Steve Yep - I agree with your new definition of the rule taking into account justification-independence and quoted text. I'm not sure I like the nested if-else in the new text though, as it took a couple of reads to understand. How about the text below? It splits out the second 'if/else', which I find easier to understand. When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Cheers, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Steve Hanson/UK/IBM To: Andrew Edwards/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 15:39 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Andy I agree that the existing words do not cover all the scenarios. Your proposed words are on the right track but only cover left trimming a right justified text number. We need something that is independent of justification and can handle patterns where there is quoted text as well as signs. One can envisage some bizarre scenarios. Eg, Text number pattern is "#0'000'" - an attempt to divide by 1000 using the pattern. DFDL parser would trim everything except 1 zero which would not match the pattern which expects at least 3 zeros. Trimming happens before pattern is looked at so I don't think we could cater for this (if we even wanted to). Perhaps we should say: When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then if the SimpleContent region is trimmed so that the removal of a '0' character leaves the next character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. That means that "000,000,123" would end up as "0,000,123" instead of ",000,123" today and "0000.025" would end up as "0.025" instead of ".025" today but I think that is good. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM@IBMGB To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 30/10/2014 16:43 Subject: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Sent by: dfdl-wg-bounces(a)ogf.org Hi all, I've hit an interesting case revolving around trimming and number patterns that doesn't seem quite sane to me. Consider an element with the following properties: textTrimKind='padChar' textNumberPadCharacter='0' textNumberPattern='0000+;0000-' So we have the sign character at the end of the representation. Now, imagine that the data being parsed is "0000+". The relevant rules from the DFDL specification are: Section 13.2 on textTrimKind When 'padChar', the element is trimmed of the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter, dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element. Section 13.6 on textNumberPadCharacter When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Section 13.6.1 Describes all of the pattern syntax. In our hypothetical case, the content region is not all zeros, as it ends in '+'. This means that the rule in section 13.6 does not apply and we only apply the rule in 13.2. This results in us trimming away all of the zeros and ending up with '+'. This then doesn't parse as a number. The problem seems to be that the rule in Section 13.6 doesn't take into account that the suffix of the pattern can result in text in the content region that isn't part of the digits of the number. Should the rule under section 13.56 be something more like this... When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, or the SimleContent region consists of a string of '0' characters followed by non-digit characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Thoughts? Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Fw: Action 258 - data where inactive escape character is to be retained
by Steve Hanson 12 Nov '14

12 Nov '14

Proposal below accepted. Issue raised: http://redmine.ogf.org/issues/237 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 31/10/2014 11:20 Subject: Fw: [DFDL-WG] Action 258 - data where inactive escape character is to be retained On last WG call there was still some concern about the names used. We will go with the proposal unless better names are suggested. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 31/10/2014 11:18 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 24/10/2014 13:22 Subject: Re: Fw: [DFDL-WG] Action 258 - data where inactive escape character is to be retained Proposal updated below to reflect recent WG calls - changes in red type. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 28/08/2014 15:47 Subject: Fw: [DFDL-WG] Action 258 - data where inactive escape character is to be retained Strawman proposal for action 258. 258 Consider allowing more flexible escapeCharacter schemes (Mike) 6/5: Motivated by example of an escape character which is active when in front of an in-scope delimiter, but not when in front of another character. 20/5: Can't model Mike's example with current facilities, but Mike's example is a generalisation of a particular MITRE example. Do we really need this? Jonathan to follow up. 3/6: Closed. Jonathan has provided the background to the MITRE example which was really about initiators and terminators. The generalised use case is perhaps speculative, so it was agreed not to change the DFDL spec to handle this unless a concrete use case emerges. 17/6: Re-opened. vCard 3.0 (http://tools.ietf.org/html/rfc2426) is an example of a format that exhibits the need for this. Need a proposal to handle this case, and which fits in with the existing extraEscapedCharacters and escapeEscapeCharacter property. Noted that using lengthKind 'pattern' is sometimes a way of working round this kind of thing. ... 15/7: No progress 22/7: Steve has started to write up a proposal. ... 26/8: No further progress 2/9: Strawman proposal sent by Steve for comment. Concern over name of new property. Review for next week. ... 16/9: No further progress 23/9: Discussed the naming issue, Steve to send revised proposal. Candidate for deferral to 1.1 ? 14/10: Agreed that it would be good to be able to handle vCard 3.0. With Steve to revise the proposal. New property dfdl:applyEscapeCharacter added. The description of dfdl:escapeKind is updated. No changes to dfdl:generateEscapeBlock but I've added it below by way of comparison. Property Name Description escapeKind Enum Valid values 'escapeCharacter', 'escapeBlock' The type of escape mechanism defined in the escape scheme When 'escapeCharacter': On unparsing a single character of the data is escaped by adding a dfdl:escapeCharacter or dfdl:escapeEscapeCharacter immediately before it. The characters to escape are determined by property dfdl:escapeCharacterPolicy. On parsing any in-scope terminating delimiter encountered in the data is not interpreted as such when it is immediately preceded by the dfdl:escapeCharacter (when not itself preceded by the dfdl:escapeEscapeCharacter). Occurrences of the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data as determined by property dfdl:escapeCharacterPolicy unless the dfdl:escapeCharacter is preceded by the dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter does not precede the dfdl:escapeCharacter, respectively. When 'escapeBlock': On unparsing the entire data are escaped by adding dfdl:escapeBlockStart to the beginning and dfdl:escapeBlockEnd to the end of the data. The data is either always escaped or escaped when needed as specified by dfdl:generateEscapeBlock. If the data is escaped and contains the dfdl:escapeBlockEnd then first character of each appearance of the dfdl:escapeBlockEnd is escaped by the dfdl:escapeEscapeCharacter. On parsing the dfdl:escapeBlockStart string must be the first characters in the (trimmed) data in order to activate the escape scheme. The dfdl:escapeBlockStart string is removed from the beginning of the data. Until a matching dfdl:escapeBlockEnd string (that is, one not preceded by the dfdl:escapeEscapeCharacter) is found in the data, any in-scope terminating delimiter encountered in the data is not interpreted as such, and any dfdl:escapeEscapeCharacters are removed when they precede an dfdl:escapeBlockEnd string. The matching dfdl:escapeBlockEnd string is removed from the data.. The matching dfdl:escapeBlockEnd does not have to be the last characters in the (trimmed) data in order to de-activate the escape scheme. A dfdl:escapeBlockStart occurring anywhere in the data other than the first characters has no significance. Annotation: dfdl:escapeScheme escapeCharacterPolicy Enum Valid values 'all', 'delimiters' Controls when escape characters are removed during parsing, and output during unparsing, when dfdl:escapeKind is 'escapeCharacter'. When 'all': During unparsing the following are escaped as described in dfdl:escapeKind when they are in the data. · Any in-scope terminating delimiter by escaping its first character. · dfdl:escapeCharacter (escaped by dfdl:escapeEscapeCharacter) · any dfdl:extraEscapedCharacters During parsing, occurrences of dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are interpreted and removed from the data as described in dfdl:escapeKind. When 'delimiters': During unparsing the following are escaped as described in dfdl:escapeKind when they are in the data. · Any in-scope terminating delimiter by escaping its first character. · dfdl:escapeCharacter (escaped by dfdl:escapeEscapeCharacter) During parsing, occurrences of dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are interpreted and removed from the data as described in dfdl:escapeKind, except that dfdl:escapeCharacter is only removed when it immediately precedes an in-scope terminating delimiter. Annotation: dfdl:escapeScheme generateEscapeBlock Enum Valid values 'always', 'whenNeeded' Controls when escaping is used on unparsing when dfdl:escapeKind is 'escapeBlock'. If 'always' then escaping always occurs as described in dfdl:escapeKind. If 'whenNeeded' then escaping occurs as described in dfdl:escapeKind when the data contains any of the following: · any in-scope terminating delimiter · dfdl:escapeBlockStart at the start of the data · any dfdl:extraEscapedCharacters Annotation: dfdl:escapeScheme Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2014 15:10 ----- From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/06/2014 17:28 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained I think I have come across a concrete example of this. It's from a standard called vCard 3.0 (RFC 2426 - http://tools.ietf.org/html/rfc2426) A backslash is used to a) escape itself; b) escape in-scope delimiters; c) indicate an embedded linefeed. The backslash would need removing for a) and b) but not for c). ESCAPED-CHAR = "\\" / "\;" / "\," / "\n" / "\N") ; \\ encodes \, \n or \N encodes newline ; \; encodes ;, \, encodes , Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: "Cranford, Jonathan W." <jcranford(a)mitre.org>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 04/06/2014 08:53 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained WG call 3rd June: The generalised use case is perhaps speculative, so it was agreed not to change the DFDL spec to handle this unless a concrete use case emerges. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 20/05/2014 18:11 Subject: Re: [DFDL-WG] data where inactive escape character is to be retained Sent by: dfdl-wg-bounces(a)ogf.org All, I’ll chime in with an observation and few more details on how Roger Costello got around a similar problem. Observation An escape block allows the escapeBlockEnd to be escaped with the escapeEscapeCharacter, while allowing the escapeEscapeCharacter itself to appear in the data without any special semantics as long as it is NOT followed by escapeBlockEnd. (From section 13.2.1 of the spec: “On parsing the dfdl:escapeBlockStart is removed from the beginning of the data and dfdl:escapeBlockEnd is removed from end of the data and any dfdl:escapeEscapeCharacters are removed when they precede a dfdl:escapeBlockEnd.”) This is really similar to the problem that Mike posed to the group below, where an escape character is sometimes an escape character but sometimes isn’t. While an escape block might not be suitable in all circumstances, the original problem that sparked Mike’s post was amenable to using an escape block, and that is how Roger Costello got around the problem. Some more details Roger Costello was using a quotation mark (“) as the initiator and terminator for quoted values in a data format. In this format, quotation marks can be escaped with a backslash (\); however, within a quoted string, the data could have a backslash as a normal data character (e.g. \n, representing two characters, not a single newline character). Roger posed his challenge to the Daffodil team, and then Mike created the example below to demonstrate the problem to the DFDL WG. In contrast to Mike’s example, Roger was having the problem with initiators and terminators, not a separator. At the time, we thought that that an escape block couldn’t be applied to the data format in question, so Mike may have altered the problem in order to prevent an escape block from clouding the issue as posed to the WG. It turns out, after more analysis, that an escape block could be used, and that solved the problem: escapeBlockStart=”"” escapeBlockEnd=”"” escapeEscapeCharacter=”\” Closing Observations In general, DFDL supports two different escape schemes with different behavior for the escape character. * When escapeKind=”escapeCharacter”, the escape character is always an escape character. * When escapeKind=”escapeBlock”, the escape character (escapeEscapeCharacter) is only an escape character in front of escapeBlockEnd. In this case, we were able to use an escape block to model the data format. While there may be a data format that has a character that is sometimes an escape character and sometimes isn’t, without a real world example, I echo Mike’s hesitance to add this feature to DFDL. HTH, Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Mike Beckerle Sent: Friday, May 02, 2014 2:47 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] data where inactive escape character is to be retained We have data that has ; as separator and has fields that look like this: abcd \; efgh That's a single field. The backslash escapes the ; so that the data is abcd ; efgh. This same data set also has abcd \n efgh Here the backslash precedes an ordinary non-delimiter. The data is supposed to be abcd \n efgh. That is, this data set requires the backslash to be retained in the data when it is not preceding the start of a delimiter. Am I missing something or is it impossible to model this? It would seem there needs to be a flag to indicate whether the escape characters that don't actually escape a delimiter are to be retained or not. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end
by Andrew Edwards 11 Nov '14

11 Nov '14

One further modification as we are missing a case when there is nothing but zeroes in the content region... When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, or in all of the SimpleContent region being trimmed, then the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 18:10 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Hi Steve Yep - I agree with your new definition of the rule taking into account justification-independence and quoted text. I'm not sure I like the nested if-else in the new text though, as it took a couple of reads to understand. How about the text below? It splits out the second 'if/else', which I find easier to understand. When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Cheers, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Steve Hanson/UK/IBM To: Andrew Edwards/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 15:39 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Andy I agree that the existing words do not cover all the scenarios. Your proposed words are on the right track but only cover left trimming a right justified text number. We need something that is independent of justification and can handle patterns where there is quoted text as well as signs. One can envisage some bizarre scenarios. Eg, Text number pattern is "#0'000'" - an attempt to divide by 1000 using the pattern. DFDL parser would trim everything except 1 zero which would not match the pattern which expects at least 3 zeros. Trimming happens before pattern is looked at so I don't think we could cater for this (if we even wanted to). Perhaps we should say: When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then if the SimpleContent region is trimmed so that the removal of a '0' character leaves the next character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. That means that "000,000,123" would end up as "0,000,123" instead of ",000,123" today and "0000.025" would end up as "0.025" instead of ".025" today but I think that is good. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM@IBMGB To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 30/10/2014 16:43 Subject: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Sent by: dfdl-wg-bounces(a)ogf.org Hi all, I've hit an interesting case revolving around trimming and number patterns that doesn't seem quite sane to me. Consider an element with the following properties: textTrimKind='padChar' textNumberPadCharacter='0' textNumberPattern='0000+;0000-' So we have the sign character at the end of the representation. Now, imagine that the data being parsed is "0000+". The relevant rules from the DFDL specification are: Section 13.2 on textTrimKind When 'padChar', the element is trimmed of the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter, dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element. Section 13.6 on textNumberPadCharacter When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Section 13.6.1 Describes all of the pattern syntax. In our hypothetical case, the content region is not all zeros, as it ends in '+'. This means that the rule in section 13.6 does not apply and we only apply the rule in 13.2. This results in us trimming away all of the zeros and ending up with '+'. This then doesn't parse as a number. The problem seems to be that the rule in Section 13.6 doesn't take into account that the suffix of the pattern can result in text in the content region that isn't part of the digits of the number. Should the rule under section 13.56 be something more like this... When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, or the SimleContent region consists of a string of '0' characters followed by non-digit characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Thoughts? Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2014-11-11
by Steve Hanson 10 Nov '14

10 Nov '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13373?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call 2014-11-04 - cancelled
by Steve Hanson 04 Nov '14

04 Nov '14

Today's call is cancelled as not enough people available. Next meeting is 11th. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: DFDL Working Group Call
by Steve Hanson 04 Nov '14

04 Nov '14

Cancelling call as not enough people available

1 0

Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end
by Andrew Edwards 03 Nov '14

03 Nov '14

Hi Steve Yep - I agree with your new definition of the rule taking into account justification-independence and quoted text. I'm not sure I like the nested if-else in the new text though, as it took a couple of reads to understand. How about the text below? It splits out the second 'if/else', which I find easier to understand. When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then the SimpleContent region is trimmed of the '0' characters as defined by the trimming rules. If this trimming results in the next character in the SimpleContent region being a character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Cheers, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times From: Steve Hanson/UK/IBM To: Andrew Edwards/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/11/2014 15:39 Subject: Re: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Andy I agree that the existing words do not cover all the scenarios. Your proposed words are on the right track but only cover left trimming a right justified text number. We need something that is independent of justification and can handle patterns where there is quoted text as well as signs. One can envisage some bizarre scenarios. Eg, Text number pattern is "#0'000'" - an attempt to divide by 1000 using the pattern. DFDL parser would trim everything except 1 zero which would not match the pattern which expects at least 3 zeros. Trimming happens before pattern is looked at so I don't think we could cater for this (if we even wanted to). Perhaps we should say: When parsing, if the pad character is '0' and dfdl:textTrimKind is 'padChar' then if the SimpleContent region is trimmed so that the removal of a '0' character leaves the next character other than a digit, the last '0' character is re-instated and not trimmed. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. That means that "000,000,123" would end up as "0,000,123" instead of ",000,123" today and "0000.025" would end up as "0.025" instead of ".025" today but I think that is good. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM@IBMGB To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 30/10/2014 16:43 Subject: [DFDL-WG] Trimming of a text number that's all zeros when the number pattern has a sign char at the end Sent by: dfdl-wg-bounces(a)ogf.org Hi all, I've hit an interesting case revolving around trimming and number patterns that doesn't seem quite sane to me. Consider an element with the following properties: textTrimKind='padChar' textNumberPadCharacter='0' textNumberPattern='0000+;0000-' So we have the sign character at the end of the representation. Now, imagine that the data being parsed is "0000+". The relevant rules from the DFDL specification are: Section 13.2 on textTrimKind When 'padChar', the element is trimmed of the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter, dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element. Section 13.6 on textNumberPadCharacter When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Section 13.6.1 Describes all of the pattern syntax. In our hypothetical case, the content region is not all zeros, as it ends in '+'. This means that the rule in section 13.6 does not apply and we only apply the rule in 13.2. This results in us trimming away all of the zeros and ending up with '+'. This then doesn't parse as a number. The problem seems to be that the rule in Section 13.6 doesn't take into account that the suffix of the pattern can result in text in the content region that isn't part of the digits of the number. Should the rule under section 13.56 be something more like this... When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, or the SimleContent region consists of a string of '0' characters followed by non-digit characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Thoughts? Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Trimming of a text number that's all zeros when the number pattern has a sign char at the end
by Andrew Edwards 03 Nov '14

03 Nov '14

Hi all, I've hit an interesting case revolving around trimming and number patterns that doesn't seem quite sane to me. Consider an element with the following properties: textTrimKind='padChar' textNumberPadCharacter='0' textNumberPattern='0000+;0000-' So we have the sign character at the end of the representation. Now, imagine that the data being parsed is "0000+". The relevant rules from the DFDL specification are: Section 13.2 on textTrimKind When 'padChar', the element is trimmed of the dfdl:textStringPadCharacter, dfdl:textNumberPadCharacter, dfdl:textBooleanPadCharacter or dfdl:textCalendarPadCharacter depending on the type of the element. Section 13.6 on textNumberPadCharacter When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Section 13.6.1 Describes all of the pattern syntax. In our hypothetical case, the content region is not all zeros, as it ends in '+'. This means that the rule in section 13.6 does not apply and we only apply the rule in 13.2. This results in us trimming away all of the zeros and ending up with '+'. This then doesn't parse as a number. The problem seems to be that the rule in Section 13.6 doesn't take into account that the suffix of the pattern can result in text in the content region that isn't part of the digits of the number. Should the rule under section 13.56 be something more like this... When parsing, if the pad character is '0' and the SimpleContent region consists entirely of '0' characters, or the SimleContent region consists of a string of '0' characters followed by non-digit characters, then the last remaining '0' is not trimmed and a single '0' is the result of the trimming. This rule also applies when the pad character is a DFDL character entity equivalent to '0'. This rule does not apply when the pad character is any other character nor when a pad byte is specified. Thoughts? Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Agenda 2014-11-04
by Steve Hanson 03 Nov '14

03 Nov '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13370?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Minutes 2014-10-28
by Steve Hanson 03 Nov '14

03 Nov '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13369?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

elementFormDefault and the form attribute on element declarations
by Mike Beckerle 03 Nov '14

03 Nov '14

Neither the term "elementFormDefault" nor the term "unqualified" appear in the DFDL Spec. So we're silent about whether these are allowed or not in DFDL schemas. We should specify what of this XML Schema behavior is allowed, which I believe means add a line to each of the lists in section 5.1 of things used and not-used/reserved from XML Schema. FYI: Today the Daffodil implementation implements elementFormDefault on the xs:schema but not the form attribute on element declarations. We have tests that exercise elementFormDefault in qualified and unqualified and not specified at all (which defaults to "unqualified" behavior in XSD, so we're consistent with that.) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Action 274 - on qnames in path expressions in DFDL
by Mike Beckerle 03 Nov '14

03 Nov '14

I did some experiments with the XML validation built into eclipse, and the key constraint, which has paths (in the selector and field attributes) that refer to other parts of the XML infoset/document and so involve the names of elements. I also have been reading the XPath 2.0 specs including the formal semantics. To my interpretation XPath 2.0 is pretty explicit: An element or type QName consisting only of a local part NCName expands to the default element/type namespace and the local part. They don't qualify this in any way that I can find for the names used in path steps. The formal semantics is pretty hard to understand, but this detail seems to be swept under the rug. There would need to be specific treatment of this in the naming environment, but I cannot find this. There is discussion of name resolution in some of their judgements, but nothing I can see that creates an exception for names in path steps. (Called name tests) What I was looking for was language that specifies something about a non-prefixed name in a path step and that it is NOT augmented by the default namespace. I could find no such exception. But if I am right and this detail isn't there, i.e., If there is no such exception, then the implementation of XPath in eclipse (which is xerces I believe) is incorrect in the way it resolves names that appear in path steps. Here's how it works according to eclipse: In xpath expressions in the selectors and fields of unique and key constraints, xmlns (the default namespace) is not considered as implicitly augmenting unqualified names that appear in path steps. An identifier in a path step that has no prefix (e.g., 'foo' in the path './p1:bar/foo/p2:baz') is always interpreted as identifying either 1) a child element defined by a local element declaration with form "unqualified". or 2) this corner case: a child element defined by an element reference to a global element declaration but only when that global element declaration has no effective target namespace. (Use of 'effective' here rules out an element decl inside a schema document that itself has no target namespace but is included into a schema that has one. Such an element declaration effectively has a target namespace, so this corner case doesn't apply.) Case (2) here is a rather obscure corner case, but there nevertheless. In no case (that I could construct) does the name in a path step get augmented by a schema's default namespace (defined by xmlns="...whatever..."). The above is true for names without prefix appearing in path steps of expressions. The QNames that are used in an XML schema to reference a global declaration or definition (in the ref of an element ref, type attribute in a type reference, base in a simple type derivation, etc.) are different. Those names DO pickup a default namespace if one is defined and they have no explicit prefix themselves. This creates a corner case for these references (to global decls/defs) which is if a definition/declaration appears in no effective target namespace, then there is no way to reference it at all from a schema that has a default namespace definition surrounding the point of reference. There is no syntactic way to turn-off use of a default namespace. (The semantics of xpath allows for a "none" default namespace definition, but there is no xml syntax for expressing explicitly this "none". It is simply what you have if there is no prefix and no default namespace). You can change which namespace the default is, but there's no way to say explicitly "I meant this QName reference to be to something not in any namespace. Unless/until we can find language in some XML Schema spec indicating how path step names (name tests) are specifically resolved, then the above is just "how one system does it." But we may end up having to follow precedent here from implementations. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 4

clarification: fn:exactly-one function
by Mike Beckerle 03 Nov '14

03 Nov '14

According to the DFDL spec, fn:exactly-one returns "True if the provided sequence contains exactly one node/value," which suggests that the function would return false if the sequence contains > or < 1 node/value. The XPath description states that fn:exactly-one "returns $arg if it contains exactly one item. Otherwise, raises an error." It seems of value to have a true/false result, because something that raises an error like this can only be used as an assertion effectively. On the other hand, that's not what XPath does. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 2