- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2014-04-29
by Steve Hanson 01 May '14

01 May '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13244?download=. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario)
by Steve Hanson 30 Apr '14

30 Apr '14

248 Discriminators and potential points of uncertainty (Steve) 28/1: Steve to write up a proposal to prevent a discriminator from behaving in a non-obvious manner when used with a potential point of uncertainty that turns out not to be an actual point of uncertainty. 5/2: Steve sent an email to check whether choice branches, unordered elements and floating elements should always be actual points of uncertainty, as there are times when there is no uncertainty, eg, last choice branch; all floating elements found. It was decided that they are always actual points of uncertainty. To do otherwise will complicate implementations and result in fragile schemas. Steve will proceed with the proposal on that basis. Based on the above, which reflects the email discussion below, here is what I propose to resolve this action. If a potential point of uncertainty is always an actual point of uncertainty (choice branch, element in unordered sequence, floating element, ock 'parsed') then a discriminator that applies to it will only ever resolve that point of uncertainty. It never has an effect on any enclosing point of uncertainty. If a potential point of uncertainty is sometimes an actual point of uncertainty (ock 'implicit') then a discriminator that applies it will only ever resolve, or have no effect on, that point of uncertainty. It never has an effect on any enclosing point of uncertainty. If a potential point of uncertainty is never an actual point of uncertainty (ock 'fixed', 'expression', 'stopValue') then a discriminator that applies to it will never have an effect on that point of uncertainty. Nor does it ever have an effect on any enclosing point of uncertainty. I think 1 and 2 are not controversial, but there is an alternative for 3: 3. If a potential point of uncertainty is never an actual point of uncertainty (ock 'fixed', 'expression', 'stopValue') then a discriminator that applies to it will never have an effect on that point of uncertainty. Instead the discriminator is applied to any enclosing point of uncertainty. The alternative means that changing an element from (say) ock 'parsed' to ock 'expression' has the same effect on a discriminator as changing the element to (1,1). The discriminator that applied to it now applies to any enclosing pou. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 05/02/2014 12:04 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Thanks Tim, all good points. Comments to your comments. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 05/02/2014 11:01 Subject: Re: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) A couple of comments below. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 05/02/2014 10:50 Subject: [DFDL-WG] Action 248 (was Thoughts on a discriminator scenario) Sent by: dfdl-wg-bounces(a)ogf.org 248 Discriminators and potential points of uncertainty (Steve) 28/1: Steve to write up a proposal to prevent a discriminator from behaving in a non-obvious manner when used with a potential point of uncertainty that turns out not to be an actual point of uncertainty. 5/2: With Steve I started on this by reading section 9.3.3 on points of uncertainty, which lists the potential PoUs. Here's the list to save getting the spec out. 1. An xs:choice branch 2. All xs:elements in an unordered xs:sequence (dfdl:sequenceKind is 'unordered') 3. An optional xs:element 4. An array xs:element 5. All xs:elements in an xs:sequence containing one or more floating xs:elements. The section then looks at each in turn and gives the circumstances when it is an actual PoU or not. As currently written, it is only 3 and 4 where a potential PoU might not be an actual PoU. For 1, 2 and 5 it says they are always actual PoUs. But I'm not sure that's correct. A deeper analysis of what is actually going on with 1, 2 and 5 says to me that there are times when there might not be an actual PoU. 1. Given that there is no concept in DFDL of optional choice branches, then if the last branch is reached then there is no longer a PoU. It must be that branch else it is a processing error. TK: I think of it slightly differently. It is a PoU, even if the branch is the only remaining branch. If we say that the final choice branch is not a PoU then diagnostics become confused - the parser reports the error code as 'error while parsing root/choice/lastBranch/field1' when the correct error code would be 'none of the branches of root/choice were found in the data'. SMH: I see your point. My thinking was that choices have finite branches and a choice is (1,1). If I have got to the last branch then I am not one of the other branches so I must be this one. If there is any other possibility then the model is missing a branch, even if it is just one that contains an empty sequence with an assert {fn:false()}. In practice of course users forget to add that last branch (there's no XSDL equivalent to the 'default' branch of a switch/case statement), so yes they could end up with an unclear diagnostic. 2. There can come a point in an unordered sequence when all that can be encountered is one element, and if that is (1,1) then there is no longer a PoU. TK: It's still a PoU. The specification says that occursCountKind is 'parsed' for all members of an unordered group, so min/maxOccurs do not come into play. SMH: Interesting. The spec says that if a member is optional or an array then it must be 'parsed'. If it is (1,1) though it does not have an occursCountKind. The specific case I was thinking of is when all members are (1,1), so when you have one element to go there is no PoU. However, the rewrite into a repeating choice has the effect of making everything 'parsed', which is really the point you are making. So I agree with you, it is easier to say that everything is an actual PoU else it complicates the rewrite semantic. 5. If all floating elements are (1,1) and all are encountered, then from that point on there are no longer any PoUs due to floating elements. TK: I suspect that floating elements are somewhat like unordered branches - most users will not want min/maxOccurs to affect the parsing of the group. Schema validation ( or more complex validation applied in the receiving application ) will deal with non-conformances. SMH: Possibly yes. With something like X12 NTE segments, that is the case. But we don't express the floating semantic as a rewrite of the whole sequence like we do for unordered, it's more of a per element thing. And if that is done dynamically as we go through the sequence, having no PoU can result. I'd like us to get straight on this before I proceed with the action proper. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/02/2014 10:12 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 27/01/2014 17:39 Subject: Fw: Thoughts on a discriminator scenario Been thinking some more on the discriminator scenario below that I mailed out before xmas, and discussing it with the IBM DFDL team. The 'confusing' aspect of the behaviour is that a discriminator within a potential PoU will act on a higher level PoU if the potential PoU is not an actual PoU. In the example, the array element 'Type1' is not an actual PoU for occurrence 1, only for occurrences 2+. So when the discriminator fires for occurrence 1 it will resolve a higher level unresolved PoU if one exists. Perhaps the spec should say that a discriminator can't 'leak' beyond the potential PoU that encloses it ? If so, then for occurrence 1 the discriminator has no effect, and only has an effect for occurrences 2+. This makes for more predictable and robust schemas. We'd need to go through spec section 9.3.3 carefully to see if this does not break any of the potential PoUs that are listed. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/01/2014 09:55 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 20/12/2013 13:20 Subject: Thoughts on a discriminator scenario Take the following schema (simplified) for element Type1 (1,10) being a loop for elements A,B,C. Type 1 does not have an initiator so I need to use a discriminator to establish the existence of an occurrence of Type1 so that incorrect backtracking does not occur after an error. Because occursCountKind is 'implicit', the 1st occurrence is not a point of uncertainty so the discriminator acts instead on any enclosing point of uncertainty, but for 2nd and subsequent occurrences it acts on Type1. That is all working as designed, but I think users find will the 1st occurrence behaviour a bit confusing. There are workarounds to avoid the problem, eg, use occursCountKind 'parsed' or split Type1 into two as (1,1) and (0,9). I think this is worth documenting in a tutorial as this is quite subtle stuff. <xs:element name="Type1" maxOccurs="10" dfdl:occursCountKind="implicit"> <dfdl:discriminator test="{fn:exists(A)}" /> <xs:complexType> <xs:sequence> <xs:element name="A" dfdl:initiator="A:" ... /> <xs:element name="B" dfdl:initiator="B:" ... /> <xs:element name="C" dfdl:initiator="C:"... /> </xs:sequence> </xs:complexType> Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

validating expressions on elements in a choice or unordered sequence
by Mark Frost 30 Apr '14

30 Apr '14

When we were implementing unordered sequences, this raised some questions around evaluating relative paths in expressions, for elements in a choice or unordered sequence : DFDL spec: (gwdrp-dfdl-v1.0.4 section 15) "When processing a choice group the parser validates any contained path expressions. If a path expression contained inside a choice branch refers to any other branch of the choice, then it is a schema definition error." 1. I'm not clear what benefit this restriction on path expressions gives. It seems redundant since in any single instance of a choice group, if the branch being processed exists, then by definition none of it's sibling branches exist. Any expression path referring to a non-existent branch would correctly return <empty sequence> If the choice group is inside a repeating structure, then expressions referring to choice branches within other instances of the choice could be useful. Should an expression referring to branches in other instances of a choice cause a schemadef error? Example expression on el_b could be { fn:count(../../el_choice/el_a) } - parent [sequence] - el_choice [minOccurs=5 maxOccurs=5] [choice] - el_a - el_b 2. Should an expression that potentially refers to branches in the choice cause a schemadef error? Example identically named elements in and out of a choice expression on el_c could be { fn:count(../el_a) } - parent [sequence] - el_a - el_b - [embedded choice group] - el_a - el_c Regards, Mark Frost _____________________________________ MP 211, IBM Hursley, Winchester, SO21 2JN Phone: (01962) 817009 or x247009 Email: frostmar(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

6 6

Confirmed: DFDL Working Group Call (29 Apr 16:00 GDT in DE3S09)
by Steve Hanson 29 Apr '14

29 Apr '14

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Call is on today

1 0

OGF DFDL WG Call Agenda 2014-04-29
by Steve Hanson 29 Apr '14

29 Apr '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13244?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] validating expressions on elements in a choice or unordered sequence
by Steve Hanson 29 Apr '14

29 Apr '14

Tim, thanks for clarifying where you are coming from on this. I interpreted "some component of the static context" differently to you. The components are the 'variables' in the static context, not schema objects. The XPST0001 error says "It is a static error if analysis of an expression relies on some component of the static context that has not been assigned a value. " , so I think this message would be issued if, for example, an XPath processor tried to look up an element in the In-scope schema definitions component but found that the component itself had not been built. I think the error that applies to the case we are discussing is XPST0008 as quoted below. Otherwise I don't see when XPST0008 would be issued. Consequently I am not sure that XPath rules prohibit the genuinely-impossible situation where an expression on choiceElement[1]/A refers to choiceElement[1]/B, because I can't see a crisp definition of in-scope. Maybe I'm missing it? Common sense says that it should be prohibited. But I agree that XPath does not prohibit choiceElement/A referring to choiceElement/B generally. My conclusions about the rules in 23.1 was perhaps too general, and they should be looked at on an individual basis. But the reason they exist is because the parser is not acting on the whole message and in general is prohibited from looking ahead. Clearly the first two rules are needed (assert/discriminator can look down but not ahead, and outputValueCalc can look ahead). The other three are the ones in question. The current wording came from this action which was spun off from the unordered action 199: 214 Expression Language Data Model (All) 16/7: Augment section 23.1 so that it covers the cases where an XPath references an element that is in a different choice branch or that is in the same unordered sequence or that is floating. These cases could be detected statically (though to do this 100% reliably is not easy) or they could be left until runtime and fail if the element does not exist. Both are schema definition errors as explained by errata 2.120. 23/7: Closed. Runtime schema definition error if the element does not exist. Errata taken. (Erratum 2.120 is just the one that categorised all the different errors.) For the choice branch rule: If XPath disallows choice/x[1] referring to choice/y[1] but allows general refs, then I am ok with changing this to match XPath (ie, we drop our rule as it is implied). If it doesn't disallow choice/x[1] referring to choice/y[1] then perhaps we should keep our rule but tighten it up. I think the unordered rule and floating rule both stem from the choice branch rule - because of the rewrite semantic that turns unordered into a choice. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: Mark Frost/UK/IBM@IBMGB Date: 28/04/2014 22:03 Subject: Re: [DFDL-WG] validating expressions on elements in a choice or unordered sequence I don't think I am missing the point. DFDL's usage of a partial XDM is not in play here. The static context is defined thus: [Definition: The static context of an expression is the information that is available during static analysis of the expression, prior to its evaluation.] This information can be used to decide whether the expression contains a static error. If analysis of an expression relies on some component of the static context that has not been assigned a value, a static error is raised [err:XPST0001]. This is the definition that is relevant, because the errors are issued by our static analysis ( performed by the DFDL validator ). I believe the rule already prohibits the genuinely-impossible situation where an expression on choiceElement[1]/A refers to choiceElement[1]/B. But it does not prohibit choiceElement/A referring to choiceElement/B - and I don't think DFDL should either. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 28/04/2014 13:51 Subject: Re: [DFDL-WG] validating expressions on elements in a choice or unordered sequence It is certainly easier if we can just do the same as XPath 2.0 stipulates. But I think that this misses the point here. The XPath error for statically detecting that an expression refers to something that can never exist is XPST0008, which says: It is a static error if an expression refers to an element name, attribute name, schema type name, namespace prefix, or variable name that is not defined in the static context, except for an ElementName in an ElementTest or an AttributeName in an AttributeTest. The static context has the notion of "In-scope schema definitions" being " a generic term for all the element declarations, attribute declarations, and schema type definitions that are in scope during processing of an expression.". It doesn't define exactly what is meant by "in-scope" but XPath assumes that it acts on a complete instance of an XDM. In DFDL we are different to typical XPath usage as we are applying expressions during parsing when the document is incomplete. We can use that as the justification for applying extra constraints, which is exactly why there are additional rules in section 23.1. So, if there are scenarios where a rule is going to be restrictive then we should consider dropping it. If there are not, but it makes the life of an implementer harder because it is hard to code the rule, then we should consider dropping it. Otherwise keep it. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/04/2014 14:03 Subject: Re: [DFDL-WG] validating expressions on elements in a choice or unordered sequence Sent by: dfdl-wg-bounces(a)ogf.org I would be quite uncomfortable with DFDL not being a 'proper subset' of XPath 2.0. I understand the motivation ( having personally been involved in coding a query engine for DFDL ) but I think the cure would be worse than the complaint. Consistent with that, I think I agree with Mark's suggestion - a DFDL processor should just 'do what an XPath processor would do'. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Mark Frost/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 11/04/2014 13:23 Subject: Re: [DFDL-WG] validating expressions on elements in a choice or unordered sequence Sent by: dfdl-wg-bounces(a)ogf.org Comments inline On Fri, Apr 11, 2014 at 6:22 AM, Mark Frost <FROSTMAR(a)uk.ibm.com> wrote: When we were implementing unordered sequences, this raised some questions around evaluating relative paths in expressions, for elements in a choice or unordered sequence : DFDL spec: (gwdrp-dfdl-v1.0.4 section 15) "When processing a choice group the parser validates any contained path expressions. If a path expression contained inside a choice branch refers to any other branch of the choice, then it is a schema definition error." 1. I'm not clear what benefit this restriction on path expressions gives. It seems redundant since in any single instance of a choice group, if the branch being processed exists, then by definition none of it's sibling branches exist. Any expression path referring to a non-existent branch would correctly return <empty sequence> Typically in XPath, such paths would just be empty-sequence at runtime. Making it an SDE hoists the error to (hopefully) compile time, and making it SDE (non-recoverable) changes the way one must write expressions. You can't write utter nonsense paths and have them be runnable. If the choice group is inside a repeating structure, then expressions referring to choice branches within other instances of the choice could be useful. Should an expression referring to branches in other instances of a choice cause a schemadef error? Should be no issue if you are looking at say, position() - n. If you reach to something that doesn't exist, then you'll get empty sequence. My experience so far with XPath is that this notion that non-existance returns empty sequence is painful at best and a nightmare at worst. Expressions that are utterly nonsense are accepted executed, and silently fail by returning empty sequence. The most common mistake is writing /a/b/c when you needed /ns1:a/ns2:b/ns3:c. Example expression on el_b could be { fn:count(../../el_choice/el_a) } - parent [sequence] - el_choice [minOccurs=5 maxOccurs=5] [choice] - el_a - el_b 2. Should an expression that potentially refers to branches in the choice cause a schemadef error? Example identically named elements in and out of a choice expression on el_c could be { fn:count(../el_a) } - parent [sequence] - el_a - el_b - [embedded choice group] - el_a - el_c I'd love to restrict this, because we're looking at having to create a DFDL expression language implementation for performance reasons, and complex things like this require a very complex implementation tantamount to a query-engine. I would claim that these two el_a elements are different, and we could choose to restrict a DFDL path expression to return only nodes described by the same schema component, with "same schema component" meaning same path from document element to the schema component where an element or group or type reference counts as part of that path. So two different element references to the same global element would be two different schema components. But I suspect that this is too restrictive, and implementations are just going to have to be sophisticated enough to execute queries like this one, and a good implementation will optimize simpler cases for faster execution. ...mikeb-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 253 - delimiters - misleading phrasing - was: Re: Fw: DL11A #3: (Question)[1](Open)(English)need clarification
by Mike Beckerle 28 Apr '14

28 Apr '14

There are two issues raised in the email thread started by Oya Argun. Let's look at the second one first since it is quickly addressed. This is just a typographical error "proceed" should be precede. The offending sentence was: On parsing the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data, unless the dfdl:escapeCharacter is preceded or the dfdl:escapeEscapeCharacter does not *proceed* the dfdl:escapeCharacter. I have already made the above correction to the draft-gwdrp-dfdl-v1.0.4-r07.docx draft of the spec. The other issue requires us to reword the text, as the material identified as hard to translate is actually misleading and should be cleaned up. Suggested Erratum: In the description of the dfdl:initiator property, the sentence "The initiator with the longest match is the one that is selected as having been 'found', with length-ties being resolved so that the matching initiator is selected that is first in the order written in the schema." should be dropped. In the description of the dfdl:terminator property, the sentence "The terminator with the longest match is the one that is selected as having been 'found', with length-ties being resolved so that the matching terminator is selected that is first in the order written in the schema." should be dropped. Add this sentence: "This property can be used to determine the length of an element as described in Section 12.3.2 dfdl:lengthKind 'delimited'." In the description of the dfdl:separator property, the sentence "The separator with the longest match is the one that is selected as having been 'found', with length-ties being resolved so that the matching separator is selected that is first in the order written in the schema." should be dropped. Add this sentence: "This property can be used to determine the length of an element as described in Section 12.3.2 dfdl:lengthKind 'delimited'." It is a separate issue whether we consider the description of how lengthKind delimited works in 12.3.2 to be adequate or not, and that is out of scope for this particular action item/erratum. The above suggested erratum text gets rid of the misleading language from the descriptions of the properties. Let's discuss whether improvement to 12.3.2 is required on the next DFDL WG call. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php> On Fri, Apr 11, 2014 at 10:36 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > > Regards > > Steve Hanson > Architect, *IBM DFDL*<http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 11/04/2014 15:36 ----- > > From: Steve Hanson/UK/IBM > To: Kevin Safford/UK/IBM@IBMGB, > Cc: Alex Wood1/UK/IBM@IBMGB, "Mike Beckerle" <mbeckerle(a)tresys.com> > Date: 06/03/2014 16:32 > Subject: Re: Fw: DL11A #3: (Question)[1](Open)(English)need > clarification > ------------------------------ > > > Kevin > > 1) When the dfdl:initiator property is a list of values, and the data > matches more than one value, then the longest value is the one selected. > Example: data is aaa12345, dfdl:initiator is "aa aaa", then aaa is the > match as it is longest. However if the data matches more than one value and > the values have the same length, then the one that is declared earlier in > the list is selected. > > I have to say that I am struggling to come up with an example where it > actually makes a material difference, because if it matches multiple of the > same length, then surely they must be equivalent anyway? Maybe white space > has an effect here. Mike Beckerle crafted these words, perhaps he can give > an example. (Note that same words occur dfdl:terminator and dfdl:separator). > > 2) Agree, and have added to the Minor Editorial public comment on Redmine. > > Regards > > Steve Hanson > Architect, *IBM DFDL*<http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Kevin Safford/UK/IBM > To: Steve Hanson/UK/IBM@IBMGB, Alex Wood1/UK/IBM@IBMGB, > Date: 06/03/2014 14:40 > Subject: Fw: DL11A #3: (Question)[1](Open)(English)need > clarification > ------------------------------ > > > Hi > > I need some input from you chaps on this translation query. > > Could you suggest a rephrase, or explanation of point 1? > > Point 2 I assume is a typo for precede. It's present in the spec. Can you > confirm? > > > Regards > > *Kevin Safford* > Information Developer > I work Mondays to Thursdays only. > ------------------------------ > *Phone:* 44-1962-819217 > * E-mail:* *SAFFORDS(a)uk.ibm.com* <SAFFORDS(a)uk.ibm.com> > * My web site: **sites.google.com/site/kevinsafford*<https://sites.google.com/site/kevinsafford> > * Find me on:* [image: Twitter: https://twitter.com/KevSaff]<https://twitter.com/KevSaff> > > Hursley Park > Hursley, SO21 2JN > United Kingdom > > > ----- Forwarded by Kevin Safford/UK/IBM on 06/03/2014 14:37 ----- > > From: tctadm(a)cn.ibm.com > To: undisclosed-recipients:;, > Date: 06/03/2014 10:01 > Subject: DL11A #3: (Question)[1](Open)(English)need clarification > ------------------------------ > > > *** This mail is sent by TCT system automatically. Please do NOT reply > to this ID. *** Click here to read the article and response: > > *https://www.ibm.com/software/globalization/translationcommunications/BaseServlet.wss?taskName=ReadArticlePre&prrefnum=-1&arrefnum=7701628499*<https://www.ibm.com/software/globalization/translationcommunications/BaseSe…> > *Project code* DL11A *Project admin* terrychu *Article subject* need > clarification *Article type* Question *Category name* English/Source > files - PII,DL11AAAP002_Tooling *Testcase/Filename/IU * > plugins/com.ibm.dfdl.help/f1/dfdlPorperties.xml *Current status* Open *Priority > level * 1 *Stop-Ship* English:n/a *Originator * English > *Opened* by oya.argun(a)superonline.com / Trans. Vendor, Turkey / > English (oyargun) 2014-03-06 10:00:25 Hello, > > 1)The initiator with the longest match is the one that is selected as > having been 'found', with length-ties being resolved so that the matching > initiator is selected that is first in the order written in the schema. > > I am afraid this string is very complex, please re-phrase it if possible > or provide a comprehensive explanation. > > 2) In the following string I could not quite understand this phrase: > 'or the dfdl:escapeEscapeCharacter does not proceed the > dfdl:escapeCharacter'. What is meant by 'proceed' here?" > > On parsing the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are > removed from the data, unless the dfdl:escapeCharacter is preceded or the > dfdl:escapeEscapeCharacter does not proceed the dfdl:escapeCharacter. > > Thanks in advance > Oya > > ------------------------ Welcome to the TCT website: > ------------------------ > > *https://www.ibm.com/software/globalization/translationcommunications/index.jsp*<https://www.ibm.com/software/globalization/translationcommunications/index.…> > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >

2 1

Action 242 - propose simplification for length units in dfdl:valuelength and dfdl:contentLength functions
by Mike Beckerle 28 Apr '14

28 Apr '14

These functions currently take the desired result units as the second argument. It is very complicated if the length units of the element are bytes, to ask for this length to be recast in characters due to the fragment-of-a-character on the end problem. In some sense there is quotient/remainder here, which means there are multiple operations here, not just one. There is the equivalent of floor (as many characters as fit, then fillByte) ceiling (every byte filled from some character), remainder (how many left over bytes are there). I think this is silly unless we have use cases driving this. I believe this restriction matches all use-cases I can think of: Restriction: If the element (first argument) to either valueLength or contentLength has length units of characters -> can ask for valueLength or contentLength in any of characters, bytes, or bits bytes -> can ask for valueLength or contentLength in bytes or bits bits -> can only ask for valueLength or contentLength only in bits. and it is an SDE otherwise. This insures that the result of these functions is always well defined. I understand why there is a box sized in characters and I need to know its size in bytes, but the opposite I can't see a use case for, and it isn't crisply defined even. Ditto for length in bits. If the box is sized in bytes I might want that in bits (so * 8), but if the box is sized in bits, and I ask for bytes, what do I want if there is a fragment of a byte? ceiling? floor? remainder? Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

dfdl:contentLength and complex types with interior alignmentFill
by Mike Beckerle 28 Apr '14

28 Apr '14

dfdl:contentLength interacts pretty badly with alignment because alignmentFill regions mean you cannot know the size of a complex type element without knowing its starting position. It is very likely that one will have outputValueCalc that wants to evalutate dfdl:contentLength on an infoset item before the starting position for that item is known or knowable. To fix this we say that dfdl:contentLength(e) on complex-type infoset item e, is computed as if element e's alignment was first put into place along with the rest of the LeftFraming, and then the content length computed as if the complex content was unparsed starting from after that LeftFraming up to the final RightFraming In other words, the entire element has to be unparsed, not just the complex content of it, and then the part of that unparsed data that corresponds to the complex content is used as the return value from dfdl:contentLength. This works because element e's alignment (LeftFraming generally) is not within the complex content for element e. It is before the complex content of the element, so if the schema puts an explicit alignment on element e, then it will clearly be aligned that way when it is really unparsed, but the dfdl:contentLength function will also compute a length from the infoset value as if that same alignment were in place. I think that is about the best we can do. Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

Missing restriction in 'endOfParent' length kind ??
by Mike Beckerle 28 Apr '14

28 Apr '14

I think we're missing a restriction for 'endOfParent' Restriction: If an element has lengthKind 'endOfParent', then it's lengthUnits must be the same as the lengthUnits of the parent. SDE otherwise. Rationale: If a string has fixed length, and lengthUnits 'bytes' then currently we require the textStringPadCharacter to be a single-byte character. This prevents issues where there are bytes that can't be filled in by a pad character nor trimmed as a pad character. But if the string has 'endOfParent' length, the string could have lengthUnits 'characters', encoding utf-16, and would then typically have a 2-byte pad character. But the enclosing parent could have lengthUnits bytes and an odd fixed/specified length. Hence, a byte could be left over that cannot be filled with a pad character, nor trimmed as a pad character. It is noisome to have this corner case/excess byte problem for 'endOfParent' when we have so cleverly dodged it for all the specified-length kinds with the restriction to single-byte pad characters when lengthUnits is bytes. There is language in the spec about dealing with these trailing RightFill or ElementUnused regions. With the above restriction I believe RightFill becomes associated only with the hexBinary type being extended to fill out a specified-length or endOfParent box. The much more complex issue of RightFill appearing after text characters remains, but outside of the context where padding/trimming are involved. It is really problematic language. Frought with possibilities for misinterpretation. To me it is very preferable to just make 'endOfParent' consistent with other constraints associated with padding for specified-length elements. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1