- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function
by Steve Hanson 09 Sep '14

09 Sep '14

Mark Yes, that makes more sense. The existing wording says: dfdl:occursIndex() Returns the position of the current item within an array as an xs:long. The first element is at position 1. The function may be used on non-array elements. So I blindly used xs:long for the argument type as well. On consideration the argument type should be xs:nonNegativeInteger and so should the return type. Could be xs:positiveInteger, but as you say it's not used anywhere in the spec (no XPath constructor, etc). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mark Frost/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 03/09/2014 13:35 Subject: Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function Would a parameter type xs:positiveInteger (right value space) or xs:NonNegativeInteger (less wrong value space, and already used in DFDL) be more appropriate? Regards, Mark Mark Frost IBM United Kingdom Software Engineer Hursley Park IBM DFDL, IBM Integration Bus Winchester SO21 2JN Phone: +44 (0)1962 817009 England e-mail: frostmar(a)uk.ibm.com From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 03/09/2014 13:16 Subject: Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function Sent by: dfdl-wg-bounces(a)ogf.org Closed. After discussion, settled on an xs:long argument being a 1 based count up the parent axis. It is a processing error if the argument reaches beyond the root. It is a schema definition error if the argument is <= 0. Erratum 2.167 created. To be included in draft r23. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 02/09/2014 15:21 Subject: Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function Sent by: dfdl-wg-bounces(a)ogf.org I am currently implementing functionality in the area where dfdl:occursIndex( path ) will be in the Daffodil implementation. It doesn't look very hard to put in a specific argument that is restricted to be upward only and which identifies exactly which array. This function is a special case in any DFDL runtime no matter what, and will require a fair bit of test attention for any implementation so making it simpler by allowing only the innermost array to be examined isn't saving a lot of work really. Question: is a crazy path like ../foo/../../bar/.././.. (which just so happens to be equivalent to ../../..), a schema definition error? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 2, 2014 at 9:28 AM, Mike Beckerle <mbeckerle(a)tresys.com> wrote: ________________________________ From: Steve Hanson [smh(a)uk.ibm.com] Sent: Monday, September 01, 2014 11:08 AM To: Mike Beckerle Subject: Fw: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function 269 dfdl:occursIndex() function (Steve) 29/7: Clarify behaviour and decide whether an argument is needed to make it context independent. Noted that fn:position() also does not take an argument, but it returns position in current sequence and not position in array. 4/8: With Steve but consensus is that an argument is needed. 26/8: Mike to review. If an argument is needed, we should make sure this is in next published spec as the function is used in MIL-STD-2045 schemas. Regards Steve Hanson Architect, IBM DFDL< http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 01/09/2014 16:03 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Cc: dfdl-wg-bounces(a)ogf.org Date: 31/07/2014 10:57 Subject: Re: [DFDL-WG] clarification on dfdl:occursIndex() function ________________________________ Note: The reason why fn:position() takes no argument is because it gives position in the current sequence. Which is also why we have a separate DFDL specific function dfdl:occursIndex() which is intended for giving position in enclosing array. I think we need an argument that targets the specific enclosing array. Regards Steve Hanson Architect, IBM DFDL< http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 28/07/2014 23:51 Subject: Re: [DFDL-WG] clarification on dfdl:occursIndex() function Sent by: dfdl-wg-bounces(a)ogf.org ________________________________ I agree that it is inconvenient for the 'nearest array parent' to be inaccessible. However, experience with discriminators makes me fearful of any rule that includes the phrase 'nearest enclosing' :-) I think at least one other DFDL function allows the target of the function to be specified as an argument, but insists that the argument must be in the dynamic scope of the element ( i.e. its parent/grandparent etc ). I would be much happier with that solution for occursCountIndex(). I can think of a use case where it may be useful to get a consistent behaviour for occursCountIndex(). If a DFDL schema is generated from some other data format description then the model generation code may want to refer to the nth occurrence of something else in the model, where n is the occurs index of the current element - regardless of whether this particular element is an array. regards, Tim Kimber, From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 28/07/2014 23:27 Subject: [DFDL-WG] clarification on dfdl:occursIndex() function Sent by: dfdl-wg-bounces(a)ogf.org ________________________________ This function says it can be called on non-array elements. However, it does not say what the result is. If called when "." is not itself an array element there are only two possible behaviors consistent with the fact that it is explicitly allowed on non-array elements. The result has to be either (a) 1 (b) the occursIndex of the nearest enclosing array parent, or 1 if there is no enclosing array parent. I claim (a) is fairly pointless. You will just end up having to create newVariableInstances to carry the array current index downward into expressions. I cannot think of a use case where one would want to call occursIndex() polymorphically, i.e., where you want a number in the case of an array, but 1 otherwise. So (b) is the preferable behavior. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com<http://www.tresys.com/> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy< http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU-- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 4

Sept 2014 revision is on redmine
by Mike Beckerle 09 Sep '14

09 Sep '14

I completed the editing of "r23", accepted the changes to remove the change bars, and published the MS-WORD, PDF, and Zipped HTML versions to the DFDL redmine current documents list. I removed "draft-" prefix and "-r23" suffix from the titles and file names. https://redmine.ogf.org/dmsf/dfdl-wg?folder_id=5485 I still have "r23" with changes-tracked, in case there is a need. As usual, there have sometimes been glitches in MS-Word's save-to-PDF and save-to-HTML versions. I have only given those light review. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2014-09-09
by Steve Hanson 09 Sep '14

09 Sep '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13332?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

A gap in the behaviour for when escapeKind=escapeBlock
by Andrew Edwards 09 Sep '14

09 Sep '14

While working in the escapeBlock area, I seem to have found a gap in the definition for DFDL's behaviour when using escape blocks. >From section 13.2.1, relevant extracts are: When 'escapeBlock': On unparsing the entire data are escaped by adding dfdl:escapeBlockStart to the beginning and dfdl:escapeBlockEnd to the end of the data. The data is either always escaped or escaped when needed as specified by dfdl:generateEscapeBlock. If the data is escaped and contains the dfdl:escapeBlockEnd then first character of each appearance of the dfdl:escapeBlockEnd is escaped by the dfdl:escapeEscapeCharacter. and On parsing the dfdl:escapeBlockStart string must be the first characters in the (trimmed) data in order to activate the escape scheme. The dfdl:escapeBlockStart string is removed from the beginning of the data. Until a matching dfdl:escapeBlockEnd string (that is, one not preceded by the dfdl:escapeEscapeCharacter) is found in the data, any in-scope terminating delimiter encountered in the data is not interpreted as such, and any dfdl:escapeEscapeCharacters are removed when they precede an dfdl:escapeBlockEnd string. Now consider a a model where: escapeBlockStart="start" escapeBlockEnd="end" escapeEscapeCharacter="#" Then take a logical value of: A hash is a # When we serialize the value, we wrap the value with the escapeBlockStart and escapeBlockEnd, and we preceed any instance of the escapeBlockEnd within the data with an escapeEscapeCharacter. This then gives us the physical value "startA hash is a #end". If we were to parse that data, we see the "#end" as an escaped escapeBlockEnd and report that there is no escapeBlockEnd. The gap in the behavioural definition seems to be that the specification makes no claim to do anything to escape an instance of an escapeEscapeCharacter when serializing; There is nothing to catch the case of an escapeEscapeCharacter that isn't escaping an escapeBlockEnd but ends up doing it by circumstance. Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

OGF DFDL WG Call Agenda 2014-09-09
by Steve Hanson 09 Sep '14

09 Sep '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13332?download= Please review spec draft r22 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

argument for dfdl:occursIndex( path exp ) versus integer argument
by Mike Beckerle 04 Sep '14

04 Sep '14

There are unintended consequences from using an xs:nonNegativeInteger argument to dfdl:occursIndex. Specifically, it allows the number of steps upward to be computed dynamically e.g., dfdl:occursIndex(../foo/bar/nstepsToArray) or even dfdl:occursIndex($x) where x is some DFDL variable of integer type. Ooops. This is distinctly more powerful (in a bad way) than if one instead requires a path expression to an actual array node. It completely defeats static analysis such as XPath StaticTypeFeature kind of analysis, which is really important and valuable in DFDL since unlike other applications of XPath, we always have a schema. If we instead allow a path argument which must lead upward to an enclosing array, then that path expression can be statically typed, so it becomes legal to reject paths that go up past the root as non-type-able, when an implementation can detect this. Our prior discussion of nonsense paths such as dfdl:occursIndex(../foo/..) where there is no foo peer element in the schema, those are all non-type-able, so a DFDL implementation is free to reject them statically or dynamically. So we really don't have to worry about them and try to re-jigger the arg to an integer so as to avoid them. My conclusion from this is we should go back to the argument being a path expression, that must lead to an enclosing array. It can be just "." if the current element is itself an array. Comments?

1 0

Re: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() function
by Mike Beckerle 03 Sep '14

03 Sep '14

I am currently implementing functionality in the area where dfdl:occursIndex( path ) will be in the Daffodil implementation. It doesn't look very hard to put in a specific argument that is restricted to be upward only and which identifies exactly which array. This function is a special case in any DFDL runtime no matter what, and will require a fair bit of test attention for any implementation so making it simpler by allowing only the innermost array to be examined isn't saving a lot of work really. Question: is a crazy path like ../foo/../../bar/.././.. (which just so happens to be equivalent to ../../..), a schema definition error? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Sep 2, 2014 at 9:28 AM, Mike Beckerle <mbeckerle(a)tresys.com> wrote: > > ________________________________ > From: Steve Hanson [smh(a)uk.ibm.com] > Sent: Monday, September 01, 2014 11:08 AM > To: Mike Beckerle > Subject: Fw: [DFDL-WG] Action 269: clarification on dfdl:occursIndex() > function > > 269 > dfdl:occursIndex() function (Steve) > 29/7: Clarify behaviour and decide whether an argument is needed to make > it context independent. Noted that fn:position() also does not take an > argument, but it returns position in current sequence and not position in > array. > 4/8: With Steve but consensus is that an argument is needed. > 26/8: Mike to review. If an argument is needed, we should make sure this > is in next published spec as the function is used in MIL-STD-2045 schemas. > > Regards > > Steve Hanson > Architect, IBM DFDL< > http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 01/09/2014 16:03 ----- > > From: Steve Hanson/UK/IBM > To: dfdl-wg(a)ogf.org, > Cc: dfdl-wg-bounces(a)ogf.org > Date: 31/07/2014 10:57 > Subject: Re: [DFDL-WG] clarification on dfdl:occursIndex() function > > ________________________________ > > > Note: The reason why fn:position() takes no argument is because it gives > position in the current sequence. Which is also why we have a separate DFDL > specific function dfdl:occursIndex() which is intended for giving position > in enclosing array. I think we need an argument that targets the specific > enclosing array. > > Regards > > Steve Hanson > Architect, IBM DFDL< > http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> > tel:+44-1962-815848 > > > > From: Tim Kimber/UK/IBM@IBMGB > To: dfdl-wg(a)ogf.org, > Date: 28/07/2014 23:51 > Subject: Re: [DFDL-WG] clarification on dfdl:occursIndex() function > Sent by: dfdl-wg-bounces(a)ogf.org > ________________________________ > > > > I agree that it is inconvenient for the 'nearest array parent' to be > inaccessible. However, experience with discriminators makes me fearful of > any rule that includes the phrase 'nearest enclosing' :-) > I think at least one other DFDL function allows the target of the function > to be specified as an argument, but insists that the argument must be in > the dynamic scope of the element ( i.e. its parent/grandparent etc ). I > would be much happier with that solution for occursCountIndex(). > > I can think of a use case where it may be useful to get a consistent > behaviour for occursCountIndex(). If a DFDL schema is generated from some > other data format description then the model generation code may want to > refer to the nth occurrence of something else in the model, where n is the > occurs index of the current element - regardless of whether this particular > element is an array. > > regards, > > Tim Kimber, > > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> > Date: 28/07/2014 23:27 > Subject: [DFDL-WG] clarification on dfdl:occursIndex() function > Sent by: dfdl-wg-bounces(a)ogf.org > ________________________________ > > > > This function says it can be called on non-array elements. However, it > does not say what the result is. > > If called when "." is not itself an array element there are only two > possible behaviors consistent with the fact that it is explicitly allowed > on non-array elements. > > The result has to be either > > (a) 1 > (b) the occursIndex of the nearest enclosing array parent, or 1 if there > is no enclosing array parent. > > I claim (a) is fairly pointless. You will just end up having to create > newVariableInstances to carry the array current index downward into > expressions. > > I cannot think of a use case where one would want to call occursIndex() > polymorphically, i.e., where you want a number in the case of an array, but > 1 otherwise. > > So (b) is the preferable behavior. > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > www.tresys.com<http://www.tresys.com/> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the OGF Intellectual Property Policy< > http://www.ogf.org/About/abt_policies.php> > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU-- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >

3 2

Fw: Fw: Action 261
by Steve Hanson 03 Sep '14

03 Sep '14

Closed. Agreed that 16.1.4 that says that occursCountKind 'expression' behaves like 'parsed' when unparsing is misleading, and needs rewording. The count is the number of occurrences in the augmented infoset. Separators are never suppressed, as already stated in 14.2.3. Erratum 3.11 updated. To be included in draft r23. Proposal to evaluate the occursCount expression during unparsing as an extra 'validation' check was not accepted. However, Mike has noticed that the spec does not explicitly state that asserts and discriminators are not evaluated during unparsing. Erratum 2.166 created. To be included in draft r23. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 03/09/2014 13:15 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 29/08/2014 22:20 Subject: Re: [DFDL-WG] Fw: Action 261 I also reviewed this. I like the symmetry improvement of OCK expression unparsing being 'never' suppression policy and taking the occurs count from the number of items in the augmented infoset. I don't like the evaluation of occurs expression on unparsing as a check, and length expression evaluation on unparsing as a check. This allows unparsing of data that cannot be parsed again. But there are many many such holes, and we can't plug them all. We don't evaluate assert/discriminator tests when unparsing, so why these other checks? (Note: I just did a quick search through the spec for all uses of "assert", and didn't find a statement that says they are only evaluated when parsing.) If we want to give the option for unparsing to evaluate occurs and length expressions (and assets/discrims too?) that's worth consideration, but I'd prefer that this be an implementation-specific flag/mode and not part of the standard (for now.) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Thu, Aug 28, 2014 at 10:13 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Please review for Tuesday's WG call... Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2014 15:01 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 06/08/2014 13:50 Subject: Fw: [DFDL-WG] Action 261 In the spec as it stands, occursCountKind 'expression' has behaviour like occursCountKind 'fixed' when parsing (ie, there is a count that is fixed before the occurrences are parsed), but is also stated to have behaviour like occursCountKind 'parsed' when unparsing (ie, there is no count that is fixed before occurrences are unparsed). In terms of expected separators, it behaves like 'never' when parsing, but like 'anyEmpty' when unparsing. This leads to undesirable asymmetric behaviour, such as this example. Data stream is 5,A,B,,D,E. To preserve indexes in the infoset after parsing, I make the array element nillable with ES as nil value. But on unparsing I will get 5,A,B,D,E because the 'anyEmpty' behaviour has suppressed the output of the separator for the zero-length nil value. Not only will modellers find this behaviour odd, it breaks dfdl:outputValueCalc on the count element if fn:count() is used. (Same happens if not nillable but minOccurs > 2 so empty string ends up in the infoset). The proposal is that 'expression' behaves like 'never' on unparsing, the count being taken as the number of items in the augmented infoset. That way the application or outputValueCalc can be certain that no separators will be suppressed, and outputValueCalc will work. It was proposed further down this email thread that the expression is evaluated at the start of unparsing the element and used to obtain the count. There are scenarios where that causes problems though, hence the above is preferred. However, to ensure that the output data stream can be re-parsed, it is proposed that the expression is evaluated at the end of the unparsing of the element (and any elements with expressions dependent on it), and if it fails to match the number that was output, it is a processing error. Similarly for lengthKind 'explicit' where length is an expression. The element continues to be considered variable length on unparsing as the spec says today, but the expression is evaluated at the end of the unparsing of the element, and if it fails to match the length that was output, it is a processing error. It would be great to do the analogous thing for lengthKind 'pattern' but it is not always possible as some patterns look ahead beyond the element content in order to match. So it could be done for some patterns but not all. Implementation-dependent? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/08/2014 13:08 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 25/06/2014 11:33 Subject: Fw: [DFDL-WG] Action 261 261 Implied separatorSuppressionPolicy for occursCountKind 'expression' (All) 10/6: Spec says it is 'never' (positional sequence) but you have to parse to identify the position, so isn't that non-positional? 17/6: Some other issues noted around 'expression' as per email thread. IBM have discussed this internally and will submit a proposal. As was noted in the email for Action 260, if it is decided that the meaning of "Each occurrence in the sequence can be identified by its position in the data" is more strictly that an observer of the raw data can identify an occurrence of an element in the sequence solely by counting separators then that would appear to make dfdl:occursCountKind 'expression' more like 'parsed', and not eligible to be in a Positional sequence. But if the meaning is a parser does not have to speculate to identify an occurrence of an element in the sequence then it can be in a Positional sequence. While discussing the nature of 'expression' it was noted that it is very easy for a DFDL user to create a data stream from an infoset and for that data stream to be un-parse-able. If dfdl:outputValueCalc is not used, then the element(s) in the infoset must be correctly set manually to match the number of occurrences. The same observation applies to dfdl:lengthKind 'explicit' where dfdl:length is an expression. To address this, IBM proposes the following changes to the DFDL specification for occursCountKind 'expression' when unparsing: When all occurrences have been obtained from the infoset (and defaulting applied if needed), the occursCount expression is evaluated If any element that is referenced by the expression has dfdl:outputValueCalc, then that expression is (re-)evaluated as part of the above; given that the number of occurrences is now known, the outputValueCalc expression should now evaluate successfully If the result of the occursCount does not match the number of occurrences it is a processing error This ensures the integrity of the data stream (the non-outputValueCalc use case), while still allowing the infoset to dictate the count (the outputValueCalc use case). Similarly, IBM proposes the following changes to the DFDL specification for lengthKind 'explicit', where length is an expression, when unparsing: When an element has been obtained from the infoset (and defaulting applied if needed) and unparsed but before padding is applied, the expression is evaluated to give a length. If any element that is referenced by the expression has dfdl:outputValueCalc, then that expression is (re-)evaluated; given that the unpadded length is now known, the outputValueCalc expression should now evaluate successfully Now that we have a length, the unparser behaviour is the same as if the length was obtained from a fixed dfdl:length value. (That is, truncation, padding, filling or error depending on other property settings). This ensures the integrity of the data stream (the non-outputValueCalc use case), while still allowing the infoset to dictate the length (the outputValueCalc use case). Note that it means the 'awkward' behaviour whereby lengthKind 'explicit' (expression) is a specified length when parsing but variable length when unparsing is avoided - it is now always specified length. IBM believes that none of the above should impose additional burden on implementers, as no brand new behaviour is being added. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 25/06/2014 10:24 ----- From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Alex Wood1/UK/IBM@IBMGB, Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB, Date: 16/06/2014 22:08 Subject: Re: [DFDL-WG] Action 261 I think this needs to be discussed before the meeting tomorrow. I wanted to avoid turning this thread into a monster - but I don't think it's possible to discuss without the context so I've continued the thread as before with <tk> tags. I'm clear in my own mind that I understand the issues now. We should give priority to the separatorSuppressionPolicy question because the current rules are close to being unimplementable. The occursCountKind and lengthKind questions are important but are at least implementable as they stand. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 11/06/2014 16:12 Subject: Re: [DFDL-WG] Action 261 Replies in <smh> tags Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/06/2014 13:58 Subject: Re: [DFDL-WG] Action 261 Sent by: dfdl-wg-bounces(a)ogf.org comments in <tk>tags regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 11/06/2014 10:47 Subject: Re: [DFDL-WG] Action 261 Some thoughts on this... I agree that the definition of positional sequence in the spec needs tightening as it is ambiguous as it stands and could be interpreted as a) or b). If we adopted b) then that would appear to allow 'expression' to appear in a positional sequence, but wouldn't it also allow 'stopValue'? <tk>Yes - according to definition b) stopValue would be allowable in a positional sequence. We could still disallow it if we do not believe there is any benefit in allowing it. I don't believe it introduces any particular complexities for an implementer.</tk> occursCountKind 'expression' is analogous to lengthKind 'explicit' with an expression and to lengthKind 'prefixed'. Both these lengthKinds are classified as 'specified length' when parsing but 'variable length' when unparsing. We are observing that occursCountKind 'expression' is like 'fixed' when parsing but not quite so like 'fixed' when unparsing - which is why section 16 groups 'expression' with 'parsed' for unparsing. <tk>Yes - we took a decision that the unparser should ignore the expression in lengthKind/occursCountKind, and just output whatever data happens to be in the info set. I'm not sure that it saves a lot of effort in the implementation and it certainly is not easy to justify as a consistent behaviour. For me, the unparser should treat lengthKind='explicit' the same way whether the value is static or calculated. And the unparser should treat occursCountKind='expression' the same way as occursCountKind='fixed'. </tk> When unparsing occursCountKind 'expression' you don't always have the calculated array length N. If the infoset was derived from XML, there is likely no 'count' element, just a bunch of elements with the same name that make up the 'array'. DFDL gives you the choice whether to a) manually set the count element, or b) have the unparser set it automatically via outputValueCalc. In the former case, you can create a document that can not be parsed; the unparser could check the 'count' element matches the infoset, but that would involve reverse engineering an arbitrarily complex expression and is why the specification does not say that. <tk>It would involve evaluating the expression. In most cases, that will not require any lookahead because the Length/Count field will precede the array or element. Not sure where the reverse engineering comes in?</tk> <smh>I see what you are saying. Just evaluate the expression and see what it gives for N. That handles case a) but not b) where I explicitly want the unparser to set the count via outputValueCalc - which is presumably referring to the number of elements in the array, which is not known. For case b) N has to be the number in the infoset. Given that we have to support case b) the unparser can not treat occursCountKind 'expression' exactly the same as 'fixed' when unparsing.</smh> <smh>Similarly with lengthKind 'explicit' with an expression. For the equivalent to case a) the length is known which makes the length fixed, but for the equivalent of case b) with outputValueCalc the length is not known so it is variable. When this was discussed in the past, it was decided not to bifurcate the expression scenario. Hence the spec is the way it is. </smh> <tk> That helps. So your belief is that case a) is workable but case b) is not because the number of elements in the array is not known. I don't think that holds up under scrutiny. In case b), the outputValueCalc expression cannot be evaluated until all of the array has been received. So if the info set is received as an event stream then the unparser must wait until the array completes before evaluating the expression. Note that the 'count' ( or 'length' ) field cannot be serialized until its value is known. But the array ( or variable-length field ) comes *after* the count/length field. By the time unparsing of the array/field begins, the value *is* known. The implementation of case a) is actually less straightforward. If the field is an array length and the value is less than the number of items in the info set then I think the unparser must issue an error. If greater then the unparser could output default values ( if available ) or delimiters ( if the parent sequence is a positional sequence with a delimiter ) or else an error if neither are possible. More simply, the unparser could simply insist that the value must correctly describe the data, and I think that's a reasonable rule. Similarly for the length. If the length of the *unparsed* value is greater than the value in the info set then the unparser should issue an error. If it is shorter and the field is simple then pad characters could be added. But again, I think real-world usage of length counts suggests that padding is unlikely to be wanted, and the unparser should simply insist on the length field being correct. </tk> Here's a real example of such an expression (albeit with lengthKind 'explicit' but the principle is the same): dfdl:length="{xs:nonNegativeInteger(fn:floor((../Length + 1) div 2))}" Alex brought up the case where the expression evaluates to 0. In a positional sequence, would you still expect a delimiter for this case? <tk>Yes, unless it is in the trailing optional region of the group and SSP='trailingEmpty'. In a positional sequence, every delimiter must be present until suppression begins ( if allowed )</tk> If 'yes' then the resultant zero length string must be treated as the 'absent representation' and ignored. If 'no' then is the sequence still positional? <tk>I don't understand the point. Why would it not be the 'empty representation'? Why must it be 'ignored' if it does happen to be the 'absent representation'? What does 'ignored' mean?</tk> <smh>The point is that the parser has been told there are 0 occurrences. So it would be odd if the infoset ended up containing an occurrence, which can happen if the normal nil/empty rules are followed. (Eg, nilValue=%ES;). <tk> If the occursCount expression evaluates to zero then the parser will not attempt to parse even one occurrence of the array. That's the natural meaning of 'zero occurrences'. So nothing would go into the info set apart from the 'count' field. This is entirely consistent with my definition b) of 'positional' ( the identity of every delimited field is known before parsing of the field begins ). </tk> Hence the 0 occurrence case must treat it as absent which means nothing is added to the infoset. Take the ISO8583 bitmap use case - if the bit is 0 we must not try to parse anything at all for that element - it is totally absent.</smh> <tk>Yes - that's exactly my point. The fact that there is ( or could be ) a delimiter after the 'count' field is irrelevant.</tk> Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 10/06/2014 21:22 Subject: [DFDL-WG] Action 261 Sent by: dfdl-wg-bounces(a)ogf.org Implied separatorSuppressionPolicy for occursCountKind 'expression ' (All) 10/6: Spec says it is 'never' (positional sequence) but you have to parse to identify the position, so isn't that non-positional? I think there are two alternative definitions of 'positional': a) the identity of every delimited field is known before parsing of the sequence group begins b) the identity of every delimited field is known before parsing of the field begins As an implementer, b) is sufficient because it means that the parser never needs to backtrack while parsing the group. a) allows the field identities to be statically known, but that is less important - it does not allow optimised extraction of a particular field as would be the case for a fixed-length group ( the possibility of escaped separators/terminators means that every character will need to be scanned anyway ). It may sound like a small point, but it affects two decisions 1. whether ock='expression' should be allowed within a positional sequence group ( action 261 ) 2. what the behaviour of the unparser should be w.r.t. ock='expression'. My own feeling is that ock='expression' should be treated almost exactly like ock='fixed', except that the calculated array length N is used instead of maxOccurs. - When parsing a positional sequence group it should cause N delimiters to be expected for the array. - When unparsing a positional sequence group it should cause N delimiters to be written. These rules are consistent and straightforward to describe and implement. The current rule ( unparser outputs the occurrences that are in the info set only ) allows the unparser to write a document that cannot be parsed using the same schema. regards, Tim Kimber, ----- Forwarded by Tim Kimber/UK/IBM on 10/06/2014 20:34 ----- From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 10/06/2014 17:57 Subject: [DFDL-WG] OGF DFDL WG Call Minutes 2014-06-10 Sent by: dfdl-wg-bounces(a)ogf.org Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13263?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] DFDL Action 264- ES in delimiters
by Steve Hanson 03 Sep '14

03 Sep '14

Closed. Steve @ Tresys submitted proposal, one comment from Steve @ IBM, which was accepted. Erratum 2.148 updated. To be included in draft r23. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 29/08/2014 08:46 Subject: Re: [DFDL-WG] DFDL Action 264- ES in delimiters I've re-attached the original discussion around this action to the end of this thread, it had got dropped along the way. The key argument for allowing ES is that "%WSP*;" is the same as the list "%ES; %WSP+;" (on parsing, longest match is taken; on unparsing nothing is output). I think Steve's proposal needs one additional restriction - ES as the only string literal in the list should be an SDE. In other words, there has to be the possibility of an actual terminator. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 29/08/2014 00:29 Subject: [DFDL-WG] DFDL Action 264- ES in delimiters Sent by: dfdl-wg-bounces(a)ogf.org From: Stephen Lawrence Sent: Wednesday, August 27, 2014 3:20 PM To: Mike Beckerle Subject: Re: DFDL Action 264- ES in delimiters Mike, The properties that do not allow ES but do allow WSP* (with restrictions) that I could find are: - initiator - terminator - separator - textStandardZeroRep The WSP* restriction for textStandardZeroRep makes sense to not allow ES. so I think initiator, terminator, and separator are the only properties where it makes sense to allow ES with the same restrictions as WSP*. So, here's the Erratum 2.1XX Section 12.2, 14.2. Clarify DFDL Character Class entities allowed in delimiters The initiator, terminator, and separator properties can have the ES character class entity, subject to the same restrictions as WSP*. Hopefully that's okay and makes sense, - Steve -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg ----- Forwarded by Steve Hanson/UK/IBM on 29/08/2014 08:40 ----- From: Steve Hanson/UK/IBM To: "Mike Beckerle" <mbeckerle(a)tresys.com>, Date: 28/08/2014 15:08 Subject: Fw: [DFDL-WG] ES in delimiters Reminder to follow-up with Steve Lawrence for next WG call please. 264 Use of ES entity in delimiters (Steve L) 15/7: Proposal to allows ES in delimiters subject to same restrictions as WSP* as inconsistent otherwise. Due diligence needed to see whether there is anywhere else in the spec that could also allow ES because it also allows WSP*. 22/7: With Tresys 29/7: Still with Tresys. Noted that the IBM's comments on the MIL-STD-2045 additional features document used this feature to negate the need for lengthKind 'fixedOrTerminated' (though WSP* would also work). ... 26/8: Still with Steve L Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/08/2014 15:06 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 30/06/2014 18:12 Subject: Re: [DFDL-WG] ES in delimiters On the face of it, yes it does seem inconsistent. The point being that "%WSP*;" is the same as "%ES; %WSP+;" (on parsing, longest match is taken; on unparsing nothing is output). In our NACHA xsds we have: <dfdl:defineFormat name="NACHARecord"> <dfdl:format lengthKind="explicit" length="94" terminator="%WSP*;"/> </dfdl:defineFormat> But strictly speaking it should be this because it's an optional new line that appears rather than any white space: <dfdl:defineFormat name="NACHARecord"> <dfdl:format lengthKind="explicit" length="94" terminator="%ES; %NL;"/> </dfdl:defineFormat> If we allow ES then ES on its own as the only string literal in the list should be an error. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 26/06/2014 18:17 Subject: [DFDL-WG] ES in delimiters Sent by: dfdl-wg-bounces(a)ogf.org Steve Lawrence from the Daffodil project has observed that we're not particularly consistent about delimiters and the %ES; entity. We allow WSP* with caveat that you can't have initiatedContent, but we disallow ES. There is a use case for optional delimiters e.g., initator="%CR;%LF; %ES;" meaning absorb and discard an optional CRLF. Without %ES we end up having to model this syntax as an optional hidden element. Thoughts? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2014-09-02
by Steve Hanson 03 Sep '14

03 Sep '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13330?download= Last call for reviews of spec draft r22 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0