- dfdl-wg - lists.ogf.org

clarify array expressions
by Mike Beckerle 15 Feb '13

15 Feb '13

Some folks are trying to create a DFDL model for IMF (internet message format ... basically email) There is this header structure which really wants unordered sequences. Since we don't have those yet I told them to use an array of choice like this: <xs:element name="hdrArray" minOccurs="1" maxOccurs="25"> <xs:complexType> <xs:choice> <xs:element ref="TO_HDR"/> <xs:element ref="FROM_HDR" /> </xs:choice> ... But then there are these contraints, like there should be only one FROM address, but there can be several TO addresses. So, one might want to write a DFDL expression using an XPath-like query path in it. E.g., <xs:assert>{ dfdl:occursCount( hdrArray/FROM_HDR ) eq 1 }</xs:assert> Here's the question: Is the DFDL expression in this assert legal? I advised them that no, it is not, because hdrArray is an array and isn't being indexed, while FROM_HDR is not an array, but is being used as the argument to dfdl:occursCount. I think DFDL's expression language is simply not powerful enough to express these constraints. I'm just looking for affirmation or discussion if there is more to say here.

2 1

dfdl:textOutputMinLength
by Steve Hanson 14 Feb '13

14 Feb '13

Here's the description for this property from the spec, including a change for errata 2.100. I am concerned about the red sentence about length units. textOutputMinLength Non-negative Integer. Only used when dfdl:textPadKind is 'padChar' and dfdl:lengthKind is 'delimited', 'prefixed', 'pattern' or 'endOfParent' or 'explicit' and dfdl:length is an expression, and type is not xs:string Specifies the minimum representation length during unparsing for simple types that do not allow the xs:minlength facet to be specified. The units are specified by the dfdl:lengthUnits property. If dfdl:textOutputMinLength is zero or less than the length of the representation text then no padding occurs. Annotation: dfdl:element, dfdl:simpleType However, when parsing, dfdl:lengthUnits is only used for a 'specified length' so that's dfdl:lengthKind 'explicit', 'implicit' and 'prefixed'. When 'delimited', 'pattern' or 'endOfParent' dfdl:lengthUnits is not used. I think we should be saying: "For dfdl:lengthKind 'delimited', 'pattern' and 'endOfParent' the length units are characters. For other dfdl:lengthKinds the length units are specified by the dfdl:lengthUnits property." Same argument applies to XSD minLength facet for xs:strings. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Possible issues for xs:string when dfdl:lengthUnits is 'bytes' and non-SBCS encodings
by Steve Hanson 14 Feb '13

14 Feb '13

1) For xs:string if dfdl:lengthKind is 'implicit' then xs:maxLength is used to extract N units from the data. If dfdl:lengthUnits is 'bytes' then N bytes are extracted. If validation is switched on xs:maxLength is also used to validate that no more than N characters appear in the infoset. This seems problematic where the dfdl:encoding is non-SBCS. 2) For xs:string if dfdl:lengthKind implies a variable length on output and dfdl:textPadKind is not 'none' then xs:minLength is used to ensure that at least N units are output. If dfdl:lengthUnits is 'bytes' then N bytes are written to the data. If validation is switched on xs:minLength is also used to validate that at least N characters appear in the infoset. Again this seems problematic where the dfdl:encoding is non-SBCS. Should we disallow the combinations that actually cause a problem? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

AUTO: Sunil Dandamudi is out of the office (returning 02/15/2013)
by Sunil Dandamudi 13 Feb '13

13 Feb '13

I am out of the office until 02/15/2013. Note: This is an automated response to your message "dfdl-wg Digest, Vol 80, Issue 14" sent on 02/13/2013 10:00:02. This is the only notification you will receive while this person is away.

1 0

OGF DFDL WG Call Minutes 2013-02-12
by Steve Hanson 13 Feb '13

13 Feb '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/13006?download= Errata document is now at v011.3 here: http://redmine.ogf.org/dmsf_files/8104?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] representation properties vs. format properties
by Steve Hanson 12 Feb '13

12 Feb '13

I have looked at this as part of my recent review of the updated DFDL spec. My conclusions: - The term 'DFDL property' applies to all DFDL properties carried on any DFDL annotation. We should use it in preference to 'attribute' even when an XML attribute is the only way that a DFDL property may be specified. (This is in keeping with XML Schema, where for example the 'name' property of a component is given by an XML attribute representation 'name'). - Similarly, the term 'XSD property' applies to all XML Schema properties, such as default, fixed, minOccurs, maxOccurs, etc. - The term 'format property' applies to all DFDL properties carried on DFDL format annotations. (One could envisage similar terms 'statement property' and 'defining property'). - The term 'representation property' applies to all format properties that are used to describe grammar regions. The revised grammar uses 'Rep' in grammar terms like SimpleElementEmptyRep etc. - The term 'non-representation property' applies to all format properties that are not representation properties, specifically dfdl:ref, dfdl:hiddenGroupRef, dfdl:elementId, dfdl:choiceBranchRef, dfdl:inputValueCalc, dfdl:outputValueCalc, dfdl:escapeSchemeRef. The one non-standard property is dfdl:escapeSchemeRef, because it is a non-representation property yet can still be placed into scope. All the other non-representation properties can not be placed into scope. I am ok with that, it just means that whether a format property is a non-representation property is decoupled from whether it can be placed into scope or not, and should be considered on a case-by-case basis. Alternatively we can say that dfdl:escapeSchemeRef is a representation property even though it points at a defining annotation (and so is similar in this respect to dfdl:prefixLengthType). - dfdl:representation must be qualified by the namepace prefix when referred to in the spec, and preferably either preceded by 'property' or used on its own, rather than followed by 'property', to avoid any ambiguity. The term 'attribute' should be used only when describing how a property is represented in XML terms. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 29/01/2013 10:12 Subject: Re: [DFDL-WG] representation properties vs. format properties We do have an actual problem in the spec, as the term 'representation property' is being used to mean both any DFDL property (other than 'ref') and also the property 'representation'. This is best seen in Table 13.1. However I am only seeing one example of the term 'format property' - in section 24 ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 28/01/2013 18:27 Subject: [DFDL-WG] representation properties vs. format properties Sent by: dfdl-wg-bounces(a)ogf.org We use both the terms representation property and format property extensively. Do we want to declare these to mean the same thing as each other, or globally replace one with the other. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2013-02-12
by Steve Hanson 12 Feb '13

12 Feb '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13006?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Backtracing behavior for optional elements
by Steve Hanson 12 Feb '13

12 Feb '13

If a processing error occurs for an optional element in a sequence, the speculative behaviour of the DFDL parser says that the optional element is assumed not to be present, and the next alternative in the sequence is tried. That is fine when there are no separators involved, but we need to clear on what happens when there are separators. 1) Positional separators (separatorSuppressionPolicy is 'never', 'trailingEmpty' and 'trailingEmptyStrict'). The key point about positional separators is that they are expected in the data, so if an error occurs while parsing the optional element, it does not make sense to backtrack to the start offset the element and try to match the next element. Yes there's a point of uncertainty in the sense that the element is either there or it has empty representation, but if an error occurs I think it must be treated as a hard error, and not cause backtracking. 2) Non-positional separators (separatorSuppressionPolicy is 'anyEmpty'). This behaves like the non-separator case and the next alternative in the sequence is tried from the start offset. However, because 'anyEmpty' behavior is lax, it is possible that the next thing in the data is a separator, so the parser must cater for that when the element is found to have empty representation. But if an error occurs establishing representation, I think the parser should just backtrack and try to match the next element. Does that sound correct? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 3

Fw: what do we allow in DFDL expressions after "/"
by Steve Hanson 11 Feb '13

11 Feb '13

After checking with an XPath expert about the use of ContextItemExpr, here's a consolidated proposal for the EBNF changes. Please review for next WG call: PathExpr ::= (("/" RelativePathExpr?) | RelativePathExpr) | FilterExpr StepExpr ::= AxisStep AxisStep ::= (ReverseStep | ForwardStep) Predicate? FilterExpr ::= PrimaryExpr Predicate? AbbrevForwardStep ::= NodeTest | ContextItemExpr ParenthesizedExpr ::= "(" Expr ")" Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/02/2013 12:17 ----- From: Andrew Coleman/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Date: 11/02/2013 12:11 Subject: Re: Fw: [DFDL-WG] what do we allow in DFDL expressions after "/" Hi Steve, Yes, that change looks fine. It's hard to say why the original XPath grammar didn't define '.' as part of AbbrevForwardStep. There is often more than one way of specifying a production rule. Although a '.' in the middle of a path expression makes no difference to the result, it is useful to be able to tolerate this syntax in case the expression has been created by a naive code generator. Best regards, - Andy __________________________________________ Andrew Coleman WebSphere Message Broker Development IBM Hursley Park From: Steve Hanson/UK/IBM To: Andrew Coleman/UK/IBM@IBMGB, Date: 11/02/2013 09:41 Subject: Fw: [DFDL-WG] what do we allow in DFDL expressions after "/" Hi Andy Please can I trouble you for advice on an XPath 2.0 question? As you know DFDL allows a subset of XPath 2.0. In the DFDL spec we took the EBNF from the XPath 2.0 spec and simplified it for our needs (go to http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp and search for 'relativepathexpr'). However it has been pointed out that what we have still allows too much, so we want to simplify it further. Specifically the term FilterExpr can appear in the middle of a RelativePathExpr because of the term StepExpr. A proposal is to only allow FilterExpr as the last term in a path, but that means we couldn't use ./xxx because the use of . (ContextItemExpr) is not part of AbbrevForwardStep, but only part of PrimaryExpr and hence FilterExpr. So it was suggested that the change directly below was made. AbbrevForwardStep ::= NodeTest | ContextItemExpr My question is, why didn't XPath 2.0 allow this in the first place? I am wondering if there is some subtlety going on that we break if we change AbbrevForwardStep in this manner. Thoughts? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/02/2013 09:24 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 07/02/2013 11:50 Subject: Re: [DFDL-WG] what do we allow in DFDL expressions after "/" Agree that Predicate is optional, and that is the case in XPath 2.0. The bug crept into our spec because we removed an intermediate construction PredicateList which is defined as Predicate*. However I would prefer that we marked the optionality on the user of Predicate rather than on Predicate itself. AxisStep ::= (ReverseStep | ForwardStep) Predicate* PrimaryExpr returns a sequence, so you need to allow a predicate. (DFDL does not allow a sequence to be returned from a DFDL expression, but they can appear in internal stages of an expression). Also, we need to allow .[n]. FilterExpr ::= PrimaryExpr Predicate* IBM DFDL has many tests where we use ./foo and I am sure that will be being used by users. It makes it clear that the path is relative. To handle this, like you I considered adding ContextItemExpr to AbbrevForwardStep, as syntactically that seems to work, but then I wondered why XPath hadn't done it that way. I'd like to check with our XPath expert Andy Coleman to see whether there is some subtlety going on around contexts. Is there any harm in allowing () ? I'd leave that as it is. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 06/02/2013 18:13 Subject: Re: [DFDL-WG] what do we allow in DFDL expressions after "/" The fact that in StepExpr a FilterExpr is an alternative to AxisStep is the big problem with this grammar. It allows many things to show up as path steps that we don't want to allow. At minimum, we need to change the FilterExpr production to drop Predicate, as I believe in no context do we allow such a predicate. No-op path steps like foo/./bar or ./bar or foo/. are also legal in XPath syntax technically, but I see no reason for us to allow them, and this is what we get if we simply state that in DFDL FilterExpr cannot appear inside a multi-step PathExpr. If we feel we must allow these, then I would just add ContextItemExpr as an alternative to NodeTest in AbbrevForwardStep. So, we can leave the rest of the productions as is, but with a clarifying note that FilterExpr in DFDL is limited i.e., that they cannot be used inside multi-step paths as the grammar would imply. Rather, they must appear in isolation (no "/" on either side). However, this grammar structure is very unmotivated in that case, restructuring the StepExpr and PathExpr as you have suggested below is a better choice as far as I am concerned. And we can highlight somehow (I suggest bold font) the productions or changes to productions that make it different from the XPath grammar. So here's the contemplated set of changes: drop FilterExpr as an alternative StepExpr ::= AxisStep add FilterExpr as an alternative (or we could just add PrimaryExpr instead) PathExpr ::= (("/" RelativePathExpr?) | RelativePathExpr) | FilterExpr drop predicate possibility FilterExpr ::= PrimaryExpr add "." as available forward step (I am ambivalent about this one, but I'll admit I have written "./foo" before) AbbrevForwardStep ::= NodeTest | ContextItemExpr I also think there is a bug in the grammar. Right now AxisStep requires a Predicate. The Predicate should be optional. The original XPath grammar uses this PredicateList thing to deal with zero or more predicates. We want zero or 1 predicates, Simplest to perhaps do Predicate ::= ("[" Expr "]")? The ? makes it optional. There is also a typo in the ReverseAxis production (dangling ") at the end. Lastly, do we allow () as an expression? Right now the grammar allows this, because ParenthesizedExpr is "(" Expr? ")" but I'm not sure () is meaningful in DFDL except as part of a function call with no arguments. i.e., foo() which is allowed by a separate production for function call syntax. On Wed, Feb 6, 2013 at 5:05 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Mike, More correctly, you mean Step[Index]/Step[Index]/Step[Index], but that's fine as that is handled by the definition of AxisStep as it includes Predicate. However removing FilterExpr entirely removes PrimaryExpr and all it brings with it, like literals and function calls and the use of "." so your change on its own simplifies things too much. What I think you would need is: StepExpr ::= FilterExpr | AxisStep PathExpr ::= (("/" RelativePathExpr?) | RelativePathExpr) | FilterExpr I can see (at least) one problem with only allowing FilterExpr to appear on its own, namely that you lose the ability to use "." in conjunction with a path, so even the above is not sufficient. Looking at what we have done in the original spec, all we did was edit the BNF to remove things we didn't support, making what resulted easily comparable to standard XPath 2.0. The resultant BNF implies several things that we do not support in DFDL, so we qualified the BNF with some notes. For example, DFDL only allows the use of Predicates to index arrays, so there is a note saying that a Predicate must result in an integer else it's an SDE. I think we should do the same for errata 2.101 - leave the BNF alone and add a new note to the list. And beef up the words at the start of section 23.4 to make it clear how the BNF should be read. Also noticed that the BNF in the spec is not stand-alone as the constructs for StringLiteral etc are not reproduced, and rely on the reader reading XPath 2.0 spec. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 05/02/2013 23:08 Subject: [DFDL-WG] what do we allow in DFDL expressions after "/" Sent by: dfdl-wg-bounces(a)ogf.org In reexamining the grammar for the DFDL expression language, I have the following question In a path Step/Step/Step[Index] What can Step be? I know it can be ".." or parent::QName or child::QName or just an NCName or QName, Those are all what the XPath grammar calls AxisStep. anything else? The grammar can be changed in a very small way from the original XPath grammar if the above are the only possibilities. the clause StepExpr ::= FilterExpr | AxisStep can just be changed to StepExpr ::= AxisStep -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 200 - Terminology - Git for central repo of DFDL schemas at github (was Re: Git commit policies)
by Mike Beckerle 08 Feb '13

08 Feb '13

(re-send: neglected to send to whole workgroup mailing list) I'll send a couple emails on this subject. The initial big git hurdle is terminology being so different from other SCM. E.g., index - the set of files git is tracking. If you create a new file, by default git is not tracking it. Until you "add it to the index" it is very easy to lose it. sandbox - git doesn't use this term. The equivalent is a set of 4 things: * your local repository * commits - the deltas in your local repository: includes what you originally fetched/pulled plus any commits you have done to it * any files that have been "added to the index" but not committed yet. * any untracked files (that you haven't yet added to the index) - git requires you be careful about these. commit - a purely local operation of taking all files that have changes and are being tracked (i.e., new files have been added to the index), and creating a change-set out of them. Does NOT interact with a central repository at all. checkout - purely local operation which switches branches in your local repository - does NOT interact with a central anything. Simplest policy to stay out of trouble is to always add any new files to index and commit all outstanding changes before you do this. fetch - get changes from central repository, but don't combine/merget/change what is in current 'sandbox' to include them yet. Safe to do this operation whenever. pull - fetch plus merge the changes. We tend to avoid this and use fetch then rebase. merge - locally merge changes. We tend to avoid this and use rebase. rebase - a merge which replays your changes on top of another commit point (typically a master branch head), so as to make it look like all the changes were done in one linear sequence rather than by fork/join splitting and merging. This is often much easier to deal with than fork/join merging. push - moves changes to another repository. Doesn't merge anything. (Must already have been merged locally.) On Tue, Jan 29, 2013 at 10:46 AM, Tim Kimber <KIMBERT(a)uk.ibm.com> wrote: > > http://stackoverflow.com/questions/3505409/does-there-exist-git-commit-poli… > > http://git-scm.com/book/ch5-1.html > > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: kimbert(a)uk.ibm.com > Tel. 01962-816742 > Internal tel. 37246742 > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0