- dfdl-wg - lists.ogf.org

discriminators near PoU or not
by Mike Beckerle 19 Nov '12

19 Nov '12

I thought this situation was worth discussing. I have a bunch of messages. They all work like this: <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> There are hundreds of those. These all are used in this way: <element name="topLevel"> <complexType> <sequence> <element name="message" minOccurs="1" maxOccurs="unbounded"> <complexType> <choice> <element ref="messageName1"/> <element ref="messageName2"/> .... </choice> </complexType> </element> </sequence> </complexType> </element> So, there are two points of uncertainty here. The choice, and the unbounded array surrounding it. For each message, it is not until the sequence carrying the initiator, which is down inside the message structure, that we know for sure we've got this kind of message. So currently each message element is designed to be used ONLY in the above context of two surrounding points of uncertainty, like so (same message format as above but with TWO discriminators added). <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> <annotation><appinfo...>  <dfdl:discriminator>{ true() }</dfdl:discriminator>  <dfdl:discriminator>{ true() }</dfdl:discriminator> </appinfo></annotation> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> I'd love to have a better solution to this issue. But in the absence of one this works and achieves what I want which is that once it hits the initiator, we discriminate both which message we have, and we discriminate that we in fact have a message. Interesting that I'd really prefer to discriminate them in the opposite order of their nesting. I could discriminate the array the the very start of the message. But I have no way to do that because the discriminators apply in inward-out nesting order, and the nearest enclosing PoU is the choice. So I have to discriminate that one first, and I can't discriminate that one until I see the initiator, which is later into the message. Possible fixes/improvements: allow a label/id on each PoU, and allow referencing that label from a discriminator to say exactly which PoU you are discriminating. This makes the outward reference implied by a discriminator explicit. You still have the context issue, but it's not implied anymore, it is explicit. (This makes the problem like expressions with "../../.." that reach up and out of a construct. They are reaching up and out, but at least you can see directly that they are doing so.) -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

Re: [DFDL-WG] Fw: new action item needed: DFDL URN specification
by Suman Kalia 19 Nov '12

19 Nov '12

Steve - This seems fine and is in-line with the OGF registration document you attached.. This change would be happening in the next revision of spec, perhaps we can use 2012 or 2013 as the year in URN instead of 2011.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: dfdl-wg(a)ogf.org Date: 11/19/2012 01:27 PM Subject: Fw: [DFDL-WG] new action item needed: DFDL URN specification Suman - do you have a view on this, we would like to discuss on tomorrow's WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/11/2012 18:23 ----- From: Steve Hanson/UK/IBM To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 29/10/2012 15:31 Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Using the OGF's URN registration document [GFD.191] as a reference, I suggest: Type URL Proposed URN common root urn:ogf:dfdl xmlns attribute http://www.ogf.org/dfdl/dfdl-1.0/ urn:ogf:dfdl:2011 source attribute http://www.ogf.org/dfdl/ urn:ogf:dfdl [GFD.191] http://www.ogf.org/documents/GFD.191.pdf Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia <kalia(a)ca.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 25/10/2012 17:15 Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Sent by: dfdl-wg-bounces(a)ogf.org I tend to agree that specifying urn for namespaces is better choice and URL should be used for schema locations; however convention of specifying URL for namespaces is long established some implementations use the namespace URL to return the actual schema. When I try to access the namespace URL for XML schema , it gives me reference to the document but not schema.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 10/25/2012 08:18 AM Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Sent by: dfdl-wg-bounces(a)ogf.org I wanted to add to this discussion the w3c blog page that highlights the problem of the http-based naming scheme. There are many articles about this, this is just one of them. http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic/ On Wed, Oct 24, 2012 at 10:35 AM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: Introduction: Using URLs as identifiers has caused no end of problems. E.g., in DFDL we have http://www.ogf.org/dfdl/dfdl-1.0/ as an identifier. W3C has http://www.w3.org/2001/XMLSchema and others. W3C has badly regretted establishing this convention, as they have farms of servers that do nothing but quickly return 404 errors to save network-aware applications the network-timeout delay that would otherwise occur. There are parties interested in exploiting DFDL who want DFDL schemas to NOT contain network URLs because it simply creates a concern about network access whenever a DFDL schema is inspected/used. Pre-Proposal: The new way to do this is with URNs which would look roughly like this: urn:ogf:dfdl:standard:dfdl-1.0. The whole point is that some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. The point is that it is a name in a managed namespace which cannot be confused with a network protocol URL. OGF is already establishing urn:ogf, and an ogf subgroup has already proposed urn:ogf:network for network resources. DFDL schemas aren't network resources so we don't want to be a substructure underneath network. Some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. Summary: An action item should be to specify DFDL urn, submit to OGF as a proposed namespace, and then produce errata/spec changes to specify its use. This requires a small design activity to specify a scheme for the sub-structure of the DFDL URNs (i.e., scheme for the stuff after urn:ogf:dfdl:...) where we want standard identifiers for versions of the standard, but we probably also want a few other things (e.g., I would like a space for implementations to identify themselves, i.e., an implementation-specific sub-area within our URNs.) Our existing URLs can be compatible (deprecated) practice vs the preferred URNs. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: new action item needed: DFDL URN specification
by Steve Hanson 19 Nov '12

19 Nov '12

Suman - do you have a view on this, we would like to discuss on tomorrow's WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/11/2012 18:23 ----- From: Steve Hanson/UK/IBM To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 29/10/2012 15:31 Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Using the OGF's URN registration document [GFD.191] as a reference, I suggest: Type URL Proposed URN common root urn:ogf:dfdl xmlns attribute http://www.ogf.org/dfdl/dfdl-1.0/ urn:ogf:dfdl:2011 source attribute http://www.ogf.org/dfdl/ urn:ogf:dfdl [GFD.191] http://www.ogf.org/documents/GFD.191.pdf Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia <kalia(a)ca.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 25/10/2012 17:15 Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Sent by: dfdl-wg-bounces(a)ogf.org I tend to agree that specifying urn for namespaces is better choice and URL should be used for schema locations; however convention of specifying URL for namespaces is long established some implementations use the namespace URL to return the actual schema. When I try to access the namespace URL for XML schema , it gives me reference to the document but not schema.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 10/25/2012 08:18 AM Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Sent by: dfdl-wg-bounces(a)ogf.org I wanted to add to this discussion the w3c blog page that highlights the problem of the http-based naming scheme. There are many articles about this, this is just one of them. http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic/ On Wed, Oct 24, 2012 at 10:35 AM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: Introduction: Using URLs as identifiers has caused no end of problems. E.g., in DFDL we have http://www.ogf.org/dfdl/dfdl-1.0/ as an identifier. W3C has http://www.w3.org/2001/XMLSchema and others. W3C has badly regretted establishing this convention, as they have farms of servers that do nothing but quickly return 404 errors to save network-aware applications the network-timeout delay that would otherwise occur. There are parties interested in exploiting DFDL who want DFDL schemas to NOT contain network URLs because it simply creates a concern about network access whenever a DFDL schema is inspected/used. Pre-Proposal: The new way to do this is with URNs which would look roughly like this: urn:ogf:dfdl:standard:dfdl-1.0. The whole point is that some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. The point is that it is a name in a managed namespace which cannot be confused with a network protocol URL. OGF is already establishing urn:ogf, and an ogf subgroup has already proposed urn:ogf:network for network resources. DFDL schemas aren't network resources so we don't want to be a substructure underneath network. Some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. Summary: An action item should be to specify DFDL urn, submit to OGF as a proposed namespace, and then produce errata/spec changes to specify its use. This requires a small design activity to specify a scheme for the sub-structure of the DFDL URNs (i.e., scheme for the stuff after urn:ogf:dfdl:...) where we want standard identifiers for versions of the standard, but we probably also want a few other things (e.g., I would like a space for implementations to identify themselves, i.e., an implementation-specific sub-area within our URNs.) Our existing URLs can be compatible (deprecated) practice vs the preferred URNs. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: setVariable with no value set - supported?
by Steve Hanson 19 Nov '12

19 Nov '12

See below... Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/11/2012 14:23 ----- From: Steve Hanson/UK/IBM To: Mark Frost/UK/IBM, Cc: Shane Cartledge/Toronto/IBM@IBMCA, Anita Leung/Toronto/IBM@IBMCA Date: 16/11/2012 11:46 Subject: Re: setVariable with no value set - supported? The specific example that Shane gives is invalid anyway as you can't set the value of one of the pre-defined DFDL variables. Assuming a realistic example like: <dfdl:setVariable ref="myVar"/> I'm wondering how you would set empty string for an element of type xs:string or xs:hexBinary. <dfdl:setVariable ref="myVar" value=""/> But why isn't that the same as <dfdl:setVariable ref="myVar"></dfdl:setVariable> which in XML rules is equivalent to: <dfdl:setVariable ref="myVar"/> I will raise with the DFDL WG. Thanks Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mark Frost/UK/IBM To: Shane Cartledge/Toronto/IBM@IBMCA, Steve Hanson/UK/IBM@IBMGB, Cc: Anita Leung/Toronto/IBM@IBMCA Date: 16/11/2012 11:19 Subject: Re: setVariable with no value set - supported? Hi Shane, While the spec isn't super-specific, I'd interpret it to mean "you must give exactly one of the value attribute or element value" Spec Section 7.9 : "The syntax supports both a dfdl:value attribute and the 'value' being specified by the element value. Only one or the other may be present. (Schema definition error otherwise.)" Steve: do you think altering the spec wording above to something like " Exactly one of the dfdl:value or the element value must be present. (Schema definition error otherwise.)" Regards, Mark Frost _____________________________________ MP 211, IBM Hursley, Winchester, SO21 2JN Phone: (01962) 817009 or x247009 Email: frostmar(a)uk.ibm.com From: Shane Cartledge/Toronto/IBM@IBMCA To: Mark Frost/UK/IBM@IBMGB, Cc: Anita Leung/Toronto/IBM@IBMCA Date: 15/11/2012 22:56 Subject: setVariable with no value set - supported? Hi Mark, Do you know if DFDL should supports a setVariable with no value set? Thought this may have already come up with your runtime testing. For example: <dfdl:setVariable ref="dfdl:binaryFloatRep"/> The DFDL specification is not clear on this point, and the DFDL validator does not flag it as an error. Thanks. Shane Cartledge Business Integration Tooling Phone: 1-905-413-5171, Tie Line: 313-5171 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Action 183 - chicken-and-egg situation with lengths given by expressions
by Steve Hanson 16 Nov '12

16 Nov '12

The email thread which provided the material for the discussion which led to: 2.100. Section 12.3.1. State that when unparsing an element with lengthKind ‘explicit’ and where length is an expression, then the data in the Infoset is treated as variable length and not fixed length. The behaviour is the same as lengthKind ‘prefixed’. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/11/2012 16:04 ----- From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/09/2012 12:39 Subject: Re: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Sent by: dfdl-wg-bounces(a)ogf.org Good point. The problem is that lengthKind-'explicit' is being used for two things: a) a length that is static b) a length that is calculated ...so the DFDL serializer must assume that the expression needs to be evaluated. For occursCountKind we have separate values for 'fixed' and 'expression'. If we did not, then occursCountKind would have the same problem except that it would affect defaulting rather than padding. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/09/2012 12:27 Subject: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Sent by: dfdl-wg-bounces(a)ogf.org This mail is on the expected behaviour of the DFDL unparser when writing out a 'data' element the length of which is held in an earlier 'len' element. There are several scenarios, some straightforward and some that exhibit a chicken-and-egg behaviour. The principle of what happens is understood, the action is to make sure that the behaviour is explained in enough detail in the spec to enable implementations to be consistent. (Note - IBM DFDL does not yet support outputVaueCalc so has not hit this yet). Scenarios follow. The 'data' element shown is simple, but the same principles apply if it is complex. 1) 'len' is set from infoset - 'len' can be set in augmented infoset - No issue as 'data's length expression may be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 2) 'len' is set using outputValueCalc with fixed expression - When 'len's outputValueCalc is encountered, it can be evaluated then and there - 'len' can be set in augmented infoset - No issue as 'data's length expression may be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{10}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 3) 'len' is set using outputValueCalc with reference 'data' (unpadded) - When 'len's outputValueCalc is encountered, it can not yet be evaluated as it depends on the length of 'data' - 'len' can not yet be set in augmented infoset - Problem as 'data's length expression can not be evaluated - But we do know the unpadded length of 'data' so 'len's outputValueCalc can now be evaluated - In turn this means that 'data's length expression can now be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{dfdl:unpaddedLength(/message1/data)}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 4) 'len' is set using outputValueCalc with reference 'data' (padded) - When 'len's outputValueCalc is encountered, it can not yet be evaluated as it depends on the length of 'data' - 'len' can not yet be set in augmented infoset - Problem as 'data's length expression can not be evaluated - We don't know the padded length of 'data' because we don't know 'len' - Problem: 'data's length expression can never be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{dfdl:representationLength(/message1/data)}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 04/09/2012 17:31 Subject: Fw: Behaviour for lengthKind 'endOfparent' is still not fully specified DFDL WG call 4th Sept 2012: 1) Agreed that for binary data, only xs:hexBinary and packed/BCD allowed to have endOfParent 2) Agreed this is the correct behaviour when filling to a known length 3) Agreed this is the correct behaviour when filling to a known length 4) Agreed this is the correct behaviour when filling to a known length It was noted that lengthKind 'explicit' on the parent may not result in a known length if the length is an expression. This is an example of a more general chicken-and-egg situation with lengths given by expressions, for which outputValueCalc and DFDL functions unpaddedLength() were added can be used. Action raised to ensure that the behaviour of an implementation is fully defined by the spec. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 04/09/2012 17:24 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org <dfdl-wg(a)ogf.org> Date: 04/09/2012 14:08 Subject: Behaviour for lengthKind 'endOfparent' is still not fully specified Noted when I reviewed latest spec - endOfParent and unparsing is not fully thought through. The spec today says that I can use endOfParent with binary data. There is a restriction in section 12.3.8, but it only applies when an element is endOfParent and its parent is lengthKind delimited. There are a couple of cases to consider: 1) Binary data of restricted length (see list in other email "proposed clarification/narrowing - delimited binary data should decimal"). I don't think it makes sense to allow these. We don't allow these binary reps for delimited. 2) Text data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should pad to box length with appropriate padChar, according to justification, as that is effectively a 'specified length'. Error if textPadKind is 'none'. Use parent's lengthUnits. 3) HexBinary data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should right-pad to box length with fill byte, as that is effectively a 'specified length'. 4) Packed/BCD binary data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should pad to box length with zero bytes, according to justification, as that is effectively a 'specified length'. (Must be zero bytes and not fill byte as must be numeric in order to be parsed). In relation to 2 - 4, note that lengthKind 'endOfParent' can only be used with a parent lengthKind of 'explicit', 'pattern', 'prefixed' or 'endOfParent' or a choice with choiceLengthKind 'explicit', so the box scenario when unparsing therefore occurs only when lengthKind is 'explicit' or choiceLengthKind is 'explicit' - these are the cases when the length is known. Also note that when there are nested 'endOfParent' elements (which is allowed) then all padding must be done on the simple element (ie, the innermost element), to ensure that what is output can be parsed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Clarification needed: regular expressions - does '.' match newlines by default?
by Mike Beckerle 16 Nov '12

16 Nov '12

A key behavior distinction in regular expressions is whether the '.' wildcard matches line endings or not. Regular expression libraries can be configured, usually by some sort of expression modifier, either way so that the '.' will not match a line ending or so that it will. Question is, how is it configured by default in DFDL regular expressions? This is part of the overall issue of tightening up regular expressions as part of DFDL. I.e., what exactly is the regex dialect, and how is it configured by default. ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

4 5

AUTO: Sunil Dandamudi is out of the office (returning 11/16/2012)
by Sunil Dandamudi 15 Nov '12

15 Nov '12

I am out of the office until 11/16/2012. Note: This is an automated response to your message "dfdl-wg Digest, Vol 77, Issue 19" sent on 11/15/2012 10:00:01. This is the only notification you will receive while this person is away.

1 0

OGF DFDL WG Call Minutes 2012-11-13
by Steve Hanson 14 Nov '12

14 Nov '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12965?download= Errata document is still at v010 here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: clarification needed: assert evaluation order and arrays
by Steve Hanson 14 Nov '12

14 Nov '12

Agreed on DFDL WG call that checkConstraints() will only look at facets and the fixed attribute. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 14/11/2012 11:04 ----- From: Steve Hanson/UK/IBM To: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com>, Cc: Tim Kimber/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 02/11/2012 14:51 Subject: Fw: [DFDL-WG] clarification needed: assert evaluation order and arrays Hi Mike I have checked with our resident XPath expert, and you are correct. The self-axis refers to the current occurrence. Sorry to mislead you. I think we need a short discussion about what checkConstraints() can and can't check against. I'll add to agenda for next call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/11/2012 09:19 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 01/11/2012 18:34 Subject: Re: [DFDL-WG] clarification needed: assert evaluation order and arrays Do I understand this right? I have always thought that "." refers to the current occurrence (i.e., the [dfdl:occursIndex()] is implied if the current elemetn is an array). Hence, if I have an array of a complex type with children named a and b, then "./a" is a valid expression referring to the current instance's child a. Similarly, if I type "../bar", and my parent is an array, then ".." moves me up to the current instance within that array. Upward paths are unambiguous. Current node paths are unambiguous. Only downward paths that encounter an array would need indexing brackets. I think lots of things break if the above isn't the rule. E.g., you can't take an element and add minOccurs/maxOccurs to it to make it repeat without all the paths written on it being broken. If I am right, then there is no problem with making dfdl:checkConstraints(.) take the node, ask if it is an array, and if so compare its occursIndex to maxOccurs. I do take your point that the secondary check for minOccurs has to be expressed differently with a separate assert. The fact that dfdl:occursCount(path) takes a node-set-valued path is .... well really annoying, but inevitable. We need to watch for these exceptions. On Thu, Nov 1, 2012 at 1:39 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: You can't do that with one assert I'm afraid. The main issue is that a path location must always return exactly one item, so it's use within the context of an array must be in conjunction with a predicate, which must refer to a single instance. That's what dfdl:occursIndex() is for. To check that an array meets its bounds, you need a separate assert that uses the dfdl:count() function. You'd have to hard-code the min/max values, and place it on a containing object. <sequence> <annotation><appinfo...> <dfdl:assert>{ dfdl:occursCount(./foo) ge 5 and dfdl:occursCount(./foo) le 10 }</dfdl:assert> </appinfo></annotation> <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.[dfdl:occursIndex()]) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> </sequence> I think that means our definition of dfdl:checkConstraints() in the spec is wrong. Table 34 and Section 5.2 together imply that minOccurs and maxOccurs are used in checkConstraints. I think that simply doesn't work. It should only be using 'fixed' and the facets (I think 'default' is pointless as well). Worth noting that when a path is used in the context dfdl:occursCount(<path>) then it is not an error if more than one item is returned, and it should not be wrapped with fn:exactly-one(). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 01/11/2012 16:45 Subject: [DFDL-WG] clarification needed: assert evaluation order and arrays Sent by: dfdl-wg-bounces(a)ogf.org Question: If I write <element name="foo" minOccurs="5" maxOccurs="10" dfdl:occursCountKind='parsed'> <annotation><appinfo...> <dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert> </appinfo></annotation> <simpleType> <restriction base="xs:string"> <pattern value="...some regex..."/> </restriction> </simpleType> </element> I have two sources of constraints. One is the pattern, the other the min/max occurs. Does that one assertion calling dfdl:checkConstraints mean both will be checked? That is, one check occurring as each element is parsed, and the other at the end of the array? Assume I am not using any validation option, so the DFDL processor would not otherwise check the max/minOccurs because occursCountKind is parsed. Will the checkConstraints fail as soon as we parse the 11th element (is it checking the min/max occurs for each element occurrence as it is parsed), or do we parse as many as we can, and fail only when we check and find out that the entire array has 36 elements that were successfully parsed? What I would like the above to mean is this: 1) as each element occurrence is parsed we check the pattern and parse error (assertion failed) if there is no match. 2) Also after a successful parse of an occurrence, we check that the index is <=10, and parse-error (assertion failed) if not. 3) at the end of the array, we check that the number of occurrences is >= 5. If not we get a parse error (assertion failed). Comments? ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Expressions too restrictive? Example from IPFIX format
by Mike Beckerle 13 Nov '12

13 Nov '12

I'm looking at RFC 5101 and RFC 5102 These describe a dense binary file format for observing network flows. The application is related to network security. This format has a 'meta' structure to it that I'm not sure how to deal with in DFDL currently. Here's the problem as succinctly as I can make it. One configures some network information capture tools to capture some information. This is flexible, so what is captured can be quite variable. The resulting data stream contains first sets of templates, then sets of data records, then more templates, then more records, and so on. The templates each are identified by an ID number which is an integer from 256 to 65534. A template then contains a count of how many fields are in the template, and then a field descriptor for each field, which includes the length of the field in octets. Each data record set begins with the ID of its template, then a total length in octets for the set, then the records, which are just field, field, field, each as described by the associated template with no record separators or anything. Now, I can model a template in DFDL as an array of field descriptors. I can also model a data record set as a templateID and an array of data records, and each data record as an array of fields, where the number of field occurrences is given in the template, and each field is a byte array of occurs count given in the corresponding template's field descriptor for that occursIndex. The one thing I can't figure out how to do is to create an XPath to the right template given the template ID in the data record set header. The problem, is that the TemplateID is not an array index. The templates might have arbitrary ID numbers. They might not be say, 256, 257, etc. in order. They could be scattered, etc. The standard only requires that they are unique IDs. So the set of templates is truly a set with these identifiers. So I need a way to write an XPath that would choose the template from the set, whose ID matches a particular integer value from the data record set's header. Right now I think our XPath subset doesn't allow this. We can only index arrays with integers, and we have no searching capability that processes a set of nodes to identify a node having any particular value or characteristic. So: are we being too strict here. Should we allow somewhat more complex predicates? such as { ...../template[idField eq $templateID] } (For speed reasons, I might not do that lookup each time I parse a data record. I might pre-fill an array of 65535 structures by way of inputValueCalc so that I can actually use the template IDs as indexes into that array. But either way I need the select from matching set capability.) In general, templates can be interleaved in IPFix in that the requirement is just that they are transmitted before data record sets that reference them. So in general, an application that reads IPFix data cannot say, first scrape off all the templates and generate a schema for them, and then use that schema to parse. The late availability of the templates is something that is inherent in the format. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1