- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Issue 140 spinoff - proposed revised grammar
by Mike Beckerle 19 Nov '12

19 Nov '12

Attached revised grammar 005 - proposed as "to become an errata". On Mon, Jul 30, 2012 at 6:29 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > A couple of further revisions. > > > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Steve Hanson/UK/IBM@IBMGB > Cc: Tim Kimber/UK/IBM@IBMGB > Date: 25/07/2012 16:49 > Subject: Re: Issue 140 spinoff - proposed revised grammar > ------------------------------ > > > > Revised per call today. For your consideration. > > ...mikeb > > On Wed, Jul 11, 2012 at 7:39 AM, Steve Hanson <*smh(a)uk.ibm.com*<smh(a)uk.ibm.com>> > wrote: > I've made some updates which I think solves the complex empty issue, ties > the grammar more closely to the action 140 rep definitions, and addresses > Tim's content v value distinction. > > Let me know what you think. > > > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:*+44-1962-815848* <%2B44-1962-815848> > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com> > > > To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB > Date: 10/07/2012 16:25 > Subject: Re: Issue 140 spinoff - proposed revised grammar > ------------------------------ > > > > > Accept-all-changes on that document before you try to read it. Or use the > attached. > > On Tue, Jul 10, 2012 at 11:19 AM, Mike Beckerle <*mbeckerle.dfdl(a)gmail.com > * <mbeckerle.dfdl(a)gmail.com>> wrote: > I took a crack at revising the grammar to something more accurate, more > useful. It's bigger, but not 3 times as big. > > I like it, but then I'm an implementor, not a user of this spec trying to > devise a schema. Our audience is both however. > > I am curious what you think. > > One thing it cannot capture is the complexity of complex empty's, i.e., > the fact that a complex empty requires one to sometimes descend recursively > first, and then check for empty. From the grammar you would think you can > always check for this first. > > Fact is, you have to descend recursively in order to pop back up, and have > the representation be found to match what I call EmptyElement. But the > grammar can't show that. (well, maybe it could... I'll think about that > further.) > > ...mike > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: *781-330-0412* <781-330-0412> > > > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: *781-330-0412* <781-330-0412> > [attachment "grammar-revision.docx" deleted by Steve Hanson/UK/IBM] > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > [attachment "grammar-revision-003.doc" deleted by Steve Hanson/UK/IBM] > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 5

Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger
by Steve Hanson 19 Nov '12

19 Nov '12

Agreed on DFDL WG call 4th Sept to continue to support twos-complement decimal, integer or nonNegativeInteger, but to make the maximum implementation defined. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 15/08/2012 13:10 Subject: Re: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Mike Having thought about this since our call yesterday, I am uncomfortable with the proposed narrowing. It means DFDL can not be guaranteed to handle the two's complement result of Java BigInteger.toByteArray(). I don't have a concrete use case of two's complement data > 8 bytes but I am sure that someone in the scientific community will need it at some point. Hence why it is allowed in the spec today. If we do allow it, then we have to decide what happens during unparsing and lengthKind is 'prefixed'. The equivalent of Table 16 for lengthKind 'pattern' is needed for lengthKind 'prefixed', as we discussed. That would imply that the unparsing behaviour would be 'Minimum number of bytes to represent significant digits and sign'. In other words, a two's complement binary integer is created that is just long enough to represent the infoset value. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org Date: 14/08/2012 19:33 Subject: [DFDL-WG] proposed clarification/narrowing - no twos-complement decimal, integer or nonNegativeInteger Sent by: dfdl-wg-bounces(a)ogf.org The spec is not clear on whether decimal, integer, and nonNegativeInteger support representation='binary' with binaryNumberRep='binary' (aka twos-complement). Since these types do not have any size limit, such a capability allows arbitrarily large twos-complement integers, or perhaps it would be an implementation defined limit. I believe the right choice is that it should be disallowed. That is, only binaryNumberRep of 'bcd' or 'packed' should be allowed for these types. These types have always been about decimal-style representations to me. This keeps twos-complement only for the types long, int, short, byte and their unsigned counterparts. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Agenda 2012-11-20
by Steve Hanson 19 Nov '12

19 Nov '12

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12967?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

discriminators near PoU or not
by Mike Beckerle 19 Nov '12

19 Nov '12

I thought this situation was worth discussing. I have a bunch of messages. They all work like this: <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> There are hundreds of those. These all are used in this way: <element name="topLevel"> <complexType> <sequence> <element name="message" minOccurs="1" maxOccurs="unbounded"> <complexType> <choice> <element ref="messageName1"/> <element ref="messageName2"/> .... </choice> </complexType> </element> </sequence> </complexType> </element> So, there are two points of uncertainty here. The choice, and the unbounded array surrounding it. For each message, it is not until the sequence carrying the initiator, which is down inside the message structure, that we know for sure we've got this kind of message. So currently each message element is designed to be used ONLY in the above context of two surrounding points of uncertainty, like so (same message format as above but with TWO discriminators added). <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> <annotation><appinfo...>  <dfdl:discriminator>{ true() }</dfdl:discriminator>  <dfdl:discriminator>{ true() }</dfdl:discriminator> </appinfo></annotation> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> I'd love to have a better solution to this issue. But in the absence of one this works and achieves what I want which is that once it hits the initiator, we discriminate both which message we have, and we discriminate that we in fact have a message. Interesting that I'd really prefer to discriminate them in the opposite order of their nesting. I could discriminate the array the the very start of the message. But I have no way to do that because the discriminators apply in inward-out nesting order, and the nearest enclosing PoU is the choice. So I have to discriminate that one first, and I can't discriminate that one until I see the initiator, which is later into the message. Possible fixes/improvements: allow a label/id on each PoU, and allow referencing that label from a discriminator to say exactly which PoU you are discriminating. This makes the outward reference implied by a discriminator explicit. You still have the context issue, but it's not implied anymore, it is explicit. (This makes the problem like expressions with "../../.." that reach up and out of a construct. They are reaching up and out, but at least you can see directly that they are doing so.) -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

Re: [DFDL-WG] Fw: new action item needed: DFDL URN specification
by Suman Kalia 19 Nov '12

19 Nov '12

Steve - This seems fine and is in-line with the OGF registration document you attached.. This change would be happening in the next revision of spec, perhaps we can use 2012 or 2013 as the year in URN instead of 2011.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Suman Kalia/Toronto/IBM@IBMCA, Cc: dfdl-wg(a)ogf.org Date: 11/19/2012 01:27 PM Subject: Fw: [DFDL-WG] new action item needed: DFDL URN specification Suman - do you have a view on this, we would like to discuss on tomorrow's WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/11/2012 18:23 ----- From: Steve Hanson/UK/IBM To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 29/10/2012 15:31 Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Using the OGF's URN registration document [GFD.191] as a reference, I suggest: Type URL Proposed URN common root urn:ogf:dfdl xmlns attribute http://www.ogf.org/dfdl/dfdl-1.0/ urn:ogf:dfdl:2011 source attribute http://www.ogf.org/dfdl/ urn:ogf:dfdl [GFD.191] http://www.ogf.org/documents/GFD.191.pdf Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia <kalia(a)ca.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 25/10/2012 17:15 Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Sent by: dfdl-wg-bounces(a)ogf.org I tend to agree that specifying urn for namespaces is better choice and URL should be used for schema locations; however convention of specifying URL for namespaces is long established some implementations use the namespace URL to return the actual schema. When I try to access the namespace URL for XML schema , it gives me reference to the document but not schema.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 10/25/2012 08:18 AM Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Sent by: dfdl-wg-bounces(a)ogf.org I wanted to add to this discussion the w3c blog page that highlights the problem of the http-based naming scheme. There are many articles about this, this is just one of them. http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic/ On Wed, Oct 24, 2012 at 10:35 AM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: Introduction: Using URLs as identifiers has caused no end of problems. E.g., in DFDL we have http://www.ogf.org/dfdl/dfdl-1.0/ as an identifier. W3C has http://www.w3.org/2001/XMLSchema and others. W3C has badly regretted establishing this convention, as they have farms of servers that do nothing but quickly return 404 errors to save network-aware applications the network-timeout delay that would otherwise occur. There are parties interested in exploiting DFDL who want DFDL schemas to NOT contain network URLs because it simply creates a concern about network access whenever a DFDL schema is inspected/used. Pre-Proposal: The new way to do this is with URNs which would look roughly like this: urn:ogf:dfdl:standard:dfdl-1.0. The whole point is that some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. The point is that it is a name in a managed namespace which cannot be confused with a network protocol URL. OGF is already establishing urn:ogf, and an ogf subgroup has already proposed urn:ogf:network for network resources. DFDL schemas aren't network resources so we don't want to be a substructure underneath network. Some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. Summary: An action item should be to specify DFDL urn, submit to OGF as a proposed namespace, and then produce errata/spec changes to specify its use. This requires a small design activity to specify a scheme for the sub-structure of the DFDL URNs (i.e., scheme for the stuff after urn:ogf:dfdl:...) where we want standard identifiers for versions of the standard, but we probably also want a few other things (e.g., I would like a space for implementations to identify themselves, i.e., an implementation-specific sub-area within our URNs.) Our existing URLs can be compatible (deprecated) practice vs the preferred URNs. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: new action item needed: DFDL URN specification
by Steve Hanson 19 Nov '12

19 Nov '12

Suman - do you have a view on this, we would like to discuss on tomorrow's WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/11/2012 18:23 ----- From: Steve Hanson/UK/IBM To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 29/10/2012 15:31 Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Using the OGF's URN registration document [GFD.191] as a reference, I suggest: Type URL Proposed URN common root urn:ogf:dfdl xmlns attribute http://www.ogf.org/dfdl/dfdl-1.0/ urn:ogf:dfdl:2011 source attribute http://www.ogf.org/dfdl/ urn:ogf:dfdl [GFD.191] http://www.ogf.org/documents/GFD.191.pdf Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia <kalia(a)ca.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 25/10/2012 17:15 Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Sent by: dfdl-wg-bounces(a)ogf.org I tend to agree that specifying urn for namespaces is better choice and URL should be used for schema locations; however convention of specifying URL for namespaces is long established some implementations use the namespace URL to return the actual schema. When I try to access the namespace URL for XML schema , it gives me reference to the document but not schema.. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 10/25/2012 08:18 AM Subject: Re: [DFDL-WG] new action item needed: DFDL URN specification Sent by: dfdl-wg-bounces(a)ogf.org I wanted to add to this discussion the w3c blog page that highlights the problem of the http-based naming scheme. There are many articles about this, this is just one of them. http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic/ On Wed, Oct 24, 2012 at 10:35 AM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: Introduction: Using URLs as identifiers has caused no end of problems. E.g., in DFDL we have http://www.ogf.org/dfdl/dfdl-1.0/ as an identifier. W3C has http://www.w3.org/2001/XMLSchema and others. W3C has badly regretted establishing this convention, as they have farms of servers that do nothing but quickly return 404 errors to save network-aware applications the network-timeout delay that would otherwise occur. There are parties interested in exploiting DFDL who want DFDL schemas to NOT contain network URLs because it simply creates a concern about network access whenever a DFDL schema is inspected/used. Pre-Proposal: The new way to do this is with URNs which would look roughly like this: urn:ogf:dfdl:standard:dfdl-1.0. The whole point is that some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. The point is that it is a name in a managed namespace which cannot be confused with a network protocol URL. OGF is already establishing urn:ogf, and an ogf subgroup has already proposed urn:ogf:network for network resources. DFDL schemas aren't network resources so we don't want to be a substructure underneath network. Some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. Summary: An action item should be to specify DFDL urn, submit to OGF as a proposed namespace, and then produce errata/spec changes to specify its use. This requires a small design activity to specify a scheme for the sub-structure of the DFDL URNs (i.e., scheme for the stuff after urn:ogf:dfdl:...) where we want standard identifiers for versions of the standard, but we probably also want a few other things (e.g., I would like a space for implementations to identify themselves, i.e., an implementation-specific sub-area within our URNs.) Our existing URLs can be compatible (deprecated) practice vs the preferred URNs. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: setVariable with no value set - supported?
by Steve Hanson 19 Nov '12

19 Nov '12

See below... Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/11/2012 14:23 ----- From: Steve Hanson/UK/IBM To: Mark Frost/UK/IBM, Cc: Shane Cartledge/Toronto/IBM@IBMCA, Anita Leung/Toronto/IBM@IBMCA Date: 16/11/2012 11:46 Subject: Re: setVariable with no value set - supported? The specific example that Shane gives is invalid anyway as you can't set the value of one of the pre-defined DFDL variables. Assuming a realistic example like: <dfdl:setVariable ref="myVar"/> I'm wondering how you would set empty string for an element of type xs:string or xs:hexBinary. <dfdl:setVariable ref="myVar" value=""/> But why isn't that the same as <dfdl:setVariable ref="myVar"></dfdl:setVariable> which in XML rules is equivalent to: <dfdl:setVariable ref="myVar"/> I will raise with the DFDL WG. Thanks Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mark Frost/UK/IBM To: Shane Cartledge/Toronto/IBM@IBMCA, Steve Hanson/UK/IBM@IBMGB, Cc: Anita Leung/Toronto/IBM@IBMCA Date: 16/11/2012 11:19 Subject: Re: setVariable with no value set - supported? Hi Shane, While the spec isn't super-specific, I'd interpret it to mean "you must give exactly one of the value attribute or element value" Spec Section 7.9 : "The syntax supports both a dfdl:value attribute and the 'value' being specified by the element value. Only one or the other may be present. (Schema definition error otherwise.)" Steve: do you think altering the spec wording above to something like " Exactly one of the dfdl:value or the element value must be present. (Schema definition error otherwise.)" Regards, Mark Frost _____________________________________ MP 211, IBM Hursley, Winchester, SO21 2JN Phone: (01962) 817009 or x247009 Email: frostmar(a)uk.ibm.com From: Shane Cartledge/Toronto/IBM@IBMCA To: Mark Frost/UK/IBM@IBMGB, Cc: Anita Leung/Toronto/IBM@IBMCA Date: 15/11/2012 22:56 Subject: setVariable with no value set - supported? Hi Mark, Do you know if DFDL should supports a setVariable with no value set? Thought this may have already come up with your runtime testing. For example: <dfdl:setVariable ref="dfdl:binaryFloatRep"/> The DFDL specification is not clear on this point, and the DFDL validator does not flag it as an error. Thanks. Shane Cartledge Business Integration Tooling Phone: 1-905-413-5171, Tie Line: 313-5171 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Action 183 - chicken-and-egg situation with lengths given by expressions
by Steve Hanson 16 Nov '12

16 Nov '12

The email thread which provided the material for the discussion which led to: 2.100. Section 12.3.1. State that when unparsing an element with lengthKind ‘explicit’ and where length is an expression, then the data in the Infoset is treated as variable length and not fixed length. The behaviour is the same as lengthKind ‘prefixed’. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/11/2012 16:04 ----- From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/09/2012 12:39 Subject: Re: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Sent by: dfdl-wg-bounces(a)ogf.org Good point. The problem is that lengthKind-'explicit' is being used for two things: a) a length that is static b) a length that is calculated ...so the DFDL serializer must assume that the expression needs to be evaluated. For occursCountKind we have separate values for 'fixed' and 'expression'. If we did not, then occursCountKind would have the same problem except that it would affect defaulting rather than padding. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/09/2012 12:27 Subject: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Sent by: dfdl-wg-bounces(a)ogf.org This mail is on the expected behaviour of the DFDL unparser when writing out a 'data' element the length of which is held in an earlier 'len' element. There are several scenarios, some straightforward and some that exhibit a chicken-and-egg behaviour. The principle of what happens is understood, the action is to make sure that the behaviour is explained in enough detail in the spec to enable implementations to be consistent. (Note - IBM DFDL does not yet support outputVaueCalc so has not hit this yet). Scenarios follow. The 'data' element shown is simple, but the same principles apply if it is complex. 1) 'len' is set from infoset - 'len' can be set in augmented infoset - No issue as 'data's length expression may be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 2) 'len' is set using outputValueCalc with fixed expression - When 'len's outputValueCalc is encountered, it can be evaluated then and there - 'len' can be set in augmented infoset - No issue as 'data's length expression may be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{10}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 3) 'len' is set using outputValueCalc with reference 'data' (unpadded) - When 'len's outputValueCalc is encountered, it can not yet be evaluated as it depends on the length of 'data' - 'len' can not yet be set in augmented infoset - Problem as 'data's length expression can not be evaluated - But we do know the unpadded length of 'data' so 'len's outputValueCalc can now be evaluated - In turn this means that 'data's length expression can now be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{dfdl:unpaddedLength(/message1/data)}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 4) 'len' is set using outputValueCalc with reference 'data' (padded) - When 'len's outputValueCalc is encountered, it can not yet be evaluated as it depends on the length of 'data' - 'len' can not yet be set in augmented infoset - Problem as 'data's length expression can not be evaluated - We don't know the padded length of 'data' because we don't know 'len' - Problem: 'data's length expression can never be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{dfdl:representationLength(/message1/data)}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 04/09/2012 17:31 Subject: Fw: Behaviour for lengthKind 'endOfparent' is still not fully specified DFDL WG call 4th Sept 2012: 1) Agreed that for binary data, only xs:hexBinary and packed/BCD allowed to have endOfParent 2) Agreed this is the correct behaviour when filling to a known length 3) Agreed this is the correct behaviour when filling to a known length 4) Agreed this is the correct behaviour when filling to a known length It was noted that lengthKind 'explicit' on the parent may not result in a known length if the length is an expression. This is an example of a more general chicken-and-egg situation with lengths given by expressions, for which outputValueCalc and DFDL functions unpaddedLength() were added can be used. Action raised to ensure that the behaviour of an implementation is fully defined by the spec. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 04/09/2012 17:24 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org <dfdl-wg(a)ogf.org> Date: 04/09/2012 14:08 Subject: Behaviour for lengthKind 'endOfparent' is still not fully specified Noted when I reviewed latest spec - endOfParent and unparsing is not fully thought through. The spec today says that I can use endOfParent with binary data. There is a restriction in section 12.3.8, but it only applies when an element is endOfParent and its parent is lengthKind delimited. There are a couple of cases to consider: 1) Binary data of restricted length (see list in other email "proposed clarification/narrowing - delimited binary data should decimal"). I don't think it makes sense to allow these. We don't allow these binary reps for delimited. 2) Text data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should pad to box length with appropriate padChar, according to justification, as that is effectively a 'specified length'. Error if textPadKind is 'none'. Use parent's lengthUnits. 3) HexBinary data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should right-pad to box length with fill byte, as that is effectively a 'specified length'. 4) Packed/BCD binary data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should pad to box length with zero bytes, according to justification, as that is effectively a 'specified length'. (Must be zero bytes and not fill byte as must be numeric in order to be parsed). In relation to 2 - 4, note that lengthKind 'endOfParent' can only be used with a parent lengthKind of 'explicit', 'pattern', 'prefixed' or 'endOfParent' or a choice with choiceLengthKind 'explicit', so the box scenario when unparsing therefore occurs only when lengthKind is 'explicit' or choiceLengthKind is 'explicit' - these are the cases when the length is known. Also note that when there are nested 'endOfParent' elements (which is allowed) then all padding must be done on the simple element (ie, the innermost element), to ensure that what is output can be parsed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Clarification needed: regular expressions - does '.' match newlines by default?
by Mike Beckerle 16 Nov '12

16 Nov '12

A key behavior distinction in regular expressions is whether the '.' wildcard matches line endings or not. Regular expression libraries can be configured, usually by some sort of expression modifier, either way so that the '.' will not match a line ending or so that it will. Question is, how is it configured by default in DFDL regular expressions? This is part of the overall issue of tightening up regular expressions as part of DFDL. I.e., what exactly is the regex dialect, and how is it configured by default. ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

4 5

AUTO: Sunil Dandamudi is out of the office (returning 11/16/2012)
by Sunil Dandamudi 15 Nov '12

15 Nov '12

I am out of the office until 11/16/2012. Note: This is an automated response to your message "dfdl-wg Digest, Vol 77, Issue 19" sent on 11/15/2012 10:00:01. This is the only notification you will receive while this person is away.

1 0