- dfdl-wg - lists.ogf.org

issue 145 - name dfdl:elementAlias
by Mike Beckerle 05 Apr '12

05 Apr '12

About the name dfdl:elementAlias It suggests that this is another element name, and in other contexts than a choice it might be used as well, and in addition it suggests that the dfdl:choiceBranchRef can evaluate to the element name, or to this alias. I think the definition of all this behavior should be narrower, and the names should suggest this. dfdl:choiceBranchRef dfdl:choiceBranchID seem like the right pairing. A choice branch ref can refer to exactly and only the choice branch ID, never the element name, and this ID is used only for choice branch ref purposes. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

Issue 145 - cobol 88 example with multiple definitions per 88
by Mike Beckerle 05 Apr '12

05 Apr '12

Cobol below illustrates how the 88 statement can define more than one constant to represent a selection. E.g., the definition for Consonant below. A single "Accept Char....When Consonant" test tests the char value against the entire specification including individual enums, and the ranges specified as well. This is the mechanism I've seen used to specify that several different enumerated codes correspond to the same record format. I've only seen the multiple enumerated constants in practice, not the range specifications: IDENTIFICATION DIVISION. PROGRAM-ID. Conditions. AUTHOR. Michael Coughlan. * An example program demonstrating the use of * condition names (level 88's). * The EVALUATE and PERFORM verbs are also used. DATA DIVISION. WORKING-STORAGE SECTION. 01 Char PIC X. 88 Vowel VALUE "a", "e", "i", "o", "u". 88 Consonant VALUE "b", "c", "d", "f", "g", "h" "j" THRU "n", "p" THRU "t", "v" THRU "z". 88 Digit VALUE "0" THRU "9". 88 ValidCharacter VALUE "a" THRU "z", "0" THRU "9". PROCEDURE DIVISION. Begin. DISPLAY "Enter lower case character or digit. No data ends.". ACCEPT Char. PERFORM UNTIL NOT ValidCharacter EVALUATE TRUE WHEN Vowel DISPLAY "The letter " Char " is a vowel." WHEN Consonant DISPLAY "The letter " Char " is a consonant." WHEN Digit DISPLAY Char " is a digit." WHEN OTHER DISPLAY "problems found" END-EVALUATE END-PERFORM STOP RUN. I suggest that tests as complicated as the ones above that include range specifications should turn into normal discriminators in DFDL. But what if it was just enumerated values, e.g., I take the ranges off.... 01 Char PIC XX. 88 BaseRecord VALUE "a1", "11", "52". 88 Ext-Record VALUE "bb", "b0" This is the example I've actually seen in data. So in this case we just need the ability for the elementAlias to be a list of potential values. Or we can just be satisfied with requiring each of a1, 11, and 52 to be a separate element with a different name, so the above becomes a choice of 6 things, or we can just require a regular old discriminator in any case more general than single-value dispatch. I wanted everyone to see the example, but I actually advocate the narrowest possible definition here. I think the dfdl:elementAlias should be exactly one value only, and users should use a discriminator with a test for anything more general. ...mikeb

2 1

Issue 168 - Re: Fw: Escape scheme properties that can be expressions
by Mike Beckerle 05 Apr '12

05 Apr '12

In the absence of a compelling reason to allow relative paths here, I think the conservative choice is to retain this rule from 6.3.2. However, I did think about this and my evaluation on this topic is that allowing relative paths makes things just a little more complicated analytically to insure one gets good performance, but as DFDL does not allow recursive types, it is a relatively simple analysis to convert any relative path use in DFDL into absolute paths. So I am not worried about there being performance impact. ...mikeb On Mon, Apr 2, 2012 at 7:30 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > We took action 168 on the last call to discuss the issue of relative paths > in expressions for properties in global formats. > > *168* > *Relative path expressions in global annotations (Mike)* > 28/03: Mike to check that allowing relative paths does not cause > performance problems for an implementation. > 05/04: > > > Just seen this in section 6.3.2. of the spec: "DFDL expressions that are > used to provide the value of DFDL properties in the dfd:format annotation > on the xs:schema MAY NOT contain relative paths". > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 02/04/2012 12:28 ----- > > From: Steve Hanson/UK/IBM > To: dfdl-wg(a)ogf.org > Date: 14/03/2012 16:09 > Subject: Escape scheme properties that can be expressions > ------------------------------ > > > The spec allows the escapeCharacter and escapeEscapeCharacter of DFDL > escapeScheme to be expressions, in case the characters are supplied > dynamically in the data stream. > However given that escape schemes are defined globally, should these > properties be permitted to take relative paths in the expression? > Today the spec implicitly allows this, meaning an implementation can not > entirely validate an escapeScheme annotation up front, but must validate > escapeCharacter and escapeEscapeCharacter at each point of use. > I'm not sure it makes much sense for the path to be anything than absolute. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > > > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

OGF DFDL WG Call Agenda 2012-04-05 - Call moved to Thurs & time changed !!
by Steve Hanson 02 Apr '12

02 Apr '12

Please find agenda attached for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Call is on Thurs 2pm UK (9am Eastern) for this week only. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Escape scheme properties that can be expressions
by Steve Hanson 02 Apr '12

02 Apr '12

We took action 168 on the last call to discuss the issue of relative paths in expressions for properties in global formats. 168 Relative path expressions in global annotations (Mike) 28/03: Mike to check that allowing relative paths does not cause performance problems for an implementation. 05/04: Just seen this in section 6.3.2. of the spec: "DFDL expressions that are used to provide the value of DFDL properties in the dfd:format annotation on the xs:schema MAY NOT contain relative paths". Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/04/2012 12:28 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 14/03/2012 16:09 Subject: Escape scheme properties that can be expressions The spec allows the escapeCharacter and escapeEscapeCharacter of DFDL escapeScheme to be expressions, in case the characters are supplied dynamically in the data stream. However given that escape schemes are defined globally, should these properties be permitted to take relative paths in the expression? Today the spec implicitly allows this, meaning an implementation can not entirely validate an escapeScheme annotation up front, but must validate escapeCharacter and escapeEscapeCharacter at each point of use. I'm not sure it makes much sense for the path to be anything than absolute. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] [wg-all] New document published: WS-Disagreement
by Mike Beckerle 01 Apr '12

01 Apr '12

Somebody said this spec is really relevant to the DFDL group, but I can't seem to open the document on my TRS-80. The data file begins with hex 4170 7269 6c20 466f 6f6c 73. Can somebody help me figure out what this is? ...mikeb On Sun, Apr 1, 2012 at 4:00 PM, Andre Merzky <andre(a)merzky.net> wrote: > OGF Community: > > A new document has been published in the OGF series. All OGF > documents (including some that are open for public comment!) may be > found here: > > http://www.ogf.org/gf/docs/ > > * GFD-R.199 "WS-Disagreement" Via OGF Operations. > A. Merzky, A. Papaspyrou > April 1st 2012 > https://www.ogf.org/documents/GFD.199.pdf > > > Abstract: > > This document provides information to the grid community, proposing a > standard for a WS-Disagreement protocol (WS-NO), to communicate > disagreement > messages between components, and also between humans. > > > Andre Merzky, for OGF Operations > > -- > Nothing is ever easy... > -- > The wg-all mailing list is an umbrella list for all > OGF mailing lists. If you are subscribed to any OGF > mailing list, you will receive mails to wg-all. > > You can unsubscribe from this list on > > http://www.ogf.org/cgi-bin/perl/unsubscribe-wg-all.pl > > wg-all mailing list > wg-all(a)ogf.org > https://www.ogf.org/mailman/listinfo/wg-all -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

Re: [DFDL-WG] dfdl-wg Digest, Vol 69, Issue 21
by Jeffrey C. Jacobs 29 Mar '12

29 Mar '12

FYI guys, Sorry I missed y'all. Thought the meeting was 14:00 EDT and only too late realized it was 14:00 GDT and 09:00 EDT here. That said, Wednesdays at 9am does work better for me or at least 9am does in order to avoid local traffic. But not much to report here anyway so you didn't miss anything from me. I'm going over the meeting minutes though the minutes and will post to the list if I have any questions of a general nature. Thanks Steve for getting the minutes up so quickly!

2 1

Rescheduled: OGF DFDL Working Group weekly call (5 Apr 14:00 GDT in IBM Hursley DE2K09)
by Steve Hanson 28 Mar '12

28 Mar '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Steve on vacation next Tues so moving call to Thurs 2pm UK

1 0

OGF DFDL WG Call Minutes 2012-03-28
by Steve Hanson 28 Mar '12

28 Mar '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is still at version 8 and may be found via https://forge.ogf.org/sf/projects/dfdl-wg Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 145: 'dispatch' way of discriminating a choice for better performance
by Steve Hanson 28 Mar '12

28 Mar '12

Updated to allow more than one tag value to map to the same choice branch. Please review property names and rules for discussion next week. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/03/2012 16:32 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 27/03/2012 14:09 Subject: Fw: [DFDL-WG] Action 145: 'dispatch' way of discriminating a choice for better performance Hi Mike I agree. When I wrote the SWIFT example expression that I realised this was becoming a bit too complicated. IBM MRM has a quick way of dispatching a choice that uses a similar mechanism to the one you propose. It has an optional property called 'Message Alias'. The parser first tries to match the tag against the element name, and if that fails it tries the alias. I've modified my proposal accordingly. Regarding wildcards, the approach you show was what we had agreed upon a couple of years ago when we dropped wildcards. I would prefer not to add them back in for 1.0, but I have had some feedback from IBMers modeling envelope/payload formats that they want a loose-coupling approach as that fits better with other tools and is scalable. Let's treat this as a separate issue. I'll go back to the IBMers so that I fully understand their concerns, and if necessary I will bring back to the WG as a new agenda item. So all we need to ensure for now is that whatever mechanism we come up with is extensible to wildcards, if needed. Revised details: A new dfdl:element property is added called dfdl:elementAlias of type 'List of String'. This provides one or more alternative names for the element. Only allowed on local element and global element, not on element reference or simple type. A new dfdl:choice property is added called dfdl:choiceBranchRef of type DFDL Expression. The expression must evaluate to String. The string must match (case insensitive) either the element name or one of the dfdl:elementAlias property values of one of the element branches of the choice, and if so discriminates in favour of that branch. The parser then goes straight to that branch, ignoring schema order. Because the branch is 'known to exist' no backtracking takes place if a processing error occurs. Rules: - Both properties behave like dfdl:ref and dfdl:hiddenGroupRef in that it is not possible to set a value in scope by a dfdl:format annotation, and is only set at its point of use. This is because there is nothing sensible that could be set in scope. But it has the benefit that adding support for the property to existing DFDL implementations will not suddenly cause errors to appear in existing DFDL schemas. Empty string is not an allowed value. - Both properties are only used when parsing. - When dfdl:choiceBranchRef is preent, all choice branches must be local elements or element references. It is a schema definition error otherwise. - It is a processing error if the resolved xs:string value of dfdl:choiceBranchRef does not match one of the branches. - It is a schema definition error if a choice has both dfdl:choiceBranchRef set and dfdl:initiatedContent="yes". - It is a schema definition error if individual dfdl:elementAlias values are not unique a) across all elements that are branches of a given xs:choice and b) across all global elements in the schema. So we now have the ability to apply a simple lookup before we start to process a choice. That makes the processing time for each branch of the choice independent of its order in the schema. Questions: - What happens if we encounter a discriminator once we are processing the branch and its point of uncertainty is the choice ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 27/03/2012 13:41 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 21/03/2012 15:57 Subject: Re: [DFDL-WG] Action 145: 'dispatch' way of discriminating a choice for better performance (updated) Sent by: dfdl-wg-bounces(a)ogf.org The whole point of this thing is to be faster, not more general, so my reaction is too much XPath expression complexity here. Consider this. If the tag is some 2-character code that does NOT want to be the same as the element names (for example because they're digits, so they can't be the element names exactly since element names have to begin with an alpha char. Digits also aren't useful from readability perspective as names), then you'll need a big lookup table in the choiceBranchRef expression that translates from the codes to the QNames. We don't have a case statement in the expression language, so you've just moved the big linear evaluation chain out of evaluating choice discriminators one after another and into a big if-then-else nest in the choiceBranchRef expression. I don't see a performance gain here. I suggest dropping the QName stuff, and requiring a dfdl:choiceID property on the elements that is an NCName. (Well, we might want those QName functions anyway in the expression language. But I wouldn't use them for this rapid choice dispatch feature. You could certainly use them in discriminators. ) The expression would then have to evaluate to a value that is matched against this choiceID. I suggest exact match, not respecting ignoreCase for example. That eliminates all the QName complexity and is amenable to high-speed compact lookup table implementation. I tend to think the element names want to be a little bit more descriptive than these tag values would want to be so using the element names as the tags feels undesirable to me. Particularly because we want the tags to be conveniently computed, for example by just grabbing a fixed-length string out of a data field. You end up with something like this: <element name="tag" type="string" dfdl:length="{ 2 }" ..../> <choice dfdl:choiceBranchRef="{ ../tag }"> <element name="someName" dfdl:choiceID="02" .../> <element name="anotherName" dfdl:choiceID="73" .../> .... </choice> As for the wild-card issue. I think we can finesse this. Consider this model: <element name="tag" type="string" dfdl:length="{ 2 }" ..../> <choice>  <choice dfdl:choiceBranchRef="{ ../tag }"> <element name="someName" dfdl:choiceID="02" .../> <element name="anotherName" dfdl:choiceID="73" .../> .... </choice>  <element name="extensionRecord"> <complexType> <sequence>  <element name="extensionType" type="string" dfdl:inputValueCalc="../../tag" .../> .... </sequence> </complexType> </element> </choice> The inner choice uses the fast dispatch. The outer choice lets me also have an alternative that absorbs a more general syntax to provide some way for a user to model their extensions to the choice set. The extensionType field captures the tag and stores it inside the extension record where it won't get disassociated. The user's "extensionRecord" would not be a special DFDL wildcard construct, just an element format they create which is general enough to parse their extensions. Or this extension could be predefined as part of a standard, to accept any standard-defined acceptable extension record a user might need so long as there is some set of rules all extension records must obey. Given this, do we really need special wildcard constructs? ...mikeb From: Steve Hanson <smh(a)uk.ibm.com> To: dfdl-wg(a)ogf.org, Date: 24.03.2012 02:27 Subject: [DFDL-WG] Action 145: 'dispatch' way of discriminating a choice for better performance The enveope/payload style of data format is quite common, where the envelope provides control information and the payload contains the business data. Examples are SWIFT and SAP IDocs. Typically the envelope contains a tag that identifies the payload, which can be one of many types. For SWIFT there are 300 possible types. To model this today in DFDL requires an xs:choice with each type modeled as an xs:element branch of the choice. A discriminator on each xs:element refers back to the envelope tag element thus enabling the choice to be resolved. There are two issues with this approach. 1) Performance. Even if the elements in the branches are ordered for expected frequency, there will still be cases when tens or hundreds of discriminators need to be evaluated before the choice is resolved. 2) Tight coupling. When a new type is added, a new element branch needs to be added to the choice. Action 145 proposes a mechanism to solve issue #1 and which opens the door to a possible extension to DFDL to solve issue #2 - namely a faster way to resolve a choice. Details: A new dfdl:choice property is added called dfdl:choiceBranchRef of type DFDL Expression. The expression must evaluate to a QName which corresponds to one of the element branches of the choice, and asserts 'known to exist' for that branch. Rules: - The property behaves like dfdl:ref and dfdl:hiddenGroupRef in that it is not possible to set a value in scope by a dfdl:format annotation, and is only set at its point of use. This is because there is nothing sensible that could be set in scope. But it has the benefit that adding support for the property to existing DFDL implementations will not suddenly cause errors to appear in existing DFDL schemas. - Empty string is not an allowed value. - The property is only used when parsing. - All branches must be local elements or element references. It is a schema definition error if any branch is a sequence, a choice or a group reference. - It is a processing error if the QName does not resolve to one of the branches when parsing.. - It is a schema definition error if a choice has the property set and also has dfdl:initiatedContent="yes" set locally. - Because the expression must return a QName, the expression language must provide a constructor for creating a QName from a namespace string and a name string. If you take SWIFT MT103 payload as an example, the tag in the envelope says "103" but a DFDL schema would actually model the global MT103 element with name "Document" and namespace ="urn:swift:xsd:fin.103.2011". So the dfdl:choiceBranchRef expression would have to look like: {fn:QName(fn:concat(fn:concat('urn:swift:xsd:fin.', FinMessage/Block2/MessageType), ".2011"), 'Document')} So we now have the ability to derive a QName and apply it before we start to process a choice. That makes the processing time for each branch of the choice independent of its order in the schema. We still have issue #2 so when a new payload is added, a new branch must be added to the choice. A solution to this is to allows xs:any wildcard elements back into DFDL, then provide a property dfdl:wildcardRef which works in the same way as dfdl:choiceRef. So at the point of encountering the wildcard we know its resolution in the schema. This obviously will require some further discussion, but you can see how this ability to evaluate an expression and return a QName can be used in multiple ways. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0