- dfdl-wg - lists.ogf.org

Fw: Action 145: 'dispatch' way of discriminating a choice for better performance
by Steve Hanson 28 Mar '12

28 Mar '12

Updated to allow more than one tag value to map to the same choice branch. Please review property names and rules for discussion next week. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/03/2012 16:32 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 27/03/2012 14:09 Subject: Fw: [DFDL-WG] Action 145: 'dispatch' way of discriminating a choice for better performance Hi Mike I agree. When I wrote the SWIFT example expression that I realised this was becoming a bit too complicated. IBM MRM has a quick way of dispatching a choice that uses a similar mechanism to the one you propose. It has an optional property called 'Message Alias'. The parser first tries to match the tag against the element name, and if that fails it tries the alias. I've modified my proposal accordingly. Regarding wildcards, the approach you show was what we had agreed upon a couple of years ago when we dropped wildcards. I would prefer not to add them back in for 1.0, but I have had some feedback from IBMers modeling envelope/payload formats that they want a loose-coupling approach as that fits better with other tools and is scalable. Let's treat this as a separate issue. I'll go back to the IBMers so that I fully understand their concerns, and if necessary I will bring back to the WG as a new agenda item. So all we need to ensure for now is that whatever mechanism we come up with is extensible to wildcards, if needed. Revised details: A new dfdl:element property is added called dfdl:elementAlias of type 'List of String'. This provides one or more alternative names for the element. Only allowed on local element and global element, not on element reference or simple type. A new dfdl:choice property is added called dfdl:choiceBranchRef of type DFDL Expression. The expression must evaluate to String. The string must match (case insensitive) either the element name or one of the dfdl:elementAlias property values of one of the element branches of the choice, and if so discriminates in favour of that branch. The parser then goes straight to that branch, ignoring schema order. Because the branch is 'known to exist' no backtracking takes place if a processing error occurs. Rules: - Both properties behave like dfdl:ref and dfdl:hiddenGroupRef in that it is not possible to set a value in scope by a dfdl:format annotation, and is only set at its point of use. This is because there is nothing sensible that could be set in scope. But it has the benefit that adding support for the property to existing DFDL implementations will not suddenly cause errors to appear in existing DFDL schemas. Empty string is not an allowed value. - Both properties are only used when parsing. - When dfdl:choiceBranchRef is preent, all choice branches must be local elements or element references. It is a schema definition error otherwise. - It is a processing error if the resolved xs:string value of dfdl:choiceBranchRef does not match one of the branches. - It is a schema definition error if a choice has both dfdl:choiceBranchRef set and dfdl:initiatedContent="yes". - It is a schema definition error if individual dfdl:elementAlias values are not unique a) across all elements that are branches of a given xs:choice and b) across all global elements in the schema. So we now have the ability to apply a simple lookup before we start to process a choice. That makes the processing time for each branch of the choice independent of its order in the schema. Questions: - What happens if we encounter a discriminator once we are processing the branch and its point of uncertainty is the choice ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 27/03/2012 13:41 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 21/03/2012 15:57 Subject: Re: [DFDL-WG] Action 145: 'dispatch' way of discriminating a choice for better performance (updated) Sent by: dfdl-wg-bounces(a)ogf.org The whole point of this thing is to be faster, not more general, so my reaction is too much XPath expression complexity here. Consider this. If the tag is some 2-character code that does NOT want to be the same as the element names (for example because they're digits, so they can't be the element names exactly since element names have to begin with an alpha char. Digits also aren't useful from readability perspective as names), then you'll need a big lookup table in the choiceBranchRef expression that translates from the codes to the QNames. We don't have a case statement in the expression language, so you've just moved the big linear evaluation chain out of evaluating choice discriminators one after another and into a big if-then-else nest in the choiceBranchRef expression. I don't see a performance gain here. I suggest dropping the QName stuff, and requiring a dfdl:choiceID property on the elements that is an NCName. (Well, we might want those QName functions anyway in the expression language. But I wouldn't use them for this rapid choice dispatch feature. You could certainly use them in discriminators. ) The expression would then have to evaluate to a value that is matched against this choiceID. I suggest exact match, not respecting ignoreCase for example. That eliminates all the QName complexity and is amenable to high-speed compact lookup table implementation. I tend to think the element names want to be a little bit more descriptive than these tag values would want to be so using the element names as the tags feels undesirable to me. Particularly because we want the tags to be conveniently computed, for example by just grabbing a fixed-length string out of a data field. You end up with something like this: <element name="tag" type="string" dfdl:length="{ 2 }" ..../> <choice dfdl:choiceBranchRef="{ ../tag }"> <element name="someName" dfdl:choiceID="02" .../> <element name="anotherName" dfdl:choiceID="73" .../> .... </choice> As for the wild-card issue. I think we can finesse this. Consider this model: <element name="tag" type="string" dfdl:length="{ 2 }" ..../> <choice>  <choice dfdl:choiceBranchRef="{ ../tag }"> <element name="someName" dfdl:choiceID="02" .../> <element name="anotherName" dfdl:choiceID="73" .../> .... </choice>  <element name="extensionRecord"> <complexType> <sequence>  <element name="extensionType" type="string" dfdl:inputValueCalc="../../tag" .../> .... </sequence> </complexType> </element> </choice> The inner choice uses the fast dispatch. The outer choice lets me also have an alternative that absorbs a more general syntax to provide some way for a user to model their extensions to the choice set. The extensionType field captures the tag and stores it inside the extension record where it won't get disassociated. The user's "extensionRecord" would not be a special DFDL wildcard construct, just an element format they create which is general enough to parse their extensions. Or this extension could be predefined as part of a standard, to accept any standard-defined acceptable extension record a user might need so long as there is some set of rules all extension records must obey. Given this, do we really need special wildcard constructs? ...mikeb From: Steve Hanson <smh(a)uk.ibm.com> To: dfdl-wg(a)ogf.org, Date: 24.03.2012 02:27 Subject: [DFDL-WG] Action 145: 'dispatch' way of discriminating a choice for better performance The enveope/payload style of data format is quite common, where the envelope provides control information and the payload contains the business data. Examples are SWIFT and SAP IDocs. Typically the envelope contains a tag that identifies the payload, which can be one of many types. For SWIFT there are 300 possible types. To model this today in DFDL requires an xs:choice with each type modeled as an xs:element branch of the choice. A discriminator on each xs:element refers back to the envelope tag element thus enabling the choice to be resolved. There are two issues with this approach. 1) Performance. Even if the elements in the branches are ordered for expected frequency, there will still be cases when tens or hundreds of discriminators need to be evaluated before the choice is resolved. 2) Tight coupling. When a new type is added, a new element branch needs to be added to the choice. Action 145 proposes a mechanism to solve issue #1 and which opens the door to a possible extension to DFDL to solve issue #2 - namely a faster way to resolve a choice. Details: A new dfdl:choice property is added called dfdl:choiceBranchRef of type DFDL Expression. The expression must evaluate to a QName which corresponds to one of the element branches of the choice, and asserts 'known to exist' for that branch. Rules: - The property behaves like dfdl:ref and dfdl:hiddenGroupRef in that it is not possible to set a value in scope by a dfdl:format annotation, and is only set at its point of use. This is because there is nothing sensible that could be set in scope. But it has the benefit that adding support for the property to existing DFDL implementations will not suddenly cause errors to appear in existing DFDL schemas. - Empty string is not an allowed value. - The property is only used when parsing. - All branches must be local elements or element references. It is a schema definition error if any branch is a sequence, a choice or a group reference. - It is a processing error if the QName does not resolve to one of the branches when parsing.. - It is a schema definition error if a choice has the property set and also has dfdl:initiatedContent="yes" set locally. - Because the expression must return a QName, the expression language must provide a constructor for creating a QName from a namespace string and a name string. If you take SWIFT MT103 payload as an example, the tag in the envelope says "103" but a DFDL schema would actually model the global MT103 element with name "Document" and namespace ="urn:swift:xsd:fin.103.2011". So the dfdl:choiceBranchRef expression would have to look like: {fn:QName(fn:concat(fn:concat('urn:swift:xsd:fin.', FinMessage/Block2/MessageType), ".2011"), 'Document')} So we now have the ability to derive a QName and apply it before we start to process a choice. That makes the processing time for each branch of the choice independent of its order in the schema. We still have issue #2 so when a new payload is added, a new branch must be added to the choice. A solution to this is to allows xs:any wildcard elements back into DFDL, then provide a property dfdl:wildcardRef which works in the same way as dfdl:choiceRef. So at the point of encountering the wildcard we know its resolution in the schema. This obviously will require some further discussion, but you can see how this ability to evaluate an expression and return a QName can be used in multiple ways. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Proposed Errata: textNumberPattern P, V, remove support for double, float types
by Steve Hanson 28 Mar '12

28 Mar '12

Proposal accepted, errata taken. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/03/2012 16:31 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org Date: 22/03/2012 11:13 Subject: Re: [DFDL-WG] Proposed Errata: textNumberPattern P, V, remove support for double, float types To confirm that the PICTURE clause is not used when the USAGE is COMP-1 (float) and COMP-2 (double). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org Date: 13/03/2012 15:33 Subject: [DFDL-WG] Proposed Errata: textNumberPattern P, V, remove support for double, float types Sent by: dfdl-wg-bounces(a)ogf.org Proposed Errata: The P and V pattern characters have this in their meaning description "Only used with decimal, float and double simple types." Change: remove float and decimal. "Only used with decimal type." -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] lengthKind = prefixed and length in bits when serializing...
by Steve Hanson 28 Mar '12

28 Mar '12

WG agreed that equivalent of Table 16 is needed for lengthKind 'prefixed' and 'delimited', and no longer needed for 'pattern'. WG agreed that for lengthKind 'prefixed' and lengthUnits 'bits' there is a limitation that you might not be able to unparse exactly what you parsed. This is preferable to disallowing altogether. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB Cc: Alex Wood1/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 26/03/2012 11:18 Subject: Re: lengthKind = prefixed and length in bits when serializing... Alex What you suggest makes sense. There is an implicit limitation with "prefixed" in that I can parse an element with an odd number of bits (say 13) but be unable to serialize exactly what had been parsed. I don't think there's anything you can do about that, other than use "explicit" with an expression as Tim says. However, I think what you have noted is an issue for all "prefixed" binary rep data, not just lengthUnits "bits". Take a look at Table 16 in section 12.3.5. This was added to handle the same issue for lengthKind "pattern" where all we have is the infoset data. The rightmost column says what to output for binary reps (in practice, it is virtually identical to Table 15 for "implicit" in section 12.3.3). We need this for "prefixed" too. Further, some of the rows in Table 16 are needed for "delimited". Delimited binary data is permitted under a couple of circumstances - when the binary rep is "packed" or "BCD". Section 12.3.2 does not cover what to output for binary reps. Note that errata 3.9 effectively removes Table 16 as "pattern" no longer supports binary rep data. The information in the text column is still needed though, but a closer examination says it is incorrect, as it should take into account dfdl:textOutputMinLength or xs:minLength (see wording in 12.3.2 for "delimited" and 12.3.4 for "prefixed"). Let's discuss on next DFDL WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Alex Wood1/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 23/03/2012 21:55 Subject: Re: lengthKind = prefixed and length in bits when serializing... Fair question. Off the cuff response is - follow lengthKind=implicit rules for lengthKind=prefixed - if that's not what the modeller wants then they can always use lengthKind=explicit with an expression and model the prefix length as a separate element. Then they can use an outputValueCalc expression on the length element to produce whatever result they require. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 23/03/2012 19:33 Subject: lengthKind = prefixed and length in bits when serializing... Hi Steve, Tim. Looking at one of Neil's defects for lengthKind = prefixed Case where lengthUnits = Bits How to we determine what length (in bits) to serialize from the value and logical type ? I could not find a clear definitive answer in the specification. On parsing it is clear that any length is OK as long as it is less than the natural length of the logical type. So the same rules as for lengthKind = explicit, only the length was parsed out of the prefix. On serializing, lengthKind = explicit and implicit have specific lengths to serialize. However lengthKind = prefixed does not have a length to work with. That's ok for lengthUnits = bytes or characters where we know the length of data we are putting in the bitstream. But for lengthKind = bits it's not so obvious. For example an integer value of 1 for an unsignedInt could be serialized in a length anything between 1 and 32 bits. One way would be to find the minimum number of bits we could use to serialized all significant bits of the value. However my instinct is that it ought to follow the lengthKind = implicit rules and serialize the number of bits for the logical types natural length. UnsignedLong 64 UnsignedInt 32 UnsignedShort 16 UnsignedByte 8 Boolean 32 Thoughts ? Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development Lab Advocate - PMI Mortage MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-03-28 - Call moved to Wed & time changed (again) !!
by Steve Hanson 27 Mar '12

27 Mar '12

Please find agenda attached for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Call is on Wed 2pm UK (9am Eastern) for this week only. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 145: 'dispatch' way of discriminating a choice for better performance
by Steve Hanson 27 Mar '12

27 Mar '12

Hi Mike I agree. When I wrote the SWIFT example expression that I realised this was becoming a bit too complicated. IBM MRM has a quick way of dispatching a choice that uses a similar mechanism to the one you propose. It has an optional property called 'Message Alias'. The parser first tries to match the tag against the element name, and if that fails it tries the alias. I've modified my proposal accordingly. Regarding wildcards, the approach you show was what we had agreed upon a couple of years ago when we dropped wildcards. I would prefer not to add them back in for 1.0, but I have had some feedback from IBMers modeling envelope/payload formats that they want a loose-coupling approach as that fits better with other tools and is scalable. Let's treat this as a separate issue. I'll go back to the IBMers so that I fully understand their concerns, and if necessary I will bring back to the WG as a new agenda item. So all we need to ensure for now is that whatever mechanism we come up with is extensible to wildcards, if needed. Revised details: A new dfdl:element property is added called dfdl:elementAlias of type xs:NCName. This provides an alternative name for the element. Only allowed on local element and global element, not on element reference or simple type. A new dfdl:choice property is added called dfdl:choiceBranchRef of type DFDL Expression. The expression must evaluate to xs:NCName. The string must match either the element name or the dfdl:elementAlias property of one of the element branches of the choice, and if so discriminates in favour of that branch. The parser then goes straight to that branch, ignoring schema order. Because the branch is 'known to exist' no backtracking takes place if a processing error occurs. Rules: - Both properties behave like dfdl:ref and dfdl:hiddenGroupRef in that it is not possible to set a value in scope by a dfdl:format annotation, and is only set at its point of use. This is because there is nothing sensible that could be set in scope. But it has the benefit that adding support for the property to existing DFDL implementations will not suddenly cause errors to appear in existing DFDL schemas. Empty string is not an allowed value. - Both properties are only used when parsing. - When dfdl:choiceBranchRef is preent, all choice branches must be local elements or element references. It is a schema definition error otherwise. - It is a processing error if the resolved xs:string value of dfdl:choiceBranchRef does not match one of the branches. - It is a schema definition error if a choice has both dfdl:choiceBranchRef set and dfdl:initiatedContent="yes". So we now have the ability to apply a simple lookup before we start to process a choice. That makes the processing time for each branch of the choice independent of its order in the schema. Questions: - What happens if we encounter a discriminator once we are processing the branch and its point of uncertainty is the choice ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 27/03/2012 13:41 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 21/03/2012 15:57 Subject: Re: [DFDL-WG] Action 145: 'dispatch' way of discriminating a choice for better performance (updated) Sent by: dfdl-wg-bounces(a)ogf.org The whole point of this thing is to be faster, not more general, so my reaction is too much XPath expression complexity here. Consider this. If the tag is some 2-character code that does NOT want to be the same as the element names (for example because they're digits, so they can't be the element names exactly since element names have to begin with an alpha char. Digits also aren't useful from readability perspective as names), then you'll need a big lookup table in the choiceBranchRef expression that translates from the codes to the QNames. We don't have a case statement in the expression language, so you've just moved the big linear evaluation chain out of evaluating choice discriminators one after another and into a big if-then-else nest in the choiceBranchRef expression. I don't see a performance gain here. I suggest dropping the QName stuff, and requiring a dfdl:choiceID property on the elements that is an NCName. (Well, we might want those QName functions anyway in the expression language. But I wouldn't use them for this rapid choice dispatch feature. You could certainly use them in discriminators. ) The expression would then have to evaluate to a value that is matched against this choiceID. I suggest exact match, not respecting ignoreCase for example. That eliminates all the QName complexity and is amenable to high-speed compact lookup table implementation. I tend to think the element names want to be a little bit more descriptive than these tag values would want to be so using the element names as the tags feels undesirable to me. Particularly because we want the tags to be conveniently computed, for example by just grabbing a fixed-length string out of a data field. You end up with something like this: <element name="tag" type="string" dfdl:length="{ 2 }" ..../> <choice dfdl:choiceBranchRef="{ ../tag }"> <element name="someName" dfdl:choiceID="02" .../> <element name="anotherName" dfdl:choiceID="73" .../> .... </choice> As for the wild-card issue. I think we can finesse this. Consider this model: <element name="tag" type="string" dfdl:length="{ 2 }" ..../> <choice>  <choice dfdl:choiceBranchRef="{ ../tag }"> <element name="someName" dfdl:choiceID="02" .../> <element name="anotherName" dfdl:choiceID="73" .../> .... </choice>  <element name="extensionRecord"> <complexType> <sequence>  <element name="extensionType" type="string" dfdl:inputValueCalc="../../tag" .../> .... </sequence> </complexType> </element> </choice> The inner choice uses the fast dispatch. The outer choice lets me also have an alternative that absorbs a more general syntax to provide some way for a user to model their extensions to the choice set. The extensionType field captures the tag and stores it inside the extension record where it won't get disassociated. The user's "extensionRecord" would not be a special DFDL wildcard construct, just an element format they create which is general enough to parse their extensions. Or this extension could be predefined as part of a standard, to accept any standard-defined acceptable extension record a user might need so long as there is some set of rules all extension records must obey. Given this, do we really need special wildcard constructs? ...mikeb From: Steve Hanson <smh(a)uk.ibm.com> To: dfdl-wg(a)ogf.org, Date: 24.03.2012 02:27 Subject: [DFDL-WG] Action 145: 'dispatch' way of discriminating a choice for better performance The enveope/payload style of data format is quite common, where the envelope provides control information and the payload contains the business data. Examples are SWIFT and SAP IDocs. Typically the envelope contains a tag that identifies the payload, which can be one of many types. For SWIFT there are 300 possible types. To model this today in DFDL requires an xs:choice with each type modeled as an xs:element branch of the choice. A discriminator on each xs:element refers back to the envelope tag element thus enabling the choice to be resolved. There are two issues with this approach. 1) Performance. Even if the elements in the branches are ordered for expected frequency, there will still be cases when tens or hundreds of discriminators need to be evaluated before the choice is resolved. 2) Tight coupling. When a new type is added, a new element branch needs to be added to the choice. Action 145 proposes a mechanism to solve issue #1 and which opens the door to a possible extension to DFDL to solve issue #2 - namely a faster way to resolve a choice. Details: A new dfdl:choice property is added called dfdl:choiceBranchRef of type DFDL Expression. The expression must evaluate to a QName which corresponds to one of the element branches of the choice, and asserts 'known to exist' for that branch. Rules: - The property behaves like dfdl:ref and dfdl:hiddenGroupRef in that it is not possible to set a value in scope by a dfdl:format annotation, and is only set at its point of use. This is because there is nothing sensible that could be set in scope. But it has the benefit that adding support for the property to existing DFDL implementations will not suddenly cause errors to appear in existing DFDL schemas. - Empty string is not an allowed value. - The property is only used when parsing. - All branches must be local elements or element references. It is a schema definition error if any branch is a sequence, a choice or a group reference. - It is a processing error if the QName does not resolve to one of the branches when parsing.. - It is a schema definition error if a choice has the property set and also has dfdl:initiatedContent="yes" set locally. - Because the expression must return a QName, the expression language must provide a constructor for creating a QName from a namespace string and a name string. If you take SWIFT MT103 payload as an example, the tag in the envelope says "103" but a DFDL schema would actually model the global MT103 element with name "Document" and namespace ="urn:swift:xsd:fin.103.2011". So the dfdl:choiceBranchRef expression would have to look like: {fn:QName(fn:concat(fn:concat('urn:swift:xsd:fin.', FinMessage/Block2/MessageType), ".2011"), 'Document')} So we now have the ability to derive a QName and apply it before we start to process a choice. That makes the processing time for each branch of the choice independent of its order in the schema. We still have issue #2 so when a new payload is added, a new branch must be added to the choice. A solution to this is to allows xs:any wildcard elements back into DFDL, then provide a property dfdl:wildcardRef which works in the same way as dfdl:choiceRef. So at the point of encountering the wildcard we know its resolution in the schema. This obviously will require some further discussion, but you can see how this ability to evaluate an expression and return a QName can be used in multiple ways. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] lengthKind = prefixed and length in bits when serializing...
by Steve Hanson 26 Mar '12

26 Mar '12

Alex What you suggest makes sense. There is an implicit limitation with "prefixed" in that I can parse an element with an odd number of bits (say 13) but be unable to serialize exactly what had been parsed. I don't think there's anything you can do about that, other than use "explicit" with an expression as Tim says. However, I think what you have noted is an issue for all "prefixed" binary rep data, not just lengthUnits "bits". Take a look at Table 16 in section 12.3.5. This was added to handle the same issue for lengthKind "pattern" where all we have is the infoset data. The rightmost column says what to output for binary reps (in practice, it is virtually identical to Table 15 for "implicit" in section 12.3.3). We need this for "prefixed" too. Further, some of the rows in Table 16 are needed for "delimited". Delimited binary data is permitted under a couple of circumstances - when the binary rep is "packed" or "BCD". Section 12.3.2 does not cover what to output for binary reps. Note that errata 3.9 effectively removes Table 16 as "pattern" no longer supports binary rep data. The information in the text column is still needed though, but a closer examination says it is incorrect, as it should take into account dfdl:textOutputMinLength or xs:minLength (see wording in 12.3.2 for "delimited" and 12.3.4 for "prefixed"). Let's discuss on next DFDL WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Alex Wood1/UK/IBM@IBMGB Cc: Steve Hanson/UK/IBM@IBMGB Date: 23/03/2012 21:55 Subject: Re: lengthKind = prefixed and length in bits when serializing... Fair question. Off the cuff response is - follow lengthKind=implicit rules for lengthKind=prefixed - if that's not what the modeller wants then they can always use lengthKind=explicit with an expression and model the prefix length as a separate element. Then they can use an outputValueCalc expression on the length element to produce whatever result they require. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Alex Wood1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Tim Kimber/UK/IBM@IBMGB Date: 23/03/2012 19:33 Subject: lengthKind = prefixed and length in bits when serializing... Hi Steve, Tim. Looking at one of Neil's defects for lengthKind = prefixed Case where lengthUnits = Bits How to we determine what length (in bits) to serialize from the value and logical type ? I could not find a clear definitive answer in the specification. On parsing it is clear that any length is OK as long as it is less than the natural length of the logical type. So the same rules as for lengthKind = explicit, only the length was parsed out of the prefix. On serializing, lengthKind = explicit and implicit have specific lengths to serialize. However lengthKind = prefixed does not have a length to work with. That's ok for lengthUnits = bytes or characters where we know the length of data we are putting in the bitstream. But for lengthKind = bits it's not so obvious. For example an integer value of 1 for an unsignedInt could be serialized in a length anything between 1 and 32 bits. One way would be to find the minimum number of bits we could use to serialized all significant bits of the value. However my instinct is that it ought to follow the lengthKind = implicit rules and serialize the number of bits for the logical types natural length. UnsignedLong 64 UnsignedInt 32 UnsignedShort 16 UnsignedByte 8 Boolean 32 Thoughts ? Kind Regards, - Alex Alex Wood - Software Engineer - WebSphere Message Broker Development Lab Advocate - PMI Mortage MP 211, IBM UK Labs, Hursley Park, Winchester, Hants. SO21 2JN. Tel: Internal 246272, External 01962 816272 Notes: Alex Wood1/UK/IBM@IBMGB e-mail: wooda(a)uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Rescheduled: OGF DFDL Working Group weekly call (28 Mar 14:00 GDT in IBM Hursley DE2K09)
by Steve Hanson 23 Mar '12

23 Mar '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Have to reschedule again this week due to a meeting clash. Please accept/decline the reschedule notice, and I'll see whether there are enough acceptances to go ahead with the call.

1 0

OGF DFDL WG Call Minutes 2012-03-21
by Steve Hanson 22 Mar '12

22 Mar '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is at version 8 and may be found via https://forge.ogf.org/sf/projects/dfdl-wg Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Proposed Errata: textNumberPattern P, V, remove support for double, float types
by Mike Beckerle 22 Mar '12

22 Mar '12

Proposed Errata: The P and V pattern characters have this in their meaning description "Only used with decimal, float and double simple types." Change: remove float and decimal. "Only used with decimal type." -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

Fw: Action 167: textNumberPatterns with P, V, # - allowable combinations
by Steve Hanson 22 Mar '12

22 Mar '12

Further to discussion on the call, here is what IBM COBOL manual says about PIC P. An assumed decimal scaling position. It is used to specify the location of an assumed decimal point when the point is not within the number that appears in the data item. The scaling position character P is not counted in the size of the data item. Scaling position characters are counted in determining the maximum number of digit positions (63) in numeric-edited items or in items that appear as arithmetic operands. The scaling position character P may appear only as a continuous string of Ps in the leftmost or rightmost digit positions within a PICTURE character-string. Because the scaling position character P implies an assumed decimal point (to the left of the Ps, if the Ps are leftmost PICTURE characters; to the right of the Ps, if the Ps are rightmost PICTURE characters), the assumed decimal point symbol, V, is redundant as either the leftmost or rightmost character within such a PICTURE description. In certain operations that reference a data item whose PICTURE character-string contains the symbol P, the algebraic value of the data it em is used rather than the actual character representation of the data item. This algebraic value assumes the decimal point in the prescribed location and zero in place of the digit position specified by the symbol P . The size of the value is the number of digit positions represented by the PICTURE character-string. These operations are any of the following: Any operation requiring a numeric sending operand. A MOVE statement where the sending operand is numeric and its PICTURE character-string contains the symbol P. A MOVE statement where the sending operand is numeric-edited and its PICTURE character-string contains the symbol P and the receiving operand is numeric or numeric-edited. A comparison operation where both operands are numeric. In all other operations the digit positions specified with the symbol P are ignored and are not counted in the size of the operand. This implies that the scaling should be applied as a lexical operation on the data. In other words two COBOL fields, one with PIC PP9 and value '2' and one with PP999 and value '002' do not result in the same logical number. There is an equivalence between V and P. PP999 == V99999 and 999PP == 99999V == 99999. If we consider things in these terms the reasoning is simpler. To prevent # symbol zero suppression from changing the value, rule a) must apply and there must be no # to the right of the V. That restates our rules as: a) A pattern with a V symbol must not have # symbols to the right of the V symbol. b) A pattern with P symbols at the left end must have no # symbols in the pattern. c) A pattern with P symbols at the right end has no restrictions. There is another problem though. The number can be trimmed using the pad character from either or both ends depending on justification, before applying the number pattern. If the pad character is 0 then this can also cause 0's to be lost and result in mis-application of V and P symbols. I'm not sure there is much we can do about this. Modelers need to be careful when padding/trimming that they get the justification correct. For example, we typically think of numbers as being right justified, but for a number with Ps on the left, it is effectively left justified and should be modeled as such. We added errata 2.25 which prevented trimming from leaving an empty string. I am thinking that this errata should actually say that trimming must leave at least the minimum number of digits implied by the pattern, as an extra safeguard? We mustn't disallow trimming/padding altogether as it is used to remove spaces. The ICU pad character symbol * is used to provide a pad character when the data is shorter than the pattern. This is only used to pad when unparsing, it is not used to trim. But it might be safer to disallow P and V symbols when * is used? Reading the ICU description of significant digit symbol @, explicit decimal points are disallowed. I think we should disallow P and V symbols when @ symbol is used. Errata 2.28 should be updated. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/03/2012 09:22 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 21/03/2012 10:34 Subject: Action 167: textNumberPatterns with P,V, # - allowable combinations Last week we agreed that disallowing text number patterns that contained a # symbol and either a P or V symbol was too restrictive. Accordingly the following rules are proposed to control when # may be used in the same pattern as P or V to ensure an unambiguous pattern. a) Pattern must not have # symbols to the right of the V symbol. b) If pattern has P symbols at the left end, then there must be as many 0 symbols adjacent to the rightmost P symbol as there are P symbols. c) If pattern has P symbols at the right end, there are no restrictions. If a) or b) are violated it is a schema definition error. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0