Need dfdl:inputValueCalc and dfdl:outputValueCalc on same element
So in PCAP format we find something like this: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../payloadLength }"> <complexType>.....</complexType> </element> To make this unparse we really need the payloadLength element to carry both an IVC and an OVC. Otherwise when unparsing the dfdl:length expression on the payload element can't be meaningful. We have an erratum that states that length expressions are always re-evaluated when unparsing, so as to produce a target length that can be used to insert padding/fillByte. However, the spec currently says you cannot have both IVC and OVC on the same element. I do not recall why we have this restriction, other than trying to prohibit things we didn't think were necessary, but it seems in this situation to be needed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>
If I understand the example correctly, you have added 'payloadLength' to the infoset as a convenience to the user, but at the same time you are referencing it from the 'payload' element's dfdl:length property. Your example could be changed to use the 'length' element instead: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../length - 20 }"> <complexType>.....</complexType> </element> If we allow both input & output value calc on a element, it is saying that on unparsing the element's value in the infoset is ignored, and its value never appear in the data. So it's a place to hold a transient value, don't we have variables for that? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 19/07/2016 18:49 Subject: [DFDL-WG] Need dfdl:inputValueCalc and dfdl:outputValueCalc on same element Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> So in PCAP format we find something like this: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../payloadLength }"> <complexType>.....</complexType> </element> To make this unparse we really need the payloadLength element to carry both an IVC and an OVC. Otherwise when unparsing the dfdl:length expression on the payload element can't be meaningful. We have an erratum that states that length expressions are always re-evaluated when unparsing, so as to produce a target length that can be used to insert padding/fillByte. However, the spec currently says you cannot have both IVC and OVC on the same element. I do not recall why we have this restriction, other than trying to prohibit things we didn't think were necessary, but it seems in this situation to be needed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
So, the workaround of just repeating the expression is fine. Using a variable, well .... this whole structure of mine appears inside an array in the PCAP format. So using a variable requires newVariableInstance: <dfdl:defineVariable name="PL" type="xs:int"/> ... <sequence> <annotation><appinfo ...> <dfdl:newVariableInstance ref="ex:PL">{ ../length - 20 }</dfdl:newVariableInstance> </appinfo></annotation> <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <sequence><annotation><appinfo ...> <dfdl:setVariable ref="ex:PL">{ ../length - 20 }</dfdl:setVariable> </appinfo></annotation></sequence> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ $ex:PL }" /> <element name="payload" dfdl:length="{ $ex:PL }"> <complexType>.....</complexType> </element> </sequence> <!-- ends the newVariableInstance for PL --> As you point out the payloadLength element is really just there to exhibit this length as a feature of the infoset. The variable serves to carry the expression so that it needn't be repeated. Compare the above to my re-formulated proposed version allowing both IVC and OVC on the same element: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" dfdl:outputValueCalc="{ ../length - 20 }"/> <element name="payload" dfdl:length="{ ../payloadLength }"> <complexType>.....</complexType> </element> I'm still having to repeat the ../length - 20 expression twice, and the only way I can avoid that is by introducing a newVariableInstance to factor out the common subexpression which is not worth it in this case. (Or I suppose a new dfdl:valueCalc=... property which means to compute on both parse and unparse.) Whether the computed element appears in the infoset or not depends on whether it is hidden or not. The above also solves the array-variables problem (separate problem - separate email thread to discuss this) as if there is an array of these things, the path to a specific instance of payloadLength within that array is effectively an array variable. (and would be hidden if you don't want the array variable in the infoset. ) Honestly, had we considered IVC/OVC both on the same hidden element, I might have advocated to have this in DFDL as a way to avoid having variables and especially newVariableInstance altogether. The only drawback is that one cannot use them without a complex type being involved, which is an awkwardness we inherit from XML Schema which results in many elements named "value" that serve to carry a simple type within a complex type enclosure. I.e., <mySimpleElement><value>foo</value></mySimpleElement>. This has been discussed before, and we decided not to fix it for DFDL v1.0. (We will be experimenting with allowing a hidden group ref from a simple type definition as a way to fix this. ) Now, given all that discussion (and digression) ... I do have a workaround in this case: as you suggested, I can just repeat the expression: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../length - 20 }"> <complexType>.....</complexType> </element> So that's what I will do for now. Ultimately w.r.t. other needs that arise from IVC/OVC and unparsing, what I'd like to do is park this issue. As we flesh out more issues of IVC and OVC, we'll accumulate an experience document, a set of implementation techniques etc. Then we'll see what set of concrete proposals come out of this that need to be fixes to DFDL v1.0 as opposed to improvements for DFDL 2.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Jul 26, 2016 at 3:50 AM, Steve Hanson <smh@uk.ibm.com> wrote:
If I understand the example correctly, you have added 'payloadLength' to the infoset as a convenience to the user, but at the same time you are referencing it from the 'payload' element's dfdl:length property. Your example could be changed to use the 'length' element instead:
<element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../length - 20 }"> <complexType>.....</complexType> </element>
If we allow both input & output value calc on a element, it is saying that on unparsing the element's value in the infoset is ignored, and its value never appear in the data. So it's a place to hold a transient value, don't we have variables for that?
Regards
Steve Hanson *IBM Integration Bus* <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, Hursley, UK Architect, *IBM DFDL* <http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> *smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890
From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 19/07/2016 18:49 Subject: [DFDL-WG] Need dfdl:inputValueCalc and dfdl:outputValueCalc on same element Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> ------------------------------
So in PCAP format we find something like this:
<element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../payloadLength }"> <complexType>.....</complexType> </element>
To make this unparse we really need the payloadLength element to carry both an IVC and an OVC.
Otherwise when unparsing the dfdl:length expression on the payload element can't be meaningful. We have an erratum that states that length expressions are always re-evaluated when unparsing, so as to produce a target length that can be used to insert padding/fillByte.
However, the spec currently says you cannot have both IVC and OVC on the same element.
I do not recall why we have this restriction, other than trying to prohibit things we didn't think were necessary, but it seems in this situation to be needed.
Comments?
Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | *www.tresys.com* <http://www.tresys.com/> Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <http://www.ogf.org/About/abt_policies.php> -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
"Ultimately w.r.t. other needs that arise from IVC/OVC and unparsing, what I'd like to do is park this issue. As we flesh out more issues of IVC and OVC, we'll accumulate an experience document, a set of implementation techniques etc. Then we'll see what set of concrete proposals come out of this that need to be fixes to DFDL v1.0 as opposed to improvements for DFDL 2.0. " WG agreed this was the best way forward. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 26/07/2016 15:08 Subject: Re: [DFDL-WG] Need dfdl:inputValueCalc and dfdl:outputValueCalc on same element So, the workaround of just repeating the expression is fine. Using a variable, well .... this whole structure of mine appears inside an array in the PCAP format. So using a variable requires newVariableInstance: <dfdl:defineVariable name="PL" type="xs:int"/> ... <sequence> <annotation><appinfo ...> <dfdl:newVariableInstance ref="ex:PL">{ ../length - 20 }</dfdl:newVariableInstance> </appinfo></annotation> <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <sequence><annotation><appinfo ...> <dfdl:setVariable ref="ex:PL">{ ../length - 20 }</dfdl:setVariable> </appinfo></annotation></sequence> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ $ex:PL }" /> <element name="payload" dfdl:length="{ $ex:PL }"> <complexType>.....</complexType> </element> </sequence> <!-- ends the newVariableInstance for PL --> As you point out the payloadLength element is really just there to exhibit this length as a feature of the infoset. The variable serves to carry the expression so that it needn't be repeated. Compare the above to my re-formulated proposed version allowing both IVC and OVC on the same element: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" dfdl:outputValueCalc="{ ../length - 20 }"/> <element name="payload" dfdl:length="{ ../payloadLength }"> <complexType>.....</complexType> </element> I'm still having to repeat the ../length - 20 expression twice, and the only way I can avoid that is by introducing a newVariableInstance to factor out the common subexpression which is not worth it in this case. (Or I suppose a new dfdl:valueCalc=... property which means to compute on both parse and unparse.) Whether the computed element appears in the infoset or not depends on whether it is hidden or not. The above also solves the array-variables problem (separate problem - separate email thread to discuss this) as if there is an array of these things, the path to a specific instance of payloadLength within that array is effectively an array variable. (and would be hidden if you don't want the array variable in the infoset. ) Honestly, had we considered IVC/OVC both on the same hidden element, I might have advocated to have this in DFDL as a way to avoid having variables and especially newVariableInstance altogether. The only drawback is that one cannot use them without a complex type being involved, which is an awkwardness we inherit from XML Schema which results in many elements named "value" that serve to carry a simple type within a complex type enclosure. I.e., <mySimpleElement><value>foo</value></mySimpleElement>. This has been discussed before, and we decided not to fix it for DFDL v1.0. (We will be experimenting with allowing a hidden group ref from a simple type definition as a way to fix this. ) Now, given all that discussion (and digression) ... I do have a workaround in this case: as you suggested, I can just repeat the expression: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../length - 20 }"> <complexType>.....</complexType> </element> So that's what I will do for now. Ultimately w.r.t. other needs that arise from IVC/OVC and unparsing, what I'd like to do is park this issue. As we flesh out more issues of IVC and OVC, we'll accumulate an experience document, a set of implementation techniques etc. Then we'll see what set of concrete proposals come out of this that need to be fixes to DFDL v1.0 as opposed to improvements for DFDL 2.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Jul 26, 2016 at 3:50 AM, Steve Hanson <smh@uk.ibm.com> wrote: If I understand the example correctly, you have added 'payloadLength' to the infoset as a convenience to the user, but at the same time you are referencing it from the 'payload' element's dfdl:length property. Your example could be changed to use the 'length' element instead: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../length - 20 }"> <complexType>.....</complexType> </element> If we allow both input & output value calc on a element, it is saying that on unparsing the element's value in the infoset is ignored, and its value never appear in the data. So it's a place to hold a transient value, don't we have variables for that? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh@uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: "dfdl-wg@ogf.org" <dfdl-wg@ogf.org> Date: 19/07/2016 18:49 Subject: [DFDL-WG] Need dfdl:inputValueCalc and dfdl:outputValueCalc on same element Sent by: "dfdl-wg" <dfdl-wg-bounces@ogf.org> So in PCAP format we find something like this: <element name="length" type="xs:int" dfdl:outputValueCalc="{ dfdl:contentLength(../payload) + 20 }"> <element name="payloadLength" type="xs:int" dfdl:inputValueCalc="{ ../length - 20 }" /> <element name="payload" dfdl:length="{ ../payloadLength }"> <complexType>.....</complexType> </element> To make this unparse we really need the payloadLength element to carry both an IVC and an OVC. Otherwise when unparsing the dfdl:length expression on the payload element can't be meaningful. We have an erratum that states that length expressions are always re-evaluated when unparsing, so as to produce a target length that can be used to insert padding/fillByte. However, the spec currently says you cannot have both IVC and OVC on the same element. I do not recall why we have this restriction, other than trying to prohibit things we didn't think were necessary, but it seems in this situation to be needed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
participants (2)
-
Mike Beckerle -
Steve Hanson