- dfdl-wg - lists.ogf.org

Spec erratum in tables 18 and 20
by Steve Hanson 08 Jan '15

08 Jan '15

Correction to tables below - only xs:dateTime can have binaryCalendarRep 'binarySeconds' or 'binaryMilliseconds'. Type Length text binary String The XSD maxlength facet gives length in characters, but this is also the length in bytes. (See note below: character set encoding must be single-byte.) Multply by 8 to get number of bits. Not applicable Float Not allowed 32 bits Double Not allowed 64 bits Decimal, Integer, nonNegativeInteger Not allowed packed decimal: Not allowed binary: Not allowed Long, UnsignedLong Not allowed binary: 64 bits Int, UnsignedInt Not allowed binary: 32 bits Short, UnsignedShort Not allowed binary: 16 bits Byte, UnsignedByte Not allowed binary: 8 bits DateTime Not allowed binarySeconds: 32 bits, binaryMilliseconds: 64 bits. Date Not allowed binarySeconds, binaryMilliseconds: Not applicable Time Not allowed binarySeconds, binaryMilliseconds: Not applicable Boolean Length of longest of dfdl:textBooleanTrueRep and dfdl:textBooleanFalseRep values 32 bits HexBinary Not applicable The XSD maxLength facet gives the length in bytes. Multiply by 8 to convert to number of bits. Table 18 Length in Bits for SimpleTypes when dfdl:lengthKind is 'implicit' Type Length String Not applicable Float 32 Double 64 Decimal, Integer, NonNegativeInteger Compute the minimum number of bytes to represent the value (per dfdl:binaryNumberRep) and sign (if applicable). Multiply by 8 for number of bits. Long, UnsignedLong packed decimal: as Decimal binary: 64 Int, UnsignedInt binary: 32 Short, UnsignedShort binary: 16 Byte, UnsignedByte binary: 8 DateTime binarySeconds: 32, binaryMilliseconds:64 Date binarySeconds, binaryMilliseconds: Not applicable Time binarySeconds, binaryMilliseconds: Not applicable Boolean 32 HexBinary Compute the number of bytes in the infoset value padded to the value of the XSD minLength facet (which gives minimum length in bytes) using dfdl:fillByte if necessary. This gives the unparse length in bytes. Multiply by 8 for the number of bits. Table 20 Unparse Lengths (in Bits) for Binary Data with dfdl:lengthKind 'prefixed' Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2015-01-06
by Steve Hanson 06 Jan '15

06 Jan '15

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13382?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Section 7.3: lengthUnits 'bits' description not correct
by Steve Hanson 06 Jan '15

06 Jan '15

Issue tracker created https://redmine.ogf.org/issues/253 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 05/01/2015 17:16 Subject: Section 7.3: lengthUnits 'bits' description not correct The original public comment erratum 4.1 allowed lengthUnits 'bits' to be used with packed calendars. While this has been correctly reflected in section 12.3.7.2, the description of the lengthUnits property in 12.3 was not updated. Should say: Usage is restricted as follows: �h 'characters' may only be used for complex elements and simple elements with text representation. �h 'bits' may only be used for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt and xs:unsignedLong simple types with binary representation, and for calendar simple types with binary packed representation. �h 'bytes' must be used for type xs:hexBinary. �h 'bytes' must be used for types xs:float and xs:double with binary representation Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Erratum 2.100 (was Action 183 - chicken-and-egg situation with lengths given by expressions)
by Steve Hanson 06 Jan '15

06 Jan '15

A thought. We could add dfdl:lengthKind 'expression', use dfdl:length for the expression, and add a new enum property that is the policy for the expression - that is, evaluate on parse only, or evaluate on both parse and unparse. The enum for the latter is the equivalent of the pre-erratum 2.100 'explicit' and expression behaviour as implemented by IBM DFDL today - which we would then deprecate in the spec (or even remove) and deprecate in IBM DFDL. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 10/12/2014 09:08 Subject: Re: [DFDL-WG] Erratum 2.100 (was Action 183 - chicken-and-egg situation with lengths given by expressions) Tim & Mike - replies in-line. Concensus is heading towards 'explicit' working as per IBM DFDL today, so we need to decide on the best way to express the 2.100 behaviour. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org Date: 09/12/2014 23:25 Subject: Re: [DFDL-WG] Erratum 2.100 (was Action 183 - chicken-and-egg situation with lengths given by expressions) Sent by: dfdl-wg-bounces(a)ogf.org My 2 cents: lengthKind 'explicit' should continue to work as it currently does in the IBM implementation. That is, it should use the static or calculated value of the length, and pad to that length when unparsing. Otherwise we have different rules for lengthKind='explicit' depending on whether it is an expression or a static value. If somebody wants to avoid padding, then they can put an outputValueCalc on the length field and calculate a value that requires no padding. I think the required expression would be {dfdl:valueLength(../variableLengthField)} With these rules there is a potential for a mutual dependency ( deadlock ) when outputValueCalc is used with a calculated length. If the outputValueCalc is {dfdl:contentLength(../variableLengthField)} then the entire representation length ( including the padding characters ) is being requested. This obviously cannot be satisfied until the padding has been applied, but the padding cannot be applied until the length is known...etc. I don't think this should affect the decision about lengthKind though, for two reasons: - it's already possible to create deadlocks using other calculated fields in DFDL. Any robust implementation of a serializer must include deadlock detection in the evaluation of DFDL expressions. - it would not be difficult to detect the simple case statically and report a schema definition error. SMH: Corrected function names to match spec. The only downside of this is that users who want to calculate the output length for themselves have to jump through the non-obvious hoop of adding outputValueCalc to the length field and selecting the correct (non-deadlocking) DFDL function. But for fields have representation='text' and lengthUnits='characters' there is a simpler alternative: set the length expression to {fn:length(.)}. Other cases will be derivable from the infoset value by a simple calculation, so the outputValueCalc solution should only be required for the hard cases. SMH: Your fn:length(.) does not work unfortunately. The expression does not give the desired length when parsing. regards, Tim Kimber From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/12/2014 18:06 Subject: Re: [DFDL-WG] Erratum 2.100 (was Action 183 - chicken-and-egg situation with lengths given by expressions) Sent by: dfdl-wg-bounces(a)ogf.org Idea 1: Just provide an infoset member [contentLength] which would allow one to get and (when unparsing) set the content length. This would provide the "sticky" content length memory that people are seeking when they say they want data to round-trip. If the same infoset object is parsed and then unparsed, the existence of this [contentLength] member provides the memory of the content length from the parse. An outputValueCalc could meaningfully ask for dfdl:contentLength() which would return the value of this member. If unset, then when unparsing the value of this member can be defined to be the dfdl:valueLength() of the infoset value. SMH; Not keen on this. It makes the DFDL infoset deviate too much from the XML infoset. Idea 2: Per our discussion on the call, I think we are trying to get two behaviors into the same length kind, and we can't have it both ways. So I am fine if we say lengthKind 'explicit' evaluates the expression both parsing and unparsing. I would suggest new lengthKind 'explicitParseOnly' evaluates the expression only when parsing. When this is used, an outputValueCalc would be needed to compute the length value and unparse the representation of it. That calculation would likely need to refer to the dfdl:valueLength() of the element whose length needs to be stored. SMH: Not keen on the name. Let's keep thinking! Idea 3: An alternative would be to introduce property dfdl:unparseContentLength which is an integer constant, or an expression, or one of three special enum values. When dfdl:unparseContentLength='useLength' then it would be the same value as the length (same constant, or computed by evaluating the same expression). Or if dfdl:unparseContentLength='useValueLength' would mean that the length is equivalent to writing dfdl:unparseContentLength='{ dfdl:valueLength(., LU) }', where LU is the length units of the element that is the first argument). Nothing would be automatically saving out the explicitly computed length, so an outputValueCalc would need to be able to get at the dfdl:valueLength or dfdl:contentLength - which would return the value of the dfdl:unparseContentLength) A thought: perhaps we really only need the enum values, and not the ability to specify constants or expressions? In that case I'd suggest the property have the "policy" suffix on the name, i.e., dfdl:unparseContentLengthPolicy. Other possible name choices for this property: dfdl:explicitLengthKindUnparsePolicy or dfdl:lengthKindExplicitUnparsePolicy. I guess we get our choice of whether we add new length kinds along with 'explicit' and we define lengthKind 'explicit' to be one of these behaviors, or we add this enum policy property to add additional meaning when the lengthKind is 'explicit'. The difference of the two is taste and style I think. SMH: I prefer a new enum for lengthKind. That's the point of the property. It's much easier to say in the spec '...when dfdl:lengthKind is 'prefixed' or 'expression'...' than '...when dfdl:lengthKind is 'prefixed', or 'explicit' and dfdl:length is an expression and dfdl:unparseContentLength is 'xxx' ...'. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Mon, Dec 8, 2014 at 11:53 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: This is the email thread that discussed the behaviour of lengthKind 'explicit' where length is an expression, when unparsing. Erratum 2.100 had been raised and stated that this is an example of a variable length, and behaves like lengthKind 'prefixed'. The discussion below was around whether this captured all the use cases and should the dfdl:length expression be evaluated during unparsing in order to obtain a specified length to which the value could be padded (or truncated) in the same way as a fixed length. The conclusion was that the expression should not be evaluated. This is consistent with occursCount - the expression is not evaluated on unparsing. (There was a public comment on this area but it was just observing that the spec had not been fully updated to reflect 2.100) The use cases are numbered in the discussion below 1) to 4), with examples . Updated to reflect function name changes. The original proposal is also below, which was not adopted. However, an issue is that IBM DFDL already implements use case 1) by evaluating the expression on unparsing. I discovered this when I came to change the code to ensure 2.100 behaviour was followed. It is therefore likely that there are users who are relying on this behaviour in IBM DFDL. Coincidentally the same day I was asked by a user about the following use case. User has BCD data with length prefix. He wants to parse the data, do some processing then unparse the data, but it is not acceptable for the length of the data to change. Specifically, the BCD may start with 0s. When this is parsed as a decimal or integer, it has the effect of losing leading 0s so they need to be added back during unparsing. This is not possible with lengthKind 'prefixed' and nor is it possible with lengthKind 'expression' with erratum 2.100. User is currently treating the element as a 'prefixed' hexBinary blob to preserve the leading 0s. While this works for his use case, I am worried that the blob solution won't always be appropriate. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 08/12/2014 15:53 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 08/01/2013 18:06 Subject: Fw: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Did not adopt the proposal below, and errata 2.100 remains as documented. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 08/01/2013 18:05 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 08/01/2013 14:44 Subject: Re: Fw: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions If we had the following we would have three behaviours which I think would cover the identified use cases. a) dfdl:lengthKind = "explicit", dfdl:length is an integer => length is specified on parsing & unparsing b) dfdl:lengthKind = "explicit", dfdl:length is an expression => length is specified on parsing & unparsing (expression must evaluate) - use cases 1) & 2) below - Mike's scenario where length is specified externally c) dfdl:lengthKind = "expression" (*new*), dfdl:length must be an expression => length is specified on parsing, but variable on unparsing (expression not evaluated) - new enum - use case 3) below - gives the erratum 2.100 behaviour - matches occursCountKind 'expression' behaviour Alternatively we leave errata 2.100 as it is and for Mike's scenario the application must do the padding before it passes the infoset values to the unparser. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 11/12/2012 10:36 Subject: Fw: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions The email thread which provided the material for the discussion which led to: 2.100. Section 12.3.1. State that when unparsing an element with lengthKind ‘explicit’ and where length is an expression, then the data in the Infoset is treated as variable length and not fixed length. The behaviour is the same as lengthKind ‘prefixed’. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/11/2012 16:04 ----- From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/09/2012 12:39 Subject: Re: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Sent by: dfdl-wg-bounces(a)ogf.org Good point. The problem is that lengthKind-'explicit' is being used for two things: a) a length that is static b) a length that is calculated ...so the DFDL serializer must assume that the expression needs to be evaluated. For occursCountKind we have separate values for 'fixed' and 'expression'. If we did not, then occursCountKind would have the same problem except that it would affect defaulting rather than padding. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/09/2012 12:27 Subject: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Sent by: dfdl-wg-bounces(a)ogf.org This mail is on the expected behaviour of the DFDL unparser when writing out a 'data' element the length of which is held in an earlier 'len' element. There are several scenarios, some straightforward and some that exhibit a chicken-and-egg behaviour. The principle of what happens is understood, the action is to make sure that the behaviour is explained in enough detail in the spec to enable implementations to be consistent. (Note - IBM DFDL does not yet support outputVaueCalc so has not hit this yet). Scenarios follow. The 'data' element shown is simple, but the same principles apply if it is complex. 1) 'len' is set from infoset - 'len' can be set in augmented infoset - No issue as 'data's length expression may be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 2) 'len' is set using outputValueCalc with fixed expression - When 'len's outputValueCalc is encountered, it can be evaluated then and there - 'len' can be set in augmented infoset - No issue as 'data's length expression may be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{10}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 3) 'len' is set using outputValueCalc with reference 'data' (unpadded) - When 'len's outputValueCalc is encountered, it can not yet be evaluated as it depends on the length of 'data' - 'len' can not yet be set in augmented infoset - Problem as 'data's length expression can not be evaluated - But we do know the unpadded length of 'data' so 'len's outputValueCalc can now be evaluated - In turn this means that 'data's length expression can now be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{dfdl:valueLength(/message1/data)}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 4) 'len' is set using outputValueCalc with reference 'data' (padded) - When 'len's outputValueCalc is encountered, it can not yet be evaluated as it depends on the length of 'data' - 'len' can not yet be set in augmented infoset - Problem as 'data's length expression can not be evaluated - We don't know the padded length of 'data' because we don't know 'len' - Problem: 'data's length expression can never be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{dfdl:contentLength(/message1/data)}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 04/09/2012 17:31 Subject: Fw: Behaviour for lengthKind 'endOfparent' is still not fully specified DFDL WG call 4th Sept 2012: 1) Agreed that for binary data, only xs:hexBinary and packed/BCD allowed to have endOfParent 2) Agreed this is the correct behaviour when filling to a known length 3) Agreed this is the correct behaviour when filling to a known length 4) Agreed this is the correct behaviour when filling to a known length It was noted that lengthKind 'explicit' on the parent may not result in a known length if the length is an expression. This is an example of a more general chicken-and-egg situation with lengths given by expressions, for which outputValueCalc and DFDL functions unpaddedLength() were added can be used. Action raised to ensure that the behaviour of an implementation is fully defined by the spec. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 04/09/2012 17:24 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org <dfdl-wg(a)ogf.org> Date: 04/09/2012 14:08 Subject: Behaviour for lengthKind 'endOfparent' is still not fully specified Noted when I reviewed latest spec - endOfParent and unparsing is not fully thought through. The spec today says that I can use endOfParent with binary data. There is a restriction in section 12.3.8, but it only applies when an element is endOfParent and its parent is lengthKind delimited. There are a couple of cases to consider: 1) Binary data of restricted length (see list in other email "proposed clarification/narrowing - delimited binary data should decimal"). I don't think it makes sense to allow these. We don't allow these binary reps for delimited. 2) Text data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should pad to box length with appropriate padChar, according to justification, as that is effectively a 'specified length'. Error if textPadKind is 'none'. Use parent's lengthUnits. 3) HexBinary data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should right-pad to box length with fill byte, as that is effectively a 'specified length'. 4) Packed/BCD binary data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should pad to box length with zero bytes, according to justification, as that is effectively a 'specified length'. (Must be zero bytes and not fill byte as must be numeric in order to be parsed). In relation to 2 - 4, note that lengthKind 'endOfParent' can only be used with a parent lengthKind of 'explicit', 'pattern', 'prefixed' or 'endOfParent' or a choice with choiceLengthKind 'explicit', so the box scenario when unparsing therefore occurs only when lengthKind is 'explicit' or choiceLengthKind is 'explicit' - these are the cases when the length is known. Also note that when there are nested 'endOfParent' elements (which is allowed) then all padding must be done on the simple element (ie, the innermost element), to ensure that what is output can be parsed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Section 7.3: lengthUnits 'bits' description not correct
by Steve Hanson 05 Jan '15

05 Jan '15

The original public comment erratum 4.1 allowed lengthUnits 'bits' to be used with packed calendars. While this has been correctly reflected in section 12.3.7.2, the description of the lengthUnits property in 12.3 was not updated. Should say: Usage is restricted as follows: �h 'characters' may only be used for complex elements and simple elements with text representation. �h 'bits' may only be used for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt and xs:unsignedLong simple types with binary representation, and for calendar simple types with binary packed representation. �h 'bytes' must be used for type xs:hexBinary. �h 'bytes' must be used for types xs:float and xs:double with binary representation Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2015-01-06
by Steve Hanson 05 Jan '15

05 Jan '15

Happy New Year. Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13382?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Binary decimal virtual point - unparsing behaviour
by Steve Hanson 05 Jan '15

05 Jan '15

Agree. Tracker updated. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 10/12/2014 21:12 Subject: Re: [DFDL-WG] Binary decimal virtual point - unparsing behaviour (a) is more conservative, so I tend to want to go with that. We have rounding functions in our DFDL expression language. We know we're already on a design point here where the application will have to implement truncation/rounding logic after any arithmetic operation that can mess with the number of fraction digits that will be expressed. You'd like to think that would be only multiplication/division - but I'm not sure. Much depends on what is happening with xs:decimal values in the application, and also depends on what the representation of xs:decimal values is, in the application that is manipulating them. The simplest way to see the issue is imagine the application is taking decimal numbers, converting them to float, operating on float, and then storing the result back into a decimal value of the DFDL infoset. Consider 1.10 decimal. If you convert to float and back, you might get 1.099609375. This looks like it has a lot more precision, but in fact it is the closest decimal value to the binary float value closest to 1.1 (decimal). An application has to implement base 10 rounding or truncation on the fractional part in order to insure that the behavior makes sense. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Dec 10, 2014 at 2:21 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: Sorry, but I need a clarification on this. Spec says ""When unparsing a binary number (packed decimal or twos-complement) and excess precision is supplied in the Infoset no rounding occurs. It is a processing error."" Technically, precision is the number of digits, and trailing zeros are significant even after a decimal point. So is it a processing error if I supply an infoset decimal value of 123. 990 and a BDVP of 2? While this can be serialized to an integer, the original decimal has excess precision. Similarly for an infoset decimal value of 12300.0 and a BDVP of -2. This can be serialized to an integer, but the original decimal has excess precision. What this amounts to is whether we give a processing error a) whenever the infoset supplies excess precision, or b) only when the excess precision results in a decimal that can't be converted into an integer. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/12/2014 17:45 Subject: Re: Binary decimal virtual point - unparsing behaviour DFDL WG agreed that this is a processing error. http://redmine.ogf.org/issues/249 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 05/12/2014 11:05 Subject: Binary decimal virtual point - unparsing behaviour I am not finding anywhere in the DFDL spec that says explicitly what happens during unparsing when a decimal value is given to the unparser, and its rep is binary, and the BDVP is not sufficient to remove the decimal point from the value. This is surely a processing error, because the spec states in section 13.7.1 "When unparsing a binary number (packed decimal or twos-complement) and excess precision is supplied in the Infoset no rounding occurs. It is a processing error." Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

when is the separator expression evaluated?
by Mike Beckerle 19 Dec '14

19 Dec '14

Jessie Chab came up with this interesting case. I am hoping someone else remembers somewhere in the spec where this order of evaluation issue is taken up in detail. Consider: <element name="e2"> <sequence separator="|" separatorPosition="infix"> <element name="seps" minOccurs="3" maxOccurs="3"/> <element name="data" maxOccurs='10'> <sequence separator="{ /e2/seps[dfdl:occursIndex()] }"> <element name="num" maxOccurs='10' /> </sequence> </element> </sequence> </element> So we first parse 3 strings separated by a pipe. After that's parsed, lets assume our infoset looks like this: <e2> <seps>;</seps> <seps>-</seps> <seps>#</seps> </e2> After that we will have some 'data' elements (separated by pipes) which each have a sequence of 'num' elements. The question is what are the valid separators of the 'num' elements. I see two potential interpretations. 1) The separator is evaluated once per 'data' element; occursIndex evaluates to index in the 'data' array; valid data might look something like: ;|-|#|a;b;c;d|e-f-g-h|i#j#k#l Note that this means the size of the data array must be less than or equal to the size of the seps array (though that could be worked around using mod 3 arithmetic.) 2) Everytime we need to look for a separator between a num element, we reevaluate the separator expression. This means the occursIndex() references the index in the 'num' array, and so valid data might look something like: ;|-|#|a;b-c#d|e;f-g#h|i;j-k#l Note that this means the size of the num array must be less than or equal to the size of the seps array. I recall we were considering an argument to dfdl:occursIndex() to make exactly this kind of issue clear. I believe we decided against it, as we weren't able to pin down the semantics quite clearly. E.g., in the above, how would you add an argument to the dfdl:occursIndex(...) call that points to the num array, which isn't even in scope at that point? I know we say somewhere in the spec that separator can be defined, in say, the default format of some other schema file. It can be an expression, and that expression isn't evaluated until some sequence which has that separator in scope. Which means the expression can refer to path steps and such that are meaningless at the point where it appears lexically, but will be meaningful for a sequence where that separator expression is in scope. But this problem is slightly different. The question is whether the evaluation is per-item of the sequence, or once for the sequence. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 4

Re: [DFDL-WG] Binary decimal virtual point - unparsing behaviour
by Steve Hanson 10 Dec '14

10 Dec '14

Sorry, but I need a clarification on this. Spec says ""When unparsing a binary number (packed decimal or twos-complement) and excess precision is supplied in the Infoset no rounding occurs. It is a processing error."" Technically, precision is the number of digits, and trailing zeros are significant even after a decimal point. So is it a processing error if I supply an infoset decimal value of 123. 990 and a BDVP of 2? While this can be serialized to an integer, the original decimal has excess precision. Similarly for an infoset decimal value of 12300.0 and a BDVP of -2. This can be serialized to an integer, but the original decimal has excess precision. What this amounts to is whether we give a processing error a) whenever the infoset supplies excess precision, or b) only when the excess precision results in a decimal that can't be converted into an integer. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/12/2014 17:45 Subject: Re: Binary decimal virtual point - unparsing behaviour DFDL WG agreed that this is a processing error. http://redmine.ogf.org/issues/249 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 05/12/2014 11:05 Subject: Binary decimal virtual point - unparsing behaviour I am not finding anywhere in the DFDL spec that says explicitly what happens during unparsing when a decimal value is given to the unparser, and its rep is binary, and the BDVP is not sufficient to remove the decimal point from the value. This is surely a processing error, because the spec states in section 13.7.1 "When unparsing a binary number (packed decimal or twos-complement) and excess precision is supplied in the Infoset no rounding occurs. It is a processing error." Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Erratum 2.100 (was Action 183 - chicken-and-egg situation with lengths given by expressions)
by Mike Beckerle 10 Dec '14

10 Dec '14

Idea 1: Just provide an infoset member [contentLength] which would allow one to get and (when unparsing) set the content length. This would provide the "sticky" content length memory that people are seeking when they say they want data to round-trip. If the same infoset object is parsed and then unparsed, the existence of this [contentLength] member provides the memory of the content length from the parse. An outputValueCalc could meaningfully ask for dfdl:contentLength() which would return the value of this member. If unset, then when unparsing the value of this member can be defined to be the dfdl:valueLength() of the infoset value. Idea 2: Per our discussion on the call, I think we are trying to get two behaviors into the same length kind, and we can't have it both ways. So I am fine if we say lengthKind 'explicit' evaluates the expression both parsing and unparsing. I would suggest new lengthKind 'explicitParseOnly' evaluates the expression only when parsing. When this is used, an outputValueCalc would be needed to compute the length value and unparse the representation of it. That calculation would likely need to refer to the dfdl:valueLength() of the element whose length needs to be stored. Idea 3: An alternative would be to introduce property dfdl:unparseContentLength which is an integer constant, or an expression, or one of three special enum values. When dfdl:unparseContentLength='useLength' then it would be the same value as the length (same constant, or computed by evaluating the same expression). Or if dfdl:unparseContentLength='useValueLength' would mean that the length is equivalent to writing dfdl:unparseContentLength='{ dfdl:valueLength(., LU) }', where LU is the length units of the element that is the first argument). Nothing would be automatically saving out the explicitly computed length, so an outputValueCalc would need to be able to get at the dfdl:valueLength or dfdl:contentLength - which would return the value of the dfdl:unparseContentLength) A thought: perhaps we really only need the enum values, and not the ability to specify constants or expressions? In that case I'd suggest the property have the "policy" suffix on the name, i.e., dfdl:unparseContentLengthPolicy. Other possible name choices for this property: dfdl:explicitLengthKindUnparsePolicy or dfdl:lengthKindExplicitUnparsePolicy. I guess we get our choice of whether we add new length kinds along with 'explicit' and we define lengthKind 'explicit' to be one of these behaviors, or we add this enum policy property to add additional meaning when the lengthKind is 'explicit'. The difference of the two is taste and style I think. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Mon, Dec 8, 2014 at 11:53 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > This is the email thread that discussed the behaviour of lengthKind > 'explicit' where length is an expression, when unparsing. > > Erratum 2.100 had been raised and stated that this is an example of a > variable length, and behaves like lengthKind 'prefixed'. The discussion > below was around whether this captured all the use cases and should the > dfdl:length expression be evaluated during unparsing in order to obtain a > specified length to which the value could be padded (or truncated) in the > same way as a fixed length. The conclusion was that the expression should > not be evaluated. This is consistent with occursCount - the expression is > not evaluated on unparsing. > > (There was a public comment on this area but it was just observing that > the spec had not been fully updated to reflect 2.100) > > The use cases are numbered in the discussion below 1) to 4), with examples. > Updated to reflect function name changes. > > The original proposal is also below, which was not adopted. > > However, an issue is that IBM DFDL already implements use case 1) by > evaluating the expression on unparsing. I discovered this when I came to > change the code to ensure 2.100 behaviour was followed. It is therefore > likely that there are users who are relying on this behaviour in IBM DFDL. > > Coincidentally the same day I was asked by a user about the following use > case. User has BCD data with length prefix. He wants to parse the data, do > some processing then unparse the data, but it is not acceptable for the > length of the data to change. Specifically, the BCD may start with 0s. When > this is parsed as a decimal or integer, it has the effect of losing leading > 0s so they need to be added back during unparsing. This is not possible > with lengthKind 'prefixed' and nor is it possible with lengthKind > 'expression' with erratum 2.100. User is currently treating the element as > a 'prefixed' hexBinary blob to preserve the leading 0s. While this works > for his use case, I am worried that the blob solution won't always be > appropriate. > > Regards > > Steve Hanson > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 08/12/2014 15:53 ----- > > From: Steve Hanson/UK/IBM > To: dfdl-wg(a)ogf.org > Date: 08/01/2013 18:06 > Subject: Fw: [DFDL-WG] Action 183 - chicken-and-egg situation with > lengths given by expressions > ------------------------------ > > > Did not adopt the proposal below, and errata 2.100 remains as documented. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 08/01/2013 18:05 ----- > > From: Steve Hanson/UK/IBM > To: dfdl-wg(a)ogf.org, > Date: 08/01/2013 14:44 > Subject: Re: Fw: [DFDL-WG] Action 183 - chicken-and-egg situation > with lengths given by expressions > ------------------------------ > > > If we had the following we would have three behaviours which I think would > cover the identified use cases. > > a) dfdl:lengthKind = "explicit", dfdl:length is an integer => length is > specified on parsing & unparsing > > b) dfdl:lengthKind = "explicit", dfdl:length is an expression => length is > specified on parsing & unparsing (expression must evaluate) > - use cases 1) & 2) below > - Mike's scenario where length is specified externally > > c) dfdl:lengthKind = "expression" (*new*), dfdl:length must be an > expression => length is specified on parsing, but variable on unparsing > (expression not evaluated) > - new enum > - use case 3) below > - gives the erratum 2.100 behaviour > - matches occursCountKind 'expression' behaviour > > Alternatively we leave errata 2.100 as it is and for Mike's scenario the > application must do the padding before it passes the infoset values to the > unparser. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Steve Hanson/UK/IBM > To: dfdl-wg(a)ogf.org, > Date: 11/12/2012 10:36 > Subject: Fw: [DFDL-WG] Action 183 - chicken-and-egg situation with > lengths given by expressions > ------------------------------ > > > The email thread which provided the material for the discussion which led > to: > > *2.100**. Section 12.3.1. State that when unparsing an element with > lengthKind ‘explicit’ and where length is an expression, then the data in > the Infoset is treated as variable length and not fixed length. The > behaviour is the same as lengthKind ‘prefixed’. * > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 16/11/2012 16:04 ----- > > From: Tim Kimber/UK/IBM@IBMGB > To: dfdl-wg(a)ogf.org, > Date: 11/09/2012 12:39 > Subject: Re: [DFDL-WG] Action 183 - chicken-and-egg situation with > lengths given by expressions > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > Good point. The problem is that lengthKind-'explicit' is being used for > two things: > a) a length that is static > b) a length that is calculated > ...so the DFDL serializer must assume that the expression needs to be > evaluated. > > For occursCountKind we have separate values for 'fixed' and 'expression'. > If we did not, then occursCountKind would have the same problem except that > it would affect defaulting rather than padding. > > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: kimbert(a)uk.ibm.com > Tel. 01962-816742 > Internal tel. 37246742 > > > > > From: Steve Hanson/UK/IBM@IBMGB > To: dfdl-wg(a)ogf.org, > Date: 11/09/2012 12:27 > Subject: [DFDL-WG] Action 183 - chicken-and-egg situation with > lengths given by expressions > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > This mail is on the expected behaviour of the DFDL unparser when writing > out a 'data' element the length of which is held in an earlier 'len' > element. > > There are several scenarios, some straightforward and some that exhibit a > chicken-and-egg behaviour. The principle of what happens is understood, > the action is to make sure that the behaviour is explained in enough detail > in the spec to enable implementations to be consistent. (Note - IBM DFDL > does not yet support outputVaueCalc so has not hit this yet). > > Scenarios follow. The 'data' element shown is simple, but the same > principles apply if it is complex. > > 1) 'len' is set from infoset > > - 'len' can be set in augmented infoset > - No issue as 'data's length expression may be evaluated > > <xsd:element name="message1"> > <xsd:complexType> > <xsd:sequence> > <xsd:element name="len" type="xsd:int" > dfdl:lengthKind="explicit" dfdl:length="2" /> > <xsd:element name="data" type="xsd:string" > dfdl:length="{/message1/len}" > dfdl:lengthKind="explicit" /> > </xsd:sequence> > </xsd:complexType> > </xsd:element> > > 2) 'len' is set using outputValueCalc with fixed expression > > - When 'len's outputValueCalc is encountered, it can be evaluated then and > there > - 'len' can be set in augmented infoset > - No issue as 'data's length expression may be evaluated > > <xsd:element name="message1"> > <xsd:complexType> > <xsd:sequence> > <xsd:element name="len" type="xsd:int" > dfdl:outputValueCalc="{10}" dfdl:lengthKind="explicit" > dfdl:length="2" /> > <xsd:element name="data" type="xsd:string" > dfdl:length="{/message1/len}" > dfdl:lengthKind="explicit" /> > </xsd:sequence> > </xsd:complexType> > </xsd:element> > > 3) 'len' is set using outputValueCalc with reference 'data' (unpadded) > > - When 'len's outputValueCalc is encountered, it can not yet be evaluated > as it depends on the length of 'data' > - 'len' can not yet be set in augmented infoset > - Problem as 'data's length expression can not be evaluated > - But we do know the unpadded length of 'data' so 'len's outputValueCalc > can now be evaluated > - In turn this means that 'data's length expression can now be evaluated > > <xsd:element name="message1"> > <xsd:complexType> > <xsd:sequence> > <xsd:element name="len" type="xsd:int" > > dfdl:outputValueCalc="{dfdl:valueLength(/message1/data)}" > dfdl:lengthKind="explicit" dfdl:length="2" /> > <xsd:element name="data" type="xsd:string" > dfdl:length="{/message1/len}" > dfdl:lengthKind="explicit" /> > </xsd:sequence> > </xsd:complexType> > </xsd:element> > > 4) 'len' is set using outputValueCalc with reference 'data' (padded) > > - When 'len's outputValueCalc is encountered, it can not yet be evaluated > as it depends on the length of 'data' > - 'len' can not yet be set in augmented infoset > - Problem as 'data's length expression can not be evaluated > - We don't know the padded length of 'data' because we don't know 'len' > - *Problem*: 'data's length expression can never be evaluated > > <xsd:element name="message1"> > <xsd:complexType> > <xsd:sequence> > <xsd:element name="len" type="xsd:int" > > dfdl:outputValueCalc="{dfdl:contentLength(/message1/data)}" > dfdl:lengthKind="explicit" dfdl:length="2" /> > <xsd:element name="data" type="xsd:string" > dfdl:length="{/message1/len}" > dfdl:lengthKind="explicit" /> > </xsd:sequence> > </xsd:complexType> > </xsd:element> > > > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Steve Hanson/UK/IBM > To: dfdl-wg(a)ogf.org > Date: 04/09/2012 17:31 > Subject: Fw: Behaviour for lengthKind 'endOfparent' is still not > fully specified > ------------------------------ > > > DFDL WG call 4th Sept 2012: > > 1) Agreed that for binary data, only xs:hexBinary and packed/BCD allowed > to have endOfParent > > 2) Agreed this is the correct behaviour when filling to a known length > > 3) Agreed this is the correct behaviour when filling to a known length > > 4) Agreed this is the correct behaviour when filling to a known length > > It was noted that lengthKind 'explicit' on the parent may not result in a > known length if the length is an expression. This is an example of a more > general chicken-and-egg situation with lengths given by expressions, for > which outputValueCalc and DFDL functions unpaddedLength() were added can be > used. Action raised to ensure that the behaviour of an implementation is > fully defined by the spec. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 04/09/2012 17:24 ----- > > From: Steve Hanson/UK/IBM > To: dfdl-wg(a)ogf.org <dfdl-wg(a)ogf.org> > Date: 04/09/2012 14:08 > Subject: Behaviour for lengthKind 'endOfparent' is still not fully > specified > ------------------------------ > > > Noted when I reviewed latest spec - endOfParent and unparsing is not fully > thought through. > > The spec today says that I can use endOfParent with binary data. There is > a restriction in section 12.3.8, but it only applies when an element is > endOfParent *and* its parent is lengthKind delimited. > > There are a couple of cases to consider: > > 1) Binary data of restricted length (see list in other email "proposed > clarification/narrowing - delimited binary data should decimal"). I don't > think it makes sense to allow these. We don't allow these binary reps for > delimited. > > 2) Text data of variable length when unparsing. Box scenario. If the data > in the infoset is shorter than the space in the box, what we do? I think > we should pad to box length with appropriate padChar, according to > justification, as that is effectively a 'specified length'. Error if > textPadKind is 'none'. Use parent's lengthUnits. > > 3) HexBinary data of variable length when unparsing. Box scenario. If the > data in the infoset is shorter than the space in the box, what we do? I > think we should right-pad to box length with fill byte, as that is > effectively a 'specified length'. > > 4) Packed/BCD binary data of variable length when unparsing. Box scenario. > If the data in the infoset is shorter than the space in the box, what we > do? I think we should pad to box length with zero bytes, according to > justification, as that is effectively a 'specified length'. (Must be zero > bytes and not fill byte as must be numeric in order to be parsed). > > In relation to 2 - 4, note that lengthKind 'endOfParent' can only be used > with a parent lengthKind of 'explicit', 'pattern', 'prefixed' or > 'endOfParent' or a choice with choiceLengthKind 'explicit', so the box > scenario when unparsing therefore occurs only when lengthKind is 'explicit' > or choiceLengthKind is 'explicit' - these are the cases when the length is > known. Also note that when there are nested 'endOfParent' elements (which > is allowed) then all padding must be done on the simple element (ie, the > innermost element), to ensure that what is output can be parsed. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > *https://www.ogf.org/mailman/listinfo/dfdl-wg* > <https://www.ogf.org/mailman/listinfo/dfdl-wg> > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg >

3 2