- dfdl-wg - lists.ogf.org

Action 185: Text number rounding and ICU
by Steve Hanson 11 Dec '12

11 Dec '12

As described in action 185, the DFDL spec behaviour for the DFDL properties that control text number rounding was derived from the documented behaviour for ICU4J. However the documentation is not correct and the ICU4J behaviour is as described for ICU4C. An ICU ticket was raised to clarify the position, and the intended behaviour is the observed behaviour. That affects the DFDL text number rounding properties as follows: textNumberRounding Enum Specifies how rounding is controlled during unparsing. Valid values ‘pattern', 'explicit' When dfdl:textNumberRep is 'standard' this property only applies when dfdl:textStandardBase is 10. If 'pattern' then rounding takes place according to the pattern. A rounding increment may be specified in the dfdl:textNumberPattern using digits '1' though '9', otherwise rounding is to the width of the pattern. The rounding mode is always 'roundHalfEven'. If 'explicit' then the rounding increment is specified by the dfdl:textNumberRoundingIncrement property, and any digits '1' through '9' in the dfdl:textNumberPattern are treated as digit '0'. The rounding mode is specified by the dfdl:textRoundingMode property. To disable rounding, use 'explicit' in conjunction with 'roundUnnecessary' for the dfdl:textNumberRoundingMode. If rounding is disabled then any excess precision is treated as a processing error. Annotation: dfdl:element, dfdl:simpleType textNumberRoundingMode Enum Specifies how rounding occurs during unparsing, when dfdl:textNumberRounding is 'explicit'. When dfdl:textNumberRep is 'standard' this property only applies when dfdl:textStandardBase is 10. To switch off rounding, use 'roundUnnecessary'. Valid values ‘roundCeiling’, ‘roundFloor’, ‘roundDown’, ‘roundUp’, ‘roundHalfEven’, ‘roundHalfDown’, ‘roundHalfUp', 'roundUnnecessary' Annotation: dfdl:element, dfdl:simpleType textNumberRoundingIncrement Double Specifies the rounding increment to use during unparsing, when dfdl:textNumberRounding is 'explicit'. When dfdl:textNumberRep is 'standard' this property only applies when dfdl:textStandardBase is 10. To switch off rounding, use 0.0. A negative value is a schema definition error. Annotation: dfdl:element, dfdl:simpleType Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Action 183 - chicken-and-egg situation with lengths given by expressions
by Steve Hanson 11 Dec '12

11 Dec '12

The email thread which provided the material for the discussion which led to: 2.100. Section 12.3.1. State that when unparsing an element with lengthKind ‘explicit’ and where length is an expression, then the data in the Infoset is treated as variable length and not fixed length. The behaviour is the same as lengthKind ‘prefixed’. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 16/11/2012 16:04 ----- From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/09/2012 12:39 Subject: Re: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Sent by: dfdl-wg-bounces(a)ogf.org Good point. The problem is that lengthKind-'explicit' is being used for two things: a) a length that is static b) a length that is calculated ...so the DFDL serializer must assume that the expression needs to be evaluated. For occursCountKind we have separate values for 'fixed' and 'expression'. If we did not, then occursCountKind would have the same problem except that it would affect defaulting rather than padding. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/09/2012 12:27 Subject: [DFDL-WG] Action 183 - chicken-and-egg situation with lengths given by expressions Sent by: dfdl-wg-bounces(a)ogf.org This mail is on the expected behaviour of the DFDL unparser when writing out a 'data' element the length of which is held in an earlier 'len' element. There are several scenarios, some straightforward and some that exhibit a chicken-and-egg behaviour. The principle of what happens is understood, the action is to make sure that the behaviour is explained in enough detail in the spec to enable implementations to be consistent. (Note - IBM DFDL does not yet support outputVaueCalc so has not hit this yet). Scenarios follow. The 'data' element shown is simple, but the same principles apply if it is complex. 1) 'len' is set from infoset - 'len' can be set in augmented infoset - No issue as 'data's length expression may be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 2) 'len' is set using outputValueCalc with fixed expression - When 'len's outputValueCalc is encountered, it can be evaluated then and there - 'len' can be set in augmented infoset - No issue as 'data's length expression may be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{10}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 3) 'len' is set using outputValueCalc with reference 'data' (unpadded) - When 'len's outputValueCalc is encountered, it can not yet be evaluated as it depends on the length of 'data' - 'len' can not yet be set in augmented infoset - Problem as 'data's length expression can not be evaluated - But we do know the unpadded length of 'data' so 'len's outputValueCalc can now be evaluated - In turn this means that 'data's length expression can now be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{dfdl:unpaddedLength(/message1/data)}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> 4) 'len' is set using outputValueCalc with reference 'data' (padded) - When 'len's outputValueCalc is encountered, it can not yet be evaluated as it depends on the length of 'data' - 'len' can not yet be set in augmented infoset - Problem as 'data's length expression can not be evaluated - We don't know the padded length of 'data' because we don't know 'len' - Problem: 'data's length expression can never be evaluated <xsd:element name="message1"> <xsd:complexType> <xsd:sequence> <xsd:element name="len" type="xsd:int" dfdl:outputValueCalc="{dfdl:representationLength(/message1/data)}" dfdl:lengthKind="explicit" dfdl:length="2" /> <xsd:element name="data" type="xsd:string" dfdl:length="{/message1/len}" dfdl:lengthKind="explicit" /> </xsd:sequence> </xsd:complexType> </xsd:element> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 04/09/2012 17:31 Subject: Fw: Behaviour for lengthKind 'endOfparent' is still not fully specified DFDL WG call 4th Sept 2012: 1) Agreed that for binary data, only xs:hexBinary and packed/BCD allowed to have endOfParent 2) Agreed this is the correct behaviour when filling to a known length 3) Agreed this is the correct behaviour when filling to a known length 4) Agreed this is the correct behaviour when filling to a known length It was noted that lengthKind 'explicit' on the parent may not result in a known length if the length is an expression. This is an example of a more general chicken-and-egg situation with lengths given by expressions, for which outputValueCalc and DFDL functions unpaddedLength() were added can be used. Action raised to ensure that the behaviour of an implementation is fully defined by the spec. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 04/09/2012 17:24 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org <dfdl-wg(a)ogf.org> Date: 04/09/2012 14:08 Subject: Behaviour for lengthKind 'endOfparent' is still not fully specified Noted when I reviewed latest spec - endOfParent and unparsing is not fully thought through. The spec today says that I can use endOfParent with binary data. There is a restriction in section 12.3.8, but it only applies when an element is endOfParent and its parent is lengthKind delimited. There are a couple of cases to consider: 1) Binary data of restricted length (see list in other email "proposed clarification/narrowing - delimited binary data should decimal"). I don't think it makes sense to allow these. We don't allow these binary reps for delimited. 2) Text data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should pad to box length with appropriate padChar, according to justification, as that is effectively a 'specified length'. Error if textPadKind is 'none'. Use parent's lengthUnits. 3) HexBinary data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should right-pad to box length with fill byte, as that is effectively a 'specified length'. 4) Packed/BCD binary data of variable length when unparsing. Box scenario. If the data in the infoset is shorter than the space in the box, what we do? I think we should pad to box length with zero bytes, according to justification, as that is effectively a 'specified length'. (Must be zero bytes and not fill byte as must be numeric in order to be parsed). In relation to 2 - 4, note that lengthKind 'endOfParent' can only be used with a parent lengthKind of 'explicit', 'pattern', 'prefixed' or 'endOfParent' or a choice with choiceLengthKind 'explicit', so the box scenario when unparsing therefore occurs only when lengthKind is 'explicit' or choiceLengthKind is 'explicit' - these are the cases when the length is known. Also note that when there are nested 'endOfParent' elements (which is allowed) then all padding must be done on the simple element (ie, the innermost element), to ensure that what is output can be parsed. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Unparsing - Pad to "big enough length to fit..." - can it be achieved in DFDL?
by Mike Beckerle 11 Dec '12

11 Dec '12

I have a format where fields are supposed to be padded so they all have the length of the longest one. I.e., something like this with fields terminated by | chars FIELD1|FIELD2 |ST|PH | ABC |CAPE COD|MA|888-999-0000| DEFG |BOSTON |MA|- | I believe I can warn if all the participants in a column of this tabular thing don't have the same length. I would just set a variable with the length of the first row, then on subsequent rows (parsed with a different element in the schema) I would verify that the length equals what is stored in the variable by way of an assertion. I believe this is what recoverableError kind is for actually, when you have redundant information you want to warn about. The problem I have is unparsing. How in DFDL I could determine how wide to pad each field before unparsing it. The spec requires me to measure the width of the fields (including that quasi-header row which is first, and measure how long each field is, then pad them to that maximum. I believe there is no way to do this currently in DFDL, because there is no way that an expression can refer to more than one record, nor any way to iterate or operate over sets of data. Instead an application must be written which can use DFDL v1.0 but the application has to provide some of the smarts. In this case, the application would have to walk the infoset and determine the maximum length for each such field, then provide these length maximums to the DFDL schema by way of externally set variables. Is there any other way? Second issue: In addition, the spec of my format requires the total maximum length for all the fields of a row to be at most 72 characters long. When unparsing, we don't evaluate assertions. So how can I test this? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

Re: [DFDL-WG] DFDL character entities in DFDL expressions
by Mike Beckerle 05 Dec '12

05 Dec '12

Well, yes, I think we're discussing exactly how that should work. No matter what, we do need to be clear that string values created in DFDL's expression language can contain these XML-illegal characters, since they are allowed in DFDL's infoset. This means that DFDL implementations can only re-use an existing XPath implementation to create their DFDL expression language implementation to the extent that it does NOT enforce the XML-illegal characters restrictions all over the place. I am currently working with standard Saxon-B XPath and will report back. But let's be optimistic. The question then is just what is the solution to creating a string-literal including these characters. That cannot be done without some beyond-XML mechanism. DFDL has a string-literal notation for expressing these characters, so we either say that string literals in the expression language can use the DFDL character and numeric entities, or we can do something more 'library like', and provide a function which interprets the string-literal notation, and isolate the implementation concerns a bit. As a language embedded in XML schema, we already straddle the fence of two somewhat inconsistent language environments. E.g., the literals one can use as the value of the default attribute on an element declaration cannot use DFDL character entities, as this is a purely XML Schema construct. Similarly, the regular expressions one can use for the XML schema pattern facet are more restrictive than the DFDL regular expressions one can use in a dfdl:assert, or a dfdl:lengthKind='pattern'. So, it's acceptable to me to say that expressions also have some split where the dfdl-specific aspects, like the dfdl character and numeric entities notation, is isolated in a sub-construct. On Wed, Dec 5, 2012 at 6:37 AM, Andrew Coleman <andrew_coleman(a)uk.ibm.com>wrote: > No, casting a hexBinary to a string will just write out the octets - i.e. > the string will be '00'. > > XPath itself has no mechanism for interpreting entity references or > character references. Its hosting language (XQuery or XSLT/XML) provides > this. Since DFDL is XML, wouldn't that provide a mechanism? > > Regards, > - Andy > > __________________________________________ > Andrew Coleman > WebSphere Message Broker Development > IBM Hursley Park > > > > > From: Steve Hanson/UK/IBM > To: Tim Kimber/UK/IBM@IBMGB, > Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle < > mbeckerle.dfdl(a)gmail.com>, Andrew Coleman/UK/IBM@IBMGB > Date: 05/12/2012 11:06 > Subject: Re: [DFDL-WG] DFDL character entities in DFDL expressions > ------------------------------ > > > Aren't XPath facilities sufficient here? > > outputValueCalc="{ if (fn:string-length(../s) lt 64) then > fn:concat(../s, xs:string(xs:hexBinary('00'))) else ../s }" > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > > From: Tim Kimber/UK/IBM@IBMGB > To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, > Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org > Date: 05/12/2012 10:51 > Subject: Re: [DFDL-WG] DFDL character entities in DFDL expressions > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > I think the restriction was aimed at avoiding things like this: > > outputValueCalc="{ if (fn:string-length(../s) lt 64) then > fn:concat(../s, '%#rFF;') else ../s }" > > I agree that a total ban is too restrictive. My personal preference would > be for the dfdl:string() function because it makes the usage of > DFDL-specific features obvious in the DFDL expression. But what would be > the return type of dfdl:string()? It it returned a sequence of characters > then the raw byte entity ( %#rnn; ) would still need to be disallowed. > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: kimbert(a)uk.ibm.com > Tel. 01962-816742 > Internal tel. 37246742 > > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org, > Date: 04/12/2012 23:36 > Subject: [DFDL-WG] DFDL character entities in DFDL expressions > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > > We currently have this language in the spec: > > "Within an expression, a string is never interpreted as a DFDL string > literal." > > To me this means one cannot use DFDL character entities in an expression. > > However, I need to do this: > > outputValueCalc="{ if (fn:string-length(../s) lt 64) then > fn:concat(../s, '%NUL;') else ../s }" > > Basically, I need to append a NUL on the end of the string in the output > value case. > > Unless I can put a %NUL; into an expression and have it interpreted as a > DFDL String literal, I am not sure how I can achieve this. > > At minimum I need a new DFDL function which might be an alternate string > constructor, such as dfdl:string('....') which interprets the argument as > something where the contents are to be scanned for DFDL character entities > and they are substituted so that the resulting string can contain the > characters that are disallowed in XML. (like NUL) > > -- > Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies > Tel: 781-330-0412 > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > *https://www.ogf.org/mailman/listinfo/dfdl-wg*<https://www.ogf.org/mailman/listinfo/dfdl-wg> > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

2 2

DFDL character entities in DFDL expressions
by Mike Beckerle 05 Dec '12

05 Dec '12

We currently have this language in the spec: "Within an expression, a string is never interpreted as a DFDL string literal." To me this means one cannot use DFDL character entities in an expression. However, I need to do this: outputValueCalc="{ if (fn:string-length(../s) lt 64) then fn:concat(../s, '%NUL;') else ../s }" Basically, I need to append a NUL on the end of the string in the output value case. Unless I can put a %NUL; into an expression and have it interpreted as a DFDL String literal, I am not sure how I can achieve this. At minimum I need a new DFDL function which might be an alternate string constructor, such as dfdl:string('....') which interprets the argument as something where the contents are to be scanned for DFDL character entities and they are substituted so that the resulting string can contain the characters that are disallowed in XML. (like NUL) -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

3 2

Clarification/Fix: Glossary entries about scanning
by Mike Beckerle 04 Dec '12

04 Dec '12

Definition of "scan" assumed in text throughout the spec encompasses both scanning for delimiters, and scanning for matches to regular expressions. So these glossary changes are needed to clarify that delimiter scanning is a specialized sub-class of scanning generally. Delimiter Scanning: Remove subphrase ", or simply scanning for short. Scanning also..." replace with "Delimiter scanning also ..." Scan: Remove the word "bytes". Insert after "...and terminators." this text ", or matches to regular expressions." -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

1 0

OGF DFDL WG Call Minutes 2012-12-04
by Steve Hanson 04 Dec '12

04 Dec '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12973?download= Errata document is now at v011 (draft) here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] discriminators near PoU or not
by Steve Hanson 04 Dec '12

04 Dec '12

Agreed to defer to a future DFDL release. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 19/11/2012 19:24 Subject: Re: [DFDL-WG] discriminators near PoU or not Hi Mike, I hit a similar scenario today with an EDIFACT message. There was an optional array containing a bunch of initiated records. I used 'initiatedContent' for the array's sequence, but needed to add an extra discriminator to stop an error from within one of the records from causing the parser to backtrack and conclude that the array occurrence was not there. Not easy to know where to put it. One possibility is an 'all' attribute that says it resolves all points of uncertainty that are in scope? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 29/10/2012 01:25 Subject: [DFDL-WG] discriminators near PoU or not Sent by: dfdl-wg-bounces(a)ogf.org I thought this situation was worth discussing. I have a bunch of messages. They all work like this: <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> There are hundreds of those. These all are used in this way: <element name="topLevel"> <complexType> <sequence> <element name="message" minOccurs="1" maxOccurs="unbounded"> <complexType> <choice> <element ref="messageName1"/> <element ref="messageName2"/> .... </choice> </complexType> </element> </sequence> </complexType> </element> So, there are two points of uncertainty here. The choice, and the unbounded array surrounding it. For each message, it is not until the sequence carrying the initiator, which is down inside the message structure, that we know for sure we've got this kind of message. So currently each message element is designed to be used ONLY in the above context of two surrounding points of uncertainty, like so (same message format as above but with TWO discriminators added). <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> <annotation><appinfo...>  <dfdl:discriminator>{ true() }</dfdl:discriminator>  <dfdl:discriminator>{ true() }</dfdl:discriminator> </appinfo></annotation> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> I'd love to have a better solution to this issue. But in the absence of one this works and achieves what I want which is that once it hits the initiator, we discriminate both which message we have, and we discriminate that we in fact have a message. Interesting that I'd really prefer to discriminate them in the opposite order of their nesting. I could discriminate the array the the very start of the message. But I have no way to do that because the discriminators apply in inward-out nesting order, and the nearest enclosing PoU is the choice. So I have to discriminate that one first, and I can't discriminate that one until I see the initiator, which is later into the message. Possible fixes/improvements: allow a label/id on each PoU, and allow referencing that label from a discriminator to say exactly which PoU you are discriminating. This makes the outward reference implied by a discriminator explicit. You still have the context issue, but it's not implied anymore, it is explicit. (This makes the problem like expressions with "../../.." that reach up and out of a construct. They are reaching up and out, but at least you can see directly that they are doing so.) -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

text alignment and Errata 2.95 in DFDL Errata v011
by Mike Beckerle 04 Dec '12

04 Dec '12

Steve, Please add this point to the agenda for next week. We have to revisit this errata given the 7-bit character encoding. (And, I've discovered now there are legacy 6-bit as well which do come up.) ...mikeb On Mon, Nov 26, 2012 at 10:32 PM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com>wrote: > We also need to decide on Errata 2.95 which is about whether text is > implicitly 8-bit aligned. > > Given that we have cases of text aligned in other ways (the 7-bit packed > ascii stuff), this rule seems heavy handed and inflexible. > > We could say that if alignment="implicit", then the first codepoint of any > text is implicitly aligned on an 8-bit boundary. This would apply to > delimiters, but also to any text that is being scanned with a regex. > Subsequent codepoints of the same text are not specifically aligned. This > lets us choose alignment="1", and the 7-bit packed stuff can then have > syntax in it, have regex's that it must match, etc. but if > alignment="implicit", then the normal, less error prone behavior of > skipping to a byte-boundary would be in force. > > ...mikeb > > On Mon, Nov 26, 2012 at 10:10 PM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com>wrote: > >> I reviewed this. >> >> 2.108 has Change: Format annotations.... and also Remove: Format >> Annotations. Should just have Change. >> >> 3.13 I think we missed a couple places where the previous design where we >> had replacement control is still leaking into the text. >> >> Phrase "Any character that is skipped counts as 0." should be dropped. We >> don't skip anymore. >> >> Concerning: "In the following, the match is against data where any >> character replacement has already taken place." The first sub-bullet about >> lengthKind pattern is incorrect. For lengthKind pattern I believe any >> decode error should cause the match to end, i.e., a pattern that matches >> digits only will match a string of digts followed by binary data that >> causes a decode error. Similarly, when matching literal nils, (character or >> literal value), any decode error should cause the match to fail. >> >> >> >> >> On Fri, Nov 23, 2012 at 12:41 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: >> >>> A few corrections made and added errata from Nov 20th call. Please >>> review for next call. Also now on Redmine. >>> >>> >>> >>> Regards >>> >>> Steve Hanson >>> Architect, Data Format Description Language (DFDL) >>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >>> IBM SWG, Hursley, UK* >>> **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >>> tel:+44-1962-815848 >>> ----- Forwarded by Steve Hanson/UK/IBM on 23/11/2012 17:40 ----- >>> >>> From: Steve Hanson/UK/IBM >>> To: dfdl-wg(a)ogf.org, >>> Date: 20/11/2012 14:23 >>> Subject: First draft of DFDL Errata v011 >>> ------------------------------ >>> >>> >>> Please find attached the first draft of Errata v011. >>> This incorporates all DFDL spec errata raised since v010 (June 22 2012) >>> until the last DFDL WG call (November 13 2012). >>> Change tracking: I accepted all changes from v010 so the changes you see >>> are all due to v011. >>> I would appreciate a review for correctness and omissions. >>> Please note that some of the new errata have been best handled by >>> updating existing related errata, so please read from start. >>> >>> >>> >>> Regards >>> >>> Steve Hanson >>> Architect, Data Format Description Language (DFDL) >>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >>> IBM SWG, Hursley, UK* >>> **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >>> tel:+44-1962-815848 >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >>> 3AU >>> >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >>> 3AU >>> >>> -- >>> dfdl-wg mailing list >>> dfdl-wg(a)ogf.org >>> https://www.ogf.org/mailman/listinfo/dfdl-wg >>> >> >> >> >> -- >> Mike Beckerle | OGF DFDL WG Co-Chair >> Tel: 781-330-0412 >> >> > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > > -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

1 0

Clarification: can you escape a pad char with an escape char?
by Mike Beckerle 04 Dec '12

04 Dec '12

Did we decide whether an escape character can precede a pad character hence, making it part of the content, not the padding? E.g., suppose pad character is P, escape character is E. then can 'abcdEPPPP' be interpreted as 'abcdP' then 3 pad characters, or did we decide that padding ignores escape chars? I think we decided that padding is outside of the escape blocks, so that if [ and ] are my escape block start and end, then PPP[PabcP]PPP would be 'PabcP' as content. but I don't think we commented on regular single escape character characters and pad char. It seems consistent to me that 'abcdEPPPP' is abcdP as content. but if not, then 'abcdEPPP' stripping off the trailing padding ends with the escape character, so the escape character is escaping nothing hence the content would be 'abcd' I believe. Comments? ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1