May 2016 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

[DFDL]: Apply daffodil on S2G mission schemas
by Michele Zundo 07 Jun '16

07 Jun '16

Dear Steve/Mike, we have run some of our DFDL schema through the daffodil parser and identified 3 issues described in the attached document. The second is expected, but could you please review/comment the first and the last (in red in the text ? ) Thanks This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Michele

3 3

OGF DFDL WG Call Minutes 2016-05-24
by Steve Hanson 25 May '16

25 May '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13532?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot be empty string? textStandardNaNRep and friends cannot be empty string.
by Mike Beckerle 25 May '16

25 May '16

I believe the analogy is not quite right. For zero, having textStandardZeroRep "" means that the "ordinary" string for zero i.e., "0" means zero. There's an implied string conceptually. But for NaN and Inf there is no "ordinary" string to fall back on. They would have to cause an error. I think just prohibiting these from being "" is a better choice. ....mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, May 24, 2016 at 11:04 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike > > I came to write up the issue tracker for this and had a thought. Property > textStandardZeroRep says: > > *The empty string means that there is no special literal string for zero. > * > > I wondered if the equivalent is a better interpretation of empty string > for textStandardNanRep and textStandardInfinityRep ? > > Regards > > Steve Hanson > *IBM Integration Bus* > <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, > Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > mob:+44-7717-378890 > > > > From: Steve Hanson/UK/IBM > To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> > Date: 19/05/2016 21:53 > Subject: Re: [DFDL-WG] clarification: textBooleanTrue/FalseRep > cannot be empty string? textStandardNaNRep and friends cannot be empty > string. > ------------------------------ > > > Agree that textBooleanxxxRep properties should state empty string not > allowed. > > The textStandardNanRep and textStandardInfinityRep properties should state > empty string not allowed. The others you list already mention empty string. > > Regards > > Steve Hanson > *IBM Integration Bus* > <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, > Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > mob:+44-7717-378890 > > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> > Date: 19/05/2016 17:38 > Subject: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot > be empty string? textStandardNaNRep and friends cannot be empty string. > Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> > ------------------------------ > > > > > We don't stipulate that textBooleanTrueRep nor textBooleanFalseRep cannot > be empty string. > > We do stipulate that entity ES is not allowed. So I expect these > properties cannot be "" meaning the same thing that "%ES;" would mean. > Since we disallow the latter, we should disallow just "". > > Similarly, for textStandardGroupingSeparator, textStandardExponentRep, > textStandardNaNRep, and textStandardInfinityRep we specify that the string > literal value behaves as specified in textStandardDecimalSeparator, but we > don't specify that these cannot be empty string (which is specified in > textStandardDecimalSeparator, but not in the "Text Number Character > Restrictions" part of that property description that is referenced from > these other properties.) > > ES and WSP* are not allowed, so almost certainly empty string should not > be allowed. > > > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > *www.tresys.com* <http://www.tresys.com/> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the *OGF Intellectual Property Policy* > <http://www.ogf.org/About/abt_policies.php> > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >

1 0

Re: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot be empty string? textStandardNaNRep and friends cannot be empty string.
by Steve Hanson 24 May '16

24 May '16

Mike I came to write up the issue tracker for this and had a thought. Property textStandardZeroRep says: The empty string means that there is no special literal string for zero. I wondered if the equivalent is a better interpretation of empty string for textStandardNanRep and textStandardInfinityRep ? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 19/05/2016 21:53 Subject: Re: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot be empty string? textStandardNaNRep and friends cannot be empty string. Agree that textBooleanxxxRep properties should state empty string not allowed. The textStandardNanRep and textStandardInfinityRep properties should state empty string not allowed. The others you list already mention empty string. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 19/05/2016 17:38 Subject: [DFDL-WG] clarification: textBooleanTrue/FalseRep cannot be empty string? textStandardNaNRep and friends cannot be empty string. Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> We don't stipulate that textBooleanTrueRep nor textBooleanFalseRep cannot be empty string. We do stipulate that entity ES is not allowed. So I expect these properties cannot be "" meaning the same thing that "%ES;" would mean. Since we disallow the latter, we should disallow just "". Similarly, for textStandardGroupingSeparator, textStandardExponentRep, textStandardNaNRep, and textStandardInfinityRep we specify that the string literal value behaves as specified in textStandardDecimalSeparator, but we don't specify that these cannot be empty string (which is specified in textStandardDecimalSeparator, but not in the "Text Number Character Restrictions" part of that property description that is referenced from these other properties.) ES and WSP* are not allowed, so almost certainly empty string should not be allowed. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Action 242 - valueLength and contentLength function wording
by Steve Hanson 24 May '16

24 May '16

Background for discussion on DFDL WG call today. Part of deferred action 242. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 ----- Forwarded by Steve Hanson/UK/IBM on 24/05/2016 11:58 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 15/04/2014 15:45 Subject: Fw: [DFDL-WG] Action 242 - valueLength and contentLength function wording I had a quick look at the document and Steve's review comments. The text tries to define valueLength() in terms of the info set value, or states that it is 'calculated from' the infoset value. I believe this will mislead many readers ( it already has done, in fact ). People just assume that the 'value length' is 'the length of the logical value' and they don't think any more deeply about it. So it is absolutely essential to state clearly that this function returns the length of a region in the data stream. It may be simpler to define the behaviour in terms of the length of the simpleContent or complexContent region, minus any padding. The other point is around the bytes/characters issue. Mike's words, or something similar, are definitely required because we don't mandate that the encoding must be consistent throughout a complex type. Nor do we prohibit a mixture of character and non-character content in a complex type. And someone might even call valueLength() with lengthUnits 'characters' on a complex type that contains no character data at all. In principle it would be possible for the value to be available while parsing ( for elements that follow the specified element ). If we are disallowing this then it should be stated very clearly somewhere. And probably is already. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 ----- Forwarded by Tim Kimber/UK/IBM on 15/04/2014 15:28 ----- From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 15/04/2014 15:01 Subject: Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Sent by: dfdl-wg-bounces(a)ogf.org Review comments added: Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 11/04/2014 14:04 Subject: Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Sent by: dfdl-wg-bounces(a)ogf.org Revised Action 242 proposed changes word doc attached. I have incorporated the discussion in this thread (I hope.) Please review. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Mar 25, 2014 at 10:56 AM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: This language is consistent with what we say for lengthKind pattern in section 12.3.5: "When unparsing, the dfdl:valueLength of a complex type element when the length units is 'characters' is computed as if the entire structure was unparsed into a temporary data stream beginning at position 1, and then this data stream is considered to be text in the character set encoding specified by the dfdl:encoding property, regardless of the actual representation of the complex type element or the elements contained within it. The number of characters in this temporary data stream is the value length of the complex type." The behavior of the IBM DFDL implementation for valueLength is as described is consistent with the above, excepting that it will not detect a decode error, and it gives an SDE (?) if the encoding is not fixed width. Since we have decided not to require that a complex type element is recursively all text all the way down, I believe we have to tolerate implementations having different behaviors in the potentially meaningless cases where there is binary data or encoding changes in the complex type. So I would add to the above suggested language this: "However, if creation of this data stream would cause an encoding error, or parsing of this data stream as characters would cause a decoding error, then the behavior and return value of dfdl:valueLength are implementation dependent." Looking at the DFDL spec, I am concerned that we never really say what we mean by the "length of the ComplexContent region." (Last sentence before Table 7 in section 12.3.7) Section 12.3.7.3 doesn't do it. The dfdl:valueLength function may be the first place where we have to actually say how the various sub-regions contribute to the ComplexContent region's length. I believe this is the obvious "sum of length of all contained regions", but keep in mind that alignment region lengths will vary depending on the starting alignment, so the length is, in general, dependent on the position within the bit stream. Hence when unparsing we have to specify that the dfdl:valueLength is measured as if the ComplexContent region started at position 1 (as I did above) so that internal alignment regions can be given meaningful lengths. The general clarification should be added to 12.3.7.3, or to section 12.3.7 immediately before section 12.3.7.1. Something like this: "The length of the ComplexContent region is the sum of the lengths of the contained regions. However, note that alignment regions inside the ComplexContent may be of different lengths depending on the ComplexContent's starting position alignment." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Mon, Mar 24, 2014 at 11:34 AM, Andrew Edwards <andy.edwards(a)uk.ibm.com> wrote: Steve (et al) - Resending as the last one bounced. I'll usurp Tim and respond :) Currently the IBM implementation insists on using a fixed-length encoding and returns an "unsupported" error message for a variable width encoding. With a fixed width encoding, we "do the maths" using the bytes-per-character and the bytes written by this complex element. HTH, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Steve Hanson/UK/IBM 24/03/2014 14:52 To "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, cc Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Link Note errata 3.9, my bolding: "3.9. Section 12.3.5, 7.3.1, 7.3.2. The spec originally allows lengthKind ‘pattern’ to be used when the representation of the current element, or of a child element, is binary, but imposes restrictions on the encoding that can be in force. Clarify that the encoding property must be defined for the element (else schema definition error), and that a decoding processing error is possible if the match of the regex encounters data that does not decode in that encoding, dependent on the setting of encodingErrorPolicy. Remove section 12.3.5.1. Same clarifications needed for testKind ”pattern” property for asserts and discriminators. For consistency, the restriction that a complex element of specified length and lengthUnits ‘characters’ must have children that are all text and that have the same encoding as the complex element, is dropped." That's the restriction that I was referring to in my comment below. I can see why it was dropped - basically the parser now just tries to decode n characters using the complex element's encoding (and encodingErrorPolicy). We could apply the same principle for dfdl:valueLength & dfdl:contentLength - you build the stream from the bottom up, and then decode it using the complex element's encoding (and encodingErrorPolicy ?) to get the length in characters. Note that's how unparsing for lengthKind 'prefixed' with lengthUnits 'characters' would work as well - the spec just says "For a complex element, the length is that of the ComplexContent region" which is not sufficient (12.3.4). Similar deal for lengthKind 'explicit' - in order to know the size in chars of ElementUnused the unparser needs to know the size in chars of the data first (12.3.7.3). (Of course, for a fixed width encoding, you don't need to decode, you can just do the maths, but for the general case you need to decode. Also just doing the maths does not take encodingErrorPolicy into account). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 24/03/2014 12:55 Subject: Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Mike 23.5.3.1. Value length is only a function of the dfdl:encoding property if the element has a text representation. Not sure this needs to be (re)stated here. 23.5.3.1. "The value length is computed from the DFDL infoset value, ignoring the dfdl:length or dfdl:textOutputMinLength property. Other DFDL properties which affect the length of a text or binary representation are respected, it is only an explicit length which is ignored." Last sentence is too imprecise - should be phrased in terms of the grammar. 23.5.3.1. "If the second argument is 'characters' then the element must have text representation and it is a schema definition error otherwise". Yes but only for a simple type, so should be qualified. 23.5.3.1. "If the second argument, giving the length units, is 'characters', then recursively, this complex type element must have text representation throughout all its contained elements and framing, all of which must also use a uniform character set encoding." I can't see that restriction elsewhere in the spec when it talks about length of ComplexContent and lengthUnits 'characters' - I was expecting it to be in section 12.3.4 or 12.3.7.3 which face the same issue - but it isn't. Did we decide not to have this restriction? Without such a restriction, how does the unparser come up with a meaningful length (unless it re-parses)? (Tim - what does IBM DFDL do here?) What about delimiters and padding of children that use %#r entities? 23.5.3.2. The points in 23.5.3.1 about escape characters, length as a function of encoding, and bottom up for complex elements, apply equally to 23.5.3.2. It might be easier just to say in 23.5.3.2 that dfdl:contentLength for complex elements is same as dfdl:valueLength, and for simple elements differs only by the additional inclusion of LeftPadding and RightPadOrFill regions. Also noted in passing: Specified length - An item has specified length when dfdl:lengthKind is "implicit", "explicit", or "prefixed". should be Specified length - An element has specified length when dfdl:lengthKind is "implicit" (simple type only), "explicit", or "prefixed". Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 20/03/2014 17:21 Subject: [DFDL-WG] Action 242 - valueLength and contentLength function wording Sent by: dfdl-wg-bounces(a)ogf.org See attached doc which is proposed revisions to section 23.5.3 Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy [attachment "Action-252-DFDL-Functions-23.5.3.docx" deleted by Andrew Edwards/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "Action-252-DFDL-Functions-23.5.3.docx" deleted by Steve Hanson/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU #### Action-252-DFDL-Functions-23.5.3.docx moved to MyAttachments Repository V3.8 () on 30 April 2014 by Steve Hanson. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Action 242 - propose simplification for length units in dfdl:valuelength and dfdl:contentLength functions
by Steve Hanson 24 May '16

24 May '16

Background for discussion on DFDL WG call today. Part of deferred action 242. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 ----- Forwarded by Steve Hanson/UK/IBM on 24/05/2016 11:57 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 28/04/2014 11:24 Subject: Re: [DFDL-WG] Action 242 - propose simplification for length units in dfdl:valuelength and dfdl:contentLength functions Mike I don't think you can always ensure the result is well-defined. The element (first argument) is silent about its lengthUnits when not applicable to is lengthKind - specifically 'pattern', 'delimited', 'implicit' (complex) and 'endOfParent'. For the first three we need to allow lengthUnits 'characters' to be provided as the second argument. Note for 'endOfParent' the length can't be calculated so it should be an SDE if the function is called against that element ? It should be an SDE if lengthUnits 'characters' is provided as the second argument and the representation is 'binary' or the type is hexBinary. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 24/04/2014 23:41 Subject: [DFDL-WG] Action 242 - propose simplification for length units in dfdl:valuelength and dfdl:contentLength functions Sent by: dfdl-wg-bounces(a)ogf.org These functions currently take the desired result units as the second argument. It is very complicated if the length units of the element are bytes, to ask for this length to be recast in characters due to the fragment-of-a-character on the end problem. In some sense there is quotient/remainder here, which means there are multiple operations here, not just one. There is the equivalent of floor (as many characters as fit, then fillByte) ceiling (every byte filled from some character), remainder (how many left over bytes are there). I think this is silly unless we have use cases driving this. I believe this restriction matches all use-cases I can think of: Restriction: If the element (first argument) to either valueLength or contentLength has length units of characters -> can ask for valueLength or contentLength in any of characters, bytes, or bits bytes -> can ask for valueLength or contentLength in bytes or bits bits -> can only ask for valueLength or contentLength only in bits. and it is an SDE otherwise. This insures that the result of these functions is always well defined. I understand why there is a box sized in characters and I need to know its size in bytes, but the opposite I can't see a use case for, and it isn't crisply defined even. Ditto for length in bits. If the box is sized in bytes I might want that in bits (so * 8), but if the box is sized in bits, and I ask for bytes, what do I want if there is a fragment of a byte? ceiling? floor? remainder? Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: dfdl:contentLength and complex types with interior alignmentFill
by Steve Hanson 24 May '16

24 May '16

Background for discussion on DFDL WG call today. Part of deferred action 242. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 ----- Forwarded by Steve Hanson/UK/IBM on 24/05/2016 11:58 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 28/04/2014 11:10 Subject: Re: [DFDL-WG] dfdl:contentLength and complex types with interior alignmentFill Mike This doesn't solve the problem fully, because it assumes an alignment of 0 and in practice the alignment might be some odd number that forces the first child with alignment 2, 4 or 8 onto a different offset, and hence the length is different. So I'm not sure whether the additional calculation helps much. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 24/04/2014 22:36 Subject: [DFDL-WG] dfdl:contentLength and complex types with interior alignmentFill Sent by: dfdl-wg-bounces(a)ogf.org dfdl:contentLength interacts pretty badly with alignment because alignmentFill regions mean you cannot know the size of a complex type element without knowing its starting position. It is very likely that one will have outputValueCalc that wants to evalutate dfdl:contentLength on an infoset item before the starting position for that item is known or knowable. To fix this we say that dfdl:contentLength(e) on complex-type infoset item e, is computed as if element e's alignment was first put into place along with the rest of the LeftFraming, and then the content length computed as if the complex content was unparsed starting from after that LeftFraming up to the final RightFraming In other words, the entire element has to be unparsed, not just the complex content of it, and then the part of that unparsed data that corresponds to the complex content is used as the return value from dfdl:contentLength. This works because element e's alignment (LeftFraming generally) is not within the complex content for element e. It is before the complex content of the element, so if the schema puts an explicit alignment on element e, then it will clearly be aligned that way when it is really unparsed, but the dfdl:contentLength function will also compute a length from the infoset value as if that same alignment were in place. I think that is about the best we can do. Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2016-05-24
by Steve Hanson 24 May '16

24 May '16

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13532?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Fwd: proposal: DFDL needs additional function dfdl:characterCode
by Steve Hanson 24 May '16

24 May '16

To be discussed on DFDL WG call today. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 10/05/2016 16:57 Subject: Fwd: [DFDL-WG] proposal: DFDL needs additional function dfdl:characterCode This is the thread that mentions dfdl:characterCode(string, pos): int, and dfdl:character(charCodeInt): String proposed functions. It also discusses the issue of XML-illegal characters in the XML representation of the DFDL infoset. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy ---------- Forwarded message ---------- From: Tim Kimber <KIMBERT(a)uk.ibm.com> Date: Thu, Nov 1, 2012 at 12:00 PM Subject: Re: [DFDL-WG] proposal: DFDL needs additional function dfdl:characterCode To: dfdl-wg(a)ogf.org Correct - there is no way at all of using illegal characters in an XML document. Not CDATA, not character entities. They simply must not appear anywhere. I agree with Steve that XML compatibility is not the only requirement for a DFDL info set - we should not do anything to make it XML specific in a way that harms it generality. To balance that, I also think that the DFDL Working Group should be paying attention to the issues around XML compatibility, given that DFDL is based on XML Schema and many potential adopters of DFDL will want to know about XML compatibility. Mike's proposal of mapping illegal characters into the Unicode Private Use Area sounds like a reasonable approach for implementers to use. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 01/11/2012 15:14 Subject: Re: [DFDL-WG] proposal: DFDL needs additional function dfdl:characterCode Sent by: dfdl-wg-bounces(a)ogf.org Turns out the XML char entities are not an escape scheme for putting illegal chars in. E.g.  is illegal even expressed that way. The char entities are essentially an internationalization hack so you can enter and render any legal character using only a small charset. On Nov 1, 2012 8:48 AM, "Suman Kalia" <kalia(a)ca.ibm.com> wrote: Shouldn't we be using entity references for XML syntactic character found in text/binary data while creating info set and vice versa... Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson <smh(a)uk.ibm.com> To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 11/01/2012 07:57 AM Subject: Re: [DFDL-WG] proposal: DFDL needs additional function dfdl:characterCode Sent by: dfdl-wg-bounces(a)ogf.org >From WG call minutes 2012-10-30: "Beyond the scope of DFDL 1.0. Assumption for now is that infoset needs post-processing." Mike has observed that other software systems "map the illegal characters to/from the Unicode Private Use Area." Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 04/10/2012 23:39 Subject: [DFDL-WG] proposal: DFDL needs additional function dfdl:characterCode Sent by: dfdl-wg-bounces(a)ogf.org An important use case for DFDL is converting legacy data to/from XML. XML 1.0 disallows a bunch of string characters. If the data contains those characters, then the question arises of what to turn them into that both preserves information content, but also is legal in XML so that you can convert the DFDL infoset into XML without violating XML's constraints. The natural thing to do is create an element containing the character code of the illegal character, as an integer. E.g., character code U+0001 would become an element. Such as: <ccode>1</ccode>. This could be done using a hidden element that is a string, and the element ccode above would have an inputValueCalc that converts the offending character of that string into an integer. But we need a function dfdl:characterCode(str, pos) : int The arguments would be a string, and a position (base 1) within that string, and the return result would be the character code of the character in the string at that position. If pos is out of the bounds of the string (i.e., is negative, 0, or too large), then a processing error would occur. For unparsing the inverse function would also be needed: dfdl:character(intArg) : string. This would return a string containing one character whose codepoint is the intArg. Example Consider this data: 123<0>456<1>789<2>123l where <0> means just one character with codepoint 0, etc. In hex that would be 313233 00 343536 01 373839 02 313233 The best I can think of for modeling this while preserving all information would end up with XML looking like this: <nonXMLString> <fragment><stringData>123</stringData></fragment> <fragment><nonXMLChar><charCode>0</charCode></nonXMLChar></fragment> <fragment><stringData>456</stringData></fragment> <fragment><nonXMLChar><charCode>1</charCode></nonXMLChar></fragment> <fragment><stringData>789</stringData></fragment> <fragment><nonXMLChar><charCode>2</charCode></nonXMLChar></fragment> <fragment><stringData>123</stringData></fragment> </nonXMLString> So our nonXMLString is of a type which is array of fragment, a fragment is a choice of either (legal XML) stringData, or a nonXMLChar. The nonXMLChar has a child element because it will need to convert to from a string so will use inputValueCalc and outputValueCalc to do so, so it needs to be a sequence so that it can have the other hidden elements needed to pull this off. stringData would have lengthKind="pattern" and a pattern that allows any sequence of XML-allowed characters. nonXMLChar would have a hidden first child element of type string of explicit length 1 with an assertion that the string match a pattern that is any of the illegal characters (but just one of them). The charCode child element would inputValueCalc to get the character code of the character. For 8 bit encodings it would be ok as a table lookup in XPath, but for unicode..... we'd need a function that returns a character code. If you just have one embedded illegal character, like NUL, then you could just model it as a separator, which would simplify things considerably (and is possible in a someday XML 1.1 future since NUL is then the only disallowed character.) But for XML 1.0's illegal characters, we need to be able to convert to/from some non-string representation if we are to preserve information content. Hence we need these additional functions. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0