- dfdl-wg - lists.ogf.org

Action 253 - delimiters - misleading phrasing - was: Re: Fw: DL11A #3: (Question)[1](Open)(English)need clarification
by Mike Beckerle 28 Apr '14

28 Apr '14

There are two issues raised in the email thread started by Oya Argun. Let's look at the second one first since it is quickly addressed. This is just a typographical error "proceed" should be precede. The offending sentence was: On parsing the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data, unless the dfdl:escapeCharacter is preceded or the dfdl:escapeEscapeCharacter does not *proceed* the dfdl:escapeCharacter. I have already made the above correction to the draft-gwdrp-dfdl-v1.0.4-r07.docx draft of the spec. The other issue requires us to reword the text, as the material identified as hard to translate is actually misleading and should be cleaned up. Suggested Erratum: In the description of the dfdl:initiator property, the sentence "The initiator with the longest match is the one that is selected as having been 'found', with length-ties being resolved so that the matching initiator is selected that is first in the order written in the schema." should be dropped. In the description of the dfdl:terminator property, the sentence "The terminator with the longest match is the one that is selected as having been 'found', with length-ties being resolved so that the matching terminator is selected that is first in the order written in the schema." should be dropped. Add this sentence: "This property can be used to determine the length of an element as described in Section 12.3.2 dfdl:lengthKind 'delimited'." In the description of the dfdl:separator property, the sentence "The separator with the longest match is the one that is selected as having been 'found', with length-ties being resolved so that the matching separator is selected that is first in the order written in the schema." should be dropped. Add this sentence: "This property can be used to determine the length of an element as described in Section 12.3.2 dfdl:lengthKind 'delimited'." It is a separate issue whether we consider the description of how lengthKind delimited works in 12.3.2 to be adequate or not, and that is out of scope for this particular action item/erratum. The above suggested erratum text gets rid of the misleading language from the descriptions of the properties. Let's discuss whether improvement to 12.3.2 is required on the next DFDL WG call. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php> On Fri, Apr 11, 2014 at 10:36 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > > Regards > > Steve Hanson > Architect, *IBM DFDL*<http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 11/04/2014 15:36 ----- > > From: Steve Hanson/UK/IBM > To: Kevin Safford/UK/IBM@IBMGB, > Cc: Alex Wood1/UK/IBM@IBMGB, "Mike Beckerle" <mbeckerle(a)tresys.com> > Date: 06/03/2014 16:32 > Subject: Re: Fw: DL11A #3: (Question)[1](Open)(English)need > clarification > ------------------------------ > > > Kevin > > 1) When the dfdl:initiator property is a list of values, and the data > matches more than one value, then the longest value is the one selected. > Example: data is aaa12345, dfdl:initiator is "aa aaa", then aaa is the > match as it is longest. However if the data matches more than one value and > the values have the same length, then the one that is declared earlier in > the list is selected. > > I have to say that I am struggling to come up with an example where it > actually makes a material difference, because if it matches multiple of the > same length, then surely they must be equivalent anyway? Maybe white space > has an effect here. Mike Beckerle crafted these words, perhaps he can give > an example. (Note that same words occur dfdl:terminator and dfdl:separator). > > 2) Agree, and have added to the Minor Editorial public comment on Redmine. > > Regards > > Steve Hanson > Architect, *IBM DFDL*<http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Kevin Safford/UK/IBM > To: Steve Hanson/UK/IBM@IBMGB, Alex Wood1/UK/IBM@IBMGB, > Date: 06/03/2014 14:40 > Subject: Fw: DL11A #3: (Question)[1](Open)(English)need > clarification > ------------------------------ > > > Hi > > I need some input from you chaps on this translation query. > > Could you suggest a rephrase, or explanation of point 1? > > Point 2 I assume is a typo for precede. It's present in the spec. Can you > confirm? > > > Regards > > *Kevin Safford* > Information Developer > I work Mondays to Thursdays only. > ------------------------------ > *Phone:* 44-1962-819217 > * E-mail:* *SAFFORDS(a)uk.ibm.com* <SAFFORDS(a)uk.ibm.com> > * My web site: **sites.google.com/site/kevinsafford*<https://sites.google.com/site/kevinsafford> > * Find me on:* [image: Twitter: https://twitter.com/KevSaff]<https://twitter.com/KevSaff> > > Hursley Park > Hursley, SO21 2JN > United Kingdom > > > ----- Forwarded by Kevin Safford/UK/IBM on 06/03/2014 14:37 ----- > > From: tctadm(a)cn.ibm.com > To: undisclosed-recipients:;, > Date: 06/03/2014 10:01 > Subject: DL11A #3: (Question)[1](Open)(English)need clarification > ------------------------------ > > > *** This mail is sent by TCT system automatically. Please do NOT reply > to this ID. *** Click here to read the article and response: > > *https://www.ibm.com/software/globalization/translationcommunications/BaseServlet.wss?taskName=ReadArticlePre&prrefnum=-1&arrefnum=7701628499*<https://www.ibm.com/software/globalization/translationcommunications/BaseSe…> > *Project code* DL11A *Project admin* terrychu *Article subject* need > clarification *Article type* Question *Category name* English/Source > files - PII,DL11AAAP002_Tooling *Testcase/Filename/IU * > plugins/com.ibm.dfdl.help/f1/dfdlPorperties.xml *Current status* Open *Priority > level * 1 *Stop-Ship* English:n/a *Originator * English > *Opened* by oya.argun(a)superonline.com / Trans. Vendor, Turkey / > English (oyargun) 2014-03-06 10:00:25 Hello, > > 1)The initiator with the longest match is the one that is selected as > having been 'found', with length-ties being resolved so that the matching > initiator is selected that is first in the order written in the schema. > > I am afraid this string is very complex, please re-phrase it if possible > or provide a comprehensive explanation. > > 2) In the following string I could not quite understand this phrase: > 'or the dfdl:escapeEscapeCharacter does not proceed the > dfdl:escapeCharacter'. What is meant by 'proceed' here?" > > On parsing the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are > removed from the data, unless the dfdl:escapeCharacter is preceded or the > dfdl:escapeEscapeCharacter does not proceed the dfdl:escapeCharacter. > > Thanks in advance > Oya > > ------------------------ Welcome to the TCT website: > ------------------------ > > *https://www.ibm.com/software/globalization/translationcommunications/index.jsp*<https://www.ibm.com/software/globalization/translationcommunications/index.…> > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >

2 1

Action 242 - propose simplification for length units in dfdl:valuelength and dfdl:contentLength functions
by Mike Beckerle 28 Apr '14

28 Apr '14

These functions currently take the desired result units as the second argument. It is very complicated if the length units of the element are bytes, to ask for this length to be recast in characters due to the fragment-of-a-character on the end problem. In some sense there is quotient/remainder here, which means there are multiple operations here, not just one. There is the equivalent of floor (as many characters as fit, then fillByte) ceiling (every byte filled from some character), remainder (how many left over bytes are there). I think this is silly unless we have use cases driving this. I believe this restriction matches all use-cases I can think of: Restriction: If the element (first argument) to either valueLength or contentLength has length units of characters -> can ask for valueLength or contentLength in any of characters, bytes, or bits bytes -> can ask for valueLength or contentLength in bytes or bits bits -> can only ask for valueLength or contentLength only in bits. and it is an SDE otherwise. This insures that the result of these functions is always well defined. I understand why there is a box sized in characters and I need to know its size in bytes, but the opposite I can't see a use case for, and it isn't crisply defined even. Ditto for length in bits. If the box is sized in bytes I might want that in bits (so * 8), but if the box is sized in bits, and I ask for bytes, what do I want if there is a fragment of a byte? ceiling? floor? remainder? Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

dfdl:contentLength and complex types with interior alignmentFill
by Mike Beckerle 28 Apr '14

28 Apr '14

dfdl:contentLength interacts pretty badly with alignment because alignmentFill regions mean you cannot know the size of a complex type element without knowing its starting position. It is very likely that one will have outputValueCalc that wants to evalutate dfdl:contentLength on an infoset item before the starting position for that item is known or knowable. To fix this we say that dfdl:contentLength(e) on complex-type infoset item e, is computed as if element e's alignment was first put into place along with the rest of the LeftFraming, and then the content length computed as if the complex content was unparsed starting from after that LeftFraming up to the final RightFraming In other words, the entire element has to be unparsed, not just the complex content of it, and then the part of that unparsed data that corresponds to the complex content is used as the return value from dfdl:contentLength. This works because element e's alignment (LeftFraming generally) is not within the complex content for element e. It is before the complex content of the element, so if the schema puts an explicit alignment on element e, then it will clearly be aligned that way when it is really unparsed, but the dfdl:contentLength function will also compute a length from the infoset value as if that same alignment were in place. I think that is about the best we can do. Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

Missing restriction in 'endOfParent' length kind ??
by Mike Beckerle 28 Apr '14

28 Apr '14

I think we're missing a restriction for 'endOfParent' Restriction: If an element has lengthKind 'endOfParent', then it's lengthUnits must be the same as the lengthUnits of the parent. SDE otherwise. Rationale: If a string has fixed length, and lengthUnits 'bytes' then currently we require the textStringPadCharacter to be a single-byte character. This prevents issues where there are bytes that can't be filled in by a pad character nor trimmed as a pad character. But if the string has 'endOfParent' length, the string could have lengthUnits 'characters', encoding utf-16, and would then typically have a 2-byte pad character. But the enclosing parent could have lengthUnits bytes and an odd fixed/specified length. Hence, a byte could be left over that cannot be filled with a pad character, nor trimmed as a pad character. It is noisome to have this corner case/excess byte problem for 'endOfParent' when we have so cleverly dodged it for all the specified-length kinds with the restriction to single-byte pad characters when lengthUnits is bytes. There is language in the spec about dealing with these trailing RightFill or ElementUnused regions. With the above restriction I believe RightFill becomes associated only with the hexBinary type being extended to fill out a specified-length or endOfParent box. The much more complex issue of RightFill appearing after text characters remains, but outside of the context where padding/trimming are involved. It is really problematic language. Frought with possibilities for misinterpretation. To me it is very preferable to just make 'endOfParent' consistent with other constraints associated with padding for specified-length elements. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call 2014-04-22 - cancelled
by Steve Hanson 21 Apr '14

21 Apr '14

Sorry have to cancel the call, but I am unable to make it. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: DFDL Working Group Call
by Steve Hanson 21 Apr '14

21 Apr '14

Steve unable to make call

1 0

Call for input to the US Department of Education on data APIs
by Sill, Alan 15 Apr '14

15 Apr '14

David, Mike, Steve et al., The US Department of Education is asking for input on the use of APIs to increase innovation, transparency, and access to data. I thought that the participants in the DFDL working group could work together to put together an organized response. To weigh in, you can email APIRFI(a)ed.gov by June 2, or send your input via other addresses as detailed in the online notice. Please see the link below for additional information. Topic: How Can the Department of Education Increase Innovation, Transparency and Access to Data? | ED.gov Blog Link: http://www.ed.gov/blog/2014/04/how-can-the-department-of-education-increase… Let me know if you would like any help with this. Alan

1 0

OGF DFDL WG Call Minutes 2014-04-15
by Steve Hanson 15 Apr '14

15 Apr '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13240?download=. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 242 - valueLength and contentLength function wording
by Steve Hanson 15 Apr '14

15 Apr '14

I had a quick look at the document and Steve's review comments. The text tries to define valueLength() in terms of the info set value, or states that it is 'calculated from' the infoset value. I believe this will mislead many readers ( it already has done, in fact ). People just assume that the 'value length' is 'the length of the logical value' and they don't think any more deeply about it. So it is absolutely essential to state clearly that this function returns the length of a region in the data stream. It may be simpler to define the behaviour in terms of the length of the simpleContent or complexContent region, minus any padding. The other point is around the bytes/characters issue. Mike's words, or something similar, are definitely required because we don't mandate that the encoding must be consistent throughout a complex type. Nor do we prohibit a mixture of character and non-character content in a complex type. And someone might even call valueLength() with lengthUnits 'characters' on a complex type that contains no character data at all. In principle it would be possible for the value to be available while parsing ( for elements that follow the specified element ). If we are disallowing this then it should be stated very clearly somewhere. And probably is already. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 ----- Forwarded by Tim Kimber/UK/IBM on 15/04/2014 15:28 ----- From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 15/04/2014 15:01 Subject: Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Sent by: dfdl-wg-bounces(a)ogf.org Review comments added: Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 11/04/2014 14:04 Subject: Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Sent by: dfdl-wg-bounces(a)ogf.org Revised Action 242 proposed changes word doc attached. I have incorporated the discussion in this thread (I hope.) Please review. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Mar 25, 2014 at 10:56 AM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: This language is consistent with what we say for lengthKind pattern in section 12.3.5: "When unparsing, the dfdl:valueLength of a complex type element when the length units is 'characters' is computed as if the entire structure was unparsed into a temporary data stream beginning at position 1, and then this data stream is considered to be text in the character set encoding specified by the dfdl:encoding property, regardless of the actual representation of the complex type element or the elements contained within it. The number of characters in this temporary data stream is the value length of the complex type." The behavior of the IBM DFDL implementation for valueLength is as described is consistent with the above, excepting that it will not detect a decode error, and it gives an SDE (?) if the encoding is not fixed width. Since we have decided not to require that a complex type element is recursively all text all the way down, I believe we have to tolerate implementations having different behaviors in the potentially meaningless cases where there is binary data or encoding changes in the complex type. So I would add to the above suggested language this: "However, if creation of this data stream would cause an encoding error, or parsing of this data stream as characters would cause a decoding error, then the behavior and return value of dfdl:valueLength are implementation dependent." Looking at the DFDL spec, I am concerned that we never really say what we mean by the "length of the ComplexContent region." (Last sentence before Table 7 in section 12.3.7) Section 12.3.7.3 doesn't do it. The dfdl:valueLength function may be the first place where we have to actually say how the various sub-regions contribute to the ComplexContent region's length. I believe this is the obvious "sum of length of all contained regions", but keep in mind that alignment region lengths will vary depending on the starting alignment, so the length is, in general, dependent on the position within the bit stream. Hence when unparsing we have to specify that the dfdl:valueLength is measured as if the ComplexContent region started at position 1 (as I did above) so that internal alignment regions can be given meaningful lengths. The general clarification should be added to 12.3.7.3, or to section 12.3.7 immediately before section 12.3.7.1. Something like this: "The length of the ComplexContent region is the sum of the lengths of the contained regions. However, note that alignment regions inside the ComplexContent may be of different lengths depending on the ComplexContent's starting position alignment." Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Mon, Mar 24, 2014 at 11:34 AM, Andrew Edwards <andy.edwards(a)uk.ibm.com> wrote: Steve (et al) - Resending as the last one bounced. I'll usurp Tim and respond :) Currently the IBM implementation insists on using a fixed-length encoding and returns an "unsupported" error message for a variable width encoding. With a fixed width encoding, we "do the maths" using the bytes-per-character and the bytes written by this complex element. HTH, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Steve Hanson/UK/IBM 24/03/2014 14:52 To "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, cc Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Link Note errata 3.9, my bolding: "3.9. Section 12.3.5, 7.3.1, 7.3.2. The spec originally allows lengthKind ‘pattern’ to be used when the representation of the current element, or of a child element, is binary, but imposes restrictions on the encoding that can be in force. Clarify that the encoding property must be defined for the element (else schema definition error), and that a decoding processing error is possible if the match of the regex encounters data that does not decode in that encoding, dependent on the setting of encodingErrorPolicy. Remove section 12.3.5.1. Same clarifications needed for testKind ”pattern” property for asserts and discriminators. For consistency, the restriction that a complex element of specified length and lengthUnits ‘characters’ must have children that are all text and that have the same encoding as the complex element, is dropped." That's the restriction that I was referring to in my comment below. I can see why it was dropped - basically the parser now just tries to decode n characters using the complex element's encoding (and encodingErrorPolicy). We could apply the same principle for dfdl:valueLength & dfdl:contentLength - you build the stream from the bottom up, and then decode it using the complex element's encoding (and encodingErrorPolicy ?) to get the length in characters. Note that's how unparsing for lengthKind 'prefixed' with lengthUnits 'characters' would work as well - the spec just says "For a complex element, the length is that of the ComplexContent region" which is not sufficient (12.3.4). Similar deal for lengthKind 'explicit' - in order to know the size in chars of ElementUnused the unparser needs to know the size in chars of the data first (12.3.7.3). (Of course, for a fixed width encoding, you don't need to decode, you can just do the maths, but for the general case you need to decode. Also just doing the maths does not take encodingErrorPolicy into account). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 24/03/2014 12:55 Subject: Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Mike 23.5.3.1. Value length is only a function of the dfdl:encoding property if the element has a text representation. Not sure this needs to be (re)stated here. 23.5.3.1. "The value length is computed from the DFDL infoset value, ignoring the dfdl:length or dfdl:textOutputMinLength property. Other DFDL properties which affect the length of a text or binary representation are respected, it is only an explicit length which is ignored." Last sentence is too imprecise - should be phrased in terms of the grammar. 23.5.3.1. "If the second argument is 'characters' then the element must have text representation and it is a schema definition error otherwise". Yes but only for a simple type, so should be qualified. 23.5.3.1. "If the second argument, giving the length units, is 'characters', then recursively, this complex type element must have text representation throughout all its contained elements and framing, all of which must also use a uniform character set encoding." I can't see that restriction elsewhere in the spec when it talks about length of ComplexContent and lengthUnits 'characters' - I was expecting it to be in section 12.3.4 or 12.3.7.3 which face the same issue - but it isn't. Did we decide not to have this restriction? Without such a restriction, how does the unparser come up with a meaningful length (unless it re-parses)? (Tim - what does IBM DFDL do here?) What about delimiters and padding of children that use %#r entities? 23.5.3.2. The points in 23.5.3.1 about escape characters, length as a function of encoding, and bottom up for complex elements, apply equally to 23.5.3.2. It might be easier just to say in 23.5.3.2 that dfdl:contentLength for complex elements is same as dfdl:valueLength, and for simple elements differs only by the additional inclusion of LeftPadding and RightPadOrFill regions. Also noted in passing: Specified length - An item has specified length when dfdl:lengthKind is "implicit", "explicit", or "prefixed". should be Specified length - An element has specified length when dfdl:lengthKind is "implicit" (simple type only), "explicit", or "prefixed". Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 20/03/2014 17:21 Subject: [DFDL-WG] Action 242 - valueLength and contentLength function wording Sent by: dfdl-wg-bounces(a)ogf.org See attached doc which is proposed revisions to section 23.5.3 Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy [attachment "Action-252-DFDL-Functions-23.5.3.docx" deleted by Andrew Edwards/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "Action-252-DFDL-Functions-23.5.3.docx" deleted by Steve Hanson/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording
by Andrew Edwards 15 Apr '14

15 Apr '14

Steve (et al) - Resending as the last one bounced. I'll usurp Tim and respond :) Currently the IBM implementation insists on using a fixed-length encoding and returns an "unsupported" error message for a variable width encoding. With a fixed width encoding, we "do the maths" using the bytes-per-character and the bytes written by this complex element. HTH, Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Steve Hanson/UK/IBM 24/03/2014 14:52 To "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, cc Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Note errata 3.9, my bolding: "3.9. Section 12.3.5, 7.3.1, 7.3.2. The spec originally allows lengthKind ‘pattern’ to be used when the representation of the current element, or of a child element, is binary, but imposes restrictions on the encoding that can be in force. Clarify that the encoding property must be defined for the element (else schema definition error), and that a decoding processing error is possible if the match of the regex encounters data that does not decode in that encoding, dependent on the setting of encodingErrorPolicy. Remove section 12.3.5.1. Same clarifications needed for testKind ”pattern” property for asserts and discriminators. For consistency, the restriction that a complex element of specified length and lengthUnits ‘characters’ must have children that are all text and that have the same encoding as the complex element, is dropped." That's the restriction that I was referring to in my comment below. I can see why it was dropped - basically the parser now just tries to decode n characters using the complex element's encoding (and encodingErrorPolicy). We could apply the same principle for dfdl:valueLength & dfdl:contentLength - you build the stream from the bottom up, and then decode it using the complex element's encoding (and encodingErrorPolicy ?) to get the length in characters. Note that's how unparsing for lengthKind 'prefixed' with lengthUnits 'characters' would work as well - the spec just says "For a complex element, the length is that of the ComplexContent region" which is not sufficient (12.3.4). Similar deal for lengthKind 'explicit' - in order to know the size in chars of ElementUnused the unparser needs to know the size in chars of the data first (12.3.7.3). (Of course, for a fixed width encoding, you don't need to decode, you can just do the maths, but for the general case you need to decode. Also just doing the maths does not take encodingErrorPolicy into account). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 24/03/2014 12:55 Subject: Re: [DFDL-WG] Action 242 - valueLength and contentLength function wording Mike 23.5.3.1. Value length is only a function of the dfdl:encoding property if the element has a text representation. Not sure this needs to be (re)stated here. 23.5.3.1. "The value length is computed from the DFDL infoset value, ignoring the dfdl:length or dfdl:textOutputMinLength property. Other DFDL properties which affect the length of a text or binary representation are respected, it is only an explicit length which is ignored." Last sentence is too imprecise - should be phrased in terms of the grammar. 23.5.3.1. "If the second argument is 'characters' then the element must have text representation and it is a schema definition error otherwise". Yes but only for a simple type, so should be qualified. 23.5.3.1. "If the second argument, giving the length units, is 'characters', then recursively, this complex type element must have text representation throughout all its contained elements and framing, all of which must also use a uniform character set encoding." I can't see that restriction elsewhere in the spec when it talks about length of ComplexContent and lengthUnits 'characters' - I was expecting it to be in section 12.3.4 or 12.3.7.3 which face the same issue - but it isn't. Did we decide not to have this restriction? Without such a restriction, how does the unparser come up with a meaningful length (unless it re-parses)? (Tim - what does IBM DFDL do here?) What about delimiters and padding of children that use %#r entities? 23.5.3.2. The points in 23.5.3.1 about escape characters, length as a function of encoding, and bottom up for complex elements, apply equally to 23.5.3.2. It might be easier just to say in 23.5.3.2 that dfdl:contentLength for complex elements is same as dfdl:valueLength, and for simple elements differs only by the additional inclusion of LeftPadding and RightPadOrFill regions. Also noted in passing: Specified length - An item has specified length when dfdl:lengthKind is "implicit", "explicit", or "prefixed". should be Specified length - An element has specified length when dfdl:lengthKind is "implicit" (simple type only), "explicit", or "prefixed". Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 20/03/2014 17:21 Subject: [DFDL-WG] Action 242 - valueLength and contentLength function wording Sent by: dfdl-wg-bounces(a)ogf.org See attached doc which is proposed revisions to section 23.5.3 Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy [attachment "Action-252-DFDL-Functions-23.5.3.docx" deleted by Andrew Edwards/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 3