- dfdl-wg - lists.ogf.org

ambiguity - mandatory alignment for text (12.1.2)
by Mike Beckerle 11 Sep '13

11 Sep '13

These two paragraphs contradict each other. When processing textual data, it is a schema definition error if the dfdl:alignment and dfdl:alignmentUnits properties are used to specify alignment that is not a multiple of the encoding-specified mandatory alignment. If the data is not aligned to the proper boundary for the encoding when textual data is processed, then bits are skipped (parsing) or filled from dfdl:fillByte (unparsing) to achieve the mandatory alignment. So is it to be SDE (conservative), or just move over to the mandatory alignment? E.g., <xs:sequence dfdl:encoding="utf-8" dfdl:alignment="7" dfdl:alignmentUnits="bits" dfdl:initiator="hello">.... The alignment property says 7 bits, but the initiator is utf-8 which requires 8 bits. So we either SDE, or we just add additional alignment bits to align the initiator text. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

Latest draft of DFDL 1.0 specification
by Steve Hanson 11 Sep '13

11 Sep '13

Alan Since the DFDL 1.0 specification was published in Feb 2011, the two implementation teams (IBM and the Daffodil project) have identified a number of errata in the specification. These have been recorded in an errata document held on Redmine. The number of errata is currently at around 190, and include both clarifications to the specification and changes that affect an implementation, both major and minor. Typically as errata have been raised, the implementation teams include any implied changes in the next release of their implementation. Both implementation teams, and users of the two implementations, have requested that the DFDL 1.0 specification is revised to include all errata to date, so that the specification more closely reflects the implementations. Accordingly all errata to date have been incorporated into a new revision of the specification, which as a result has grown from 168 pages to 234 pages. This new revision of the specification supersedes the original. There is no implementation that exactly reflects the original as published on the OGF web site, they both adhere more closely to the new revision. The DFDL WG would therefore like to publish the new revision. The DFDL WG also recognises that there may be comments against the new revision, and that there may still be some errata undetected by the implementation teams, so that a further revision may be necessary in the future. Nonetheless it is important that the new revision in its current form is externally visible, and not just kept as an internal working document, as there are now many dozens of DFDL users, and they need an up-to-date specification. In particular, IBM DFDL wants to ship the HTML version of the new revision to IBM customers in its next release. We are looking to the OGF for guidance on how next to proceed. Would it be possible for the OGF to join our next DFDL WG call to discuss further? The call is at 16:00 UK (11:00 Eastern). Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2013-09-10
by Steve Hanson 10 Sep '13

10 Sep '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13109?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

2 2

OGF DFDL WG Call Minutes 2013-09-03
by Steve Hanson 05 Sep '13

05 Sep '13

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13108?download= Also attached as Redmine is playing up again: Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Direct dispatch choice clarifications (action 219)
by Steve Hanson 05 Sep '13

05 Sep '13

I am leaning away from the use of 'key' because XML schema has xs:key and xs:keyRef components and people might think they are related somehow. I think a good analogy is with the switch statement in languages. Suggestions: 1) 'dfdl:switch' on the choice and 'dfdl:case' or 'dfdl:switchCase' on the element - con is clash with dfdl:ignoreCase. 2) 'dfdl:switch' on the choice and 'dfdl:switchWhen'' on the element. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Date: 30/08/2013 15:06 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications (action 219) Updated errata 3.15 to answer the open comments against that errata. Also updated errata 2.126 to add the properties to the list of non-representation properties. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Date: 30/08/2013 00:10 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications (action 219) Updated errata 3.15, so please review that in the errata doc also. Corresponding changes are marked with OPEN bubbles, except in the Property precedence part. (Did those by global search/replace). Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Aug 21, 2013 at 12:21 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: Discussed this some more on the WG call. The original proposal from Steve envisaged a single property, choiceBranchRef, which was an expression that returned a QName. The attraction of this is that no property is needed on the element. It supports the use case where the 'tag' in the data corresponds closely to the element name, which is the case for something like SWIFT. Further, it is easy to extend this idea to xs:any wildcards in a future DFDL release. However if the 'tag' is very different from the element name then the expression can become a big 'if' statement and some of the performance benefit is lost. Hence the current proposal for choiceBranchRef expression returning a simple string, and the need for a property on elements. This gets us into a problem with global elements and uniqueness. What is needed for DFDL 1.0 is a mechanism that gives good performance for the known use cases, and is local to a choice, and does not preclude the provision of a QName based solution in a future DFDL release. What we have now is a halfway house. So the following is proposed: Change the name choiceBranchRef to choiceKey (or choiceTag or TBD). Note that the use of 'Ref' is dropped as that implies QName in all other properties that use 'Ref'. Change the name elementID to branchKey (or branchTag or TBD). Disallow branchKey on a global element, so it is allowed only on local elements and element refs. This removes the need for the special rule about choiceBranchKey on element ref overriding that on a global element. The mechanism is now entirely local to a choice, and will not clash with any future QName based scheme. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 19/08/2013 15:07 Subject: Fw: [DFDL-WG] Direct dispatch choice clarifications OK, I can go with changing elementID to something less suggestive of XSDL. I've spoken with Suman and changing it is not a big deal in our model. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/08/2013 14:35 ----- From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 17/08/2013 11:51 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications Sent by: dfdl-wg-bounces(a)ogf.org I'm in general agreement with that. Just to avoid any possible confusion: I think the important thing is to stop thinking of elementID as any sort of XSD/DFDL language ID. It is not. It is a DFDL String Literal, meaning but it must describe a simple string value and it's value is matched against the return value of the choiceBranchRef expression. data. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, "dfdl-wg-bounces(a)ogf.org" <dfdl-wg-bounces(a)ogf.org>, Tim Kimber/UK/IBM@IBMGB Date: 16/08/2013 18:54 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications I am ok with leaving the name elementID as is purely due to our schedule. However, I really do think it would be good to change it now. I think it is that name which is creating all the confusion, and we really ought to be more careful. I think the important thing is to stop thinking of elementID as any sort of XSD/DFDL language ID. It is not. It is a DFDL String Literal, meaning it's value is matched against data. For example: It could be dfdl:elementID="%NUL;%NUL;%NUL;%NUL;" meaning 4 null chars. So that when the choiceBranchRef="{ ../hdr/tag }" evaluates to a 4 character string, then if they are all nul the branch is taken by direct dispatch. This character NUL isn't allowed in any namespaced identifiers in XSD or DFDL. It isn't even allowed in XML. If I had dfdl:termnator="%NUL;" you wouldn't ask "what namespace is that %NUL; in?" So I think elementID is not, in anyway, a namespace-qualified identifier any more than a delimiter is. So I think uniqueness of the elementID within the alternatives of a choice should be the only requirement. This stuff about unique within a namespace if on a global element should be dropped. The other argument for changing the name from elementID to something more choice-dispatch specific is that this workgroup seems to be speculating about using elementID for some sort of future wildcard-oriented feature. I think we should reserve names we want for that future by NOT using them now. If we use them now, we lock down part of the semantics in a way which may just not work properly with some future addition to DFDL. Do we really want elementID which is about direct dispatch choices but is not an identifier, and then in the future have dfdl:wildcardElementID which is about wildcards and IS an identifier. That's a mess. If you think elementID is a really good name for some future wildcard feature, then you should be advocating to NOT use it now for direct dispatch choices. That is, unless you have a complete design in mind for the wildcard stuff and are confident that elementID can be overloaded in a backward compatible way. I've seen nothing like this articulated. ...mike On Fri, Aug 16, 2013 at 12:32 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: IBM has started to implement choiceBranchRef, in so far as the XML schemas for DFDL have been updated to include elementID and we have regenerated all our model code. I'd like to stick with elementID please. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: Tim Kimber/UK/IBM@IBMGB, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, " dfdl-wg-bounces(a)ogf.org" <dfdl-wg-bounces(a)ogf.org> Date: 16/08/2013 17:09 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications I think we need to stay out of identifiers and namespaces and qnames for this feature. The elementID should have to be locally unique within the alternatives of the choice that is dispatching to it. If that elementID lives on a global element declaration, that means nothing at all alone. In the context of a choice that has an element ref to that global element, then it has to be unique within the arms of that choice. This means it is possible to have two global element decls which have elementID="X", and there is no conflict unless they are both used via element refs from the same choice. I think this handles every use case I know of. The "namespace" requirement for elementID is only "unique within alternatives of a choice". Since nobody has implemented this feature yet, I would posit that we should change elementID to something less suggestive of an XSD identifier/QName-ish thing. Such as choiceDispatchTag. ...mike On Fri, Aug 16, 2013 at 5:18 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: You are right about the xs:QName constructor, it takes a string which is prefix plus name. If we supplied a dfdl:QName constructor that took a URI and a name, that would simplify things. For example, the choiceBranchRef expression for SWIFT would be as below. (The element name is always Document and the namespace is used to distinguish the different messages). {dfdl:QName(fn:concat(fn:concat('urn:swift:xsd:fin.', /FinMessage/Block2/MessageType), ".2011"), 'Document')} If we stick with elementID for DFDL 1.0, then I agree with your 3 bullets, but not the defaulting to element name as it is setting a behaviour that may not be where we want to go for 2.0. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 15/08/2013 20:34 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications Sent by: dfdl-wg-bounces(a)ogf.org The other problem with using a QName is that it involves using namespace prefixes. That means that there needs to be a mapping between prefixes and namespace URIs. I can see that getting very problematic if the choice group is located in a different xsd from the global elements that it references. I think we should - keep the elementID as a simple string - insist that all branches of a choice have different elementIDs - remove the global uniqueness constraint, for the reasons explained in this email chain I think it would be easier for modellers if the elementID defaulted to the local name of the element. I understand that name clashes can, in principle, occur. If users want to avoid that then they can be explicit about elementIDs and they could even define a naming convention for their elementIDs to make them look very much like QNames. Sounds like a lot of work, but DFDL models that are complex enough to need that approach will often be code-genned anyway. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 15/08/2013 18:37 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications Sent by: dfdl-wg-bounces(a)ogf.org The more I think it through, the more I see the use of a string elementID (or local element name) causing problems when/if we extend to support xs:any in the future. In my original proposal for direct dispatch choice I proposed that choiceBranchRef returned a QName which therefore automatically selected the element, and coped with any namespace issues. The problem with QName though is that the expression to build it can become a big case statement negating some of the performance gain, if there is no automap way of getting from the 'tag' to the QName. Hence why we introduced elementID. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Suman Kalia <kalia(a)ca.ibm.com>, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Tim Kimber/UK/IBM@IBMGB Date: 15/08/2013 17:20 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications Suman, comments to yours in pink Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia <kalia(a)ca.ibm.com> To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 15/08/2013 15:31 Subject: Re: [DFDL-WG] Direct dispatch choice clarifications Sent by: dfdl-wg-bounces(a)ogf.org comments in green Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Tim Kimber <KIMBERT(a)uk.ibm.com> To: dfdl-wg(a)ogf.org, Date: 08/15/2013 09:58 AM Subject: Re: [DFDL-WG] Direct dispatch choice clarifications Sent by: dfdl-wg-bounces(a)ogf.org See comment in <TK> tags. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 15/08/2013 12:57 Subject: [DFDL-WG] Direct dispatch choice clarifications Sent by: dfdl-wg-bounces(a)ogf.org Looking at this in more detail prior to writing up behaviour for section 9, there are a couple of things missing from the spec or that need clarification: 1) Description of elementID property should say that empty string is not allowed (this was in the erratum). 2) Should say that an elementID on an elementRef overrides any elementID on the global element (this was in the erratum). 3) Section 15.1.2 says that is a schema definition error if the elementId values of global elements are not unique within a given namespace. I don't see where namespace comes into this, the elementID is just a string so surely it needs to be unique across namespaces? (Strictly elementID needs only to be unique across the global elements involved in each specific choice, but it was minuted that global uniqueness was desirable to allow future xs:any support). <TK> In XML Schema, an xs:any does not, in general, match all global elements. The 'namespace' attribute can narrow the set to elements from a specified list of namespaces. There is no way in XML Schema 1.0 to further narrow the xs:any, So the rule is designed to ensure that future usage of xs:any when a single namespace is specified and processContents!='skip' does not throw up schema definition errors. However...I note that XML Schema 1.1 allows a new way to narrow the scope of an xs:any ( by specifying a list of not-included QNames ). My feeling is that the unique-within-namespace check is fragile. </TK> <SK> As per my recollection, we put the uniqueness rules across namespaces to accommodate chameleon namespaces. Consider a global element E1 in notarget namespace having elementID E1_ID and is included in 2 schemas with different target namespaces say TNS1 and TNS2.. Consider a choice containing element references TNS1:E1 and TNS2:E1, in order to disambiguate these elements in the context choice, the element ID has to be unique in the context of namespace. This is somewhat an edge case but can come more prevalent when the support for xs:any is provided. </SK> SMH: In the choice, the element refs to TNS1:E1 and TNS2:E1 both have an elementID string 'E1_ID' from the original E1 global element. In the choice, this is an error because the elementID is not unique in the choice (we match the result of the choiceBranchRef expression, which returns a string not a QName). The only way round this is to override the elementID on one of the element refs (see 2 above) and set a value that is unique. That then works. But that does not help the (future) xs:any scenario, where there is no element ref to carry the override. I think the chameleon namespace scenario will always cause a problem with xs:any because our elementIDs are strings not QNames. I think we should leave a global element uniqueness check out of DFDL 1.0. It doesn't actually future proof anything, as once I use xs:any the whole nature of the xsd changes. 4) Spec does not explicitly say that when choiceBranchRef is present each branch of the choice must have an elementID. This must be the case, as otherwise a choice branch will never be accessible. 5) Tim has suggested that if an element was silent about elementID, the local name of the element could be used instead. So conceptually an element would have an 'effective elementID'. This makes modelling easier if the 'tag' in the data is the same as the element name. <TK>...or if the element name is derivable from the 'tag' using a simple XPath expression</TK> SMH: True. The validation checks would need to ensure that the set of 'effective elementIDs' was unique; for the global element check as currently specified (see 3) this would mean that all global elements must have unique local names, unless an elementID is carried - I think this is too limiting. SMH: While defaulting to the local name sounds attractive, I can't convince myself that it won't cause problems if we add xs:any in DFDL 2.0 and multiple/chameleon namespaces are involved. SMH: Conclusion: For DFDL 1.0 we take the conservative position and say that you must specify an elementID on an element that is used in a choice with choiceBranchRef and it must be unique in the context of the choice only. No global uniqueness check is made. >From minutes of 17th April 2012. 145 Provide a 'dispatch' way of discriminating a choice for better performance of the envelope/payload use case (Steve, Mike, Suman) 12/7: See minutes. Need to choose a proposal and flesh out. 19/07: Waiting for proposals 26/07: Waiting for proposals 16/08: Waiting for proposals. Suman added to action. ... 1/11: Steve to send a proposal ... 21/03: Steve has sent a proposal. Mike has sent a counter proposal. Steve to respond. 28/03: Steve has sent a revised proposal. Review for discussion next week. Ensure proposal handles Mike's scenario where tag value to branch mapping is not 1-1. 05/04: Discussed Mike's review comments and Suman's concerns. Agreed that name should be elementID, should be a single DFDL String Literal value, and that matching of choiceBranchRef expression result should only be against elementID to avoid QName v String confusion. Steve to recirculate with a schema example. 17/04: Closed. Discussion on whether the choiceBranchRef expression should retiurn xs:string or something else. Agreed on xs:string. Discussed whether elemenID should be a pure xs:string or a DFDL String Literal. For consistency with other DFDL properties it should be a string literal, but raw byte entities and character classes should be disallowed to avoid complications. Discussed scope of uniqueness of elementIDs. Agreed that uniqueness is both local to a choice, and across all global elements in the same namespace (the latter is not strictly needed right now but accommodates any future addition of xs:any). Agreed that elementID should be on global element, local element, and element ref (in which case it overrides any elementId on the global element, which is ok as the property does not follow the usual scoping rules). Errata taken. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 5

Re: [DFDL-WG] First draft of appendix describing string literal matching
by Tim Kimber 03 Sep '13

03 Sep '13

Thanks for reviewing. 1. Let's drop tables 2 and 4 and replace with refs to the appendix, as suggested 2. Agreed 3. Good point. I think the intention of %ES; was that it should be used on its own. I don't see any point in allowing it to be a part of a non-zero-length DFDL string literal. So I think your modification to the grammar should be put into the spec. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Tim Kimber/UK/IBM@IBMGB Date: 02/09/2013 17:46 Subject: Re: First draft of appendix describing string literal matching Good description. My comments: 1) Apart from the first three rows, the grammar table is pretty much duplicating existing tables 2 and 4 in section 6.3.1. Suggest either that the table is dropped from the appendix and anything that is missing is added back into 6.3.1, or tables 2 and 4 are dropped and replaced by refs to appendix. I think the latter is preferable as everything is then in a single table. 2) There is a bug in the grammar for DfdlStringLiteral - there should not be '{' and '}' - that's expression syntax. 3) For recognising ES, you say "The string part is recognized if the data available for matching is zero-length". That's true if we insist that ES, if present, must be present on its own. I'm not sure we actually say that. If that is the intent, we should police this in the grammar. (Note IBM DFDL does not give an error if it find '%ES;abc' ). For 2) and 3) that would give: DfdlStringLiteral ::= (DfdlStringLiteralPart)+ | DfdlESEntity DfdlCharClassName ::= DfdlNLEntity | DfdlWSPEntity | DfdlWSPStarEntity | DfdlWSPPlusEntity It still needs an errata, as it is a change to the spec document. Needs references from 6.3.1. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Tim Kimber/UK/IBM@IBMGB, Steve Hanson/UK/IBM@IBMGB, Date: 30/08/2013 00:16 Subject: Re: First draft of appendix describing string literal matching I added this in current form as appendix D. Will be in draft r14.4. I did not create an erratum for this. It's a whole new section, not an error correction or clarificatino. But we can add one if we think it useful to point out this section. There are no cross references to this section currently in the document. We might find a few places we want to reference this from. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Aug 28, 2013 at 10:43 AM, Tim Kimber <KIMBERT(a)uk.ibm.com> wrote: Thanks Mike. I agree that the wording could be misinterpreted. Revised draft attached: regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Tim Kimber/UK/IBM@IBMGB, Cc: Steve Hanson/UK/IBM@IBMGB Date: 20/08/2013 17:33 Subject: Re: First draft of appendix describing string literal matching I'm not sure I agree with the algorithm in the 1.3 section for the string literal part "LiteralString". I believe this algorithm is independent of what encoding the schema itself is written in, i.e., what is on the <? xml encoding="..." ?> slug line at the top of the schema file. What you write in the schema file is read into memory, all characters are converted to unicode codepoints by way of that reading process. So these two statements in the Recognition Algorithm for LiteralString are of concern: "The characters in the DFDL schema will be encoded using the defined encoding for the schema in which they appear." I think this just muddies the waters. Elsewhere we should state that the encoding used when authoring a DFDL schema file does not affect the behavior of the schema. All schemas behave as if authored in utf-8, etc. "The recognition algorithm must be able to compare character sequences that are encoded using different encodings." To me that says if I write my schema in ebcdic, but the dfdl:encoding="ascii", that some algorithm other than mapping both into unicode codepoints first and then comparing them is needed. I don't think this is or should be true. I think the division of things into what you call string literal parts is needed due to raw byte, and due to character class entities. Outside of that I think translation of everything to unicode should be sufficient. ...mike On Thu, Aug 15, 2013 at 7:19 PM, Tim Kimber <KIMBERT(a)uk.ibm.com> wrote: Steve, Mike, Please take a look. Comments on high-level stuff like structure/level of detail are welcome. regards Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

OGF DFDL WG Call Agenda 2013-09-03
by Steve Hanson 02 Sep '13

02 Sep '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13108?download= Note later start time - 12 Eastern. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL WG Call (3 Sep 17:00 GDT in DE3S09)
by Steve Hanson 30 Aug '13

30 Aug '13

Rescheduled: OGF DFDL WG Call 03/09/2013 - (Repeats) Location: DE3S09 Steve Hanson has rescheduled this meeting. You have not added this entry to your calendar. Repeats This entry repeats Required: jcranford(a)mitre.org, mbeckerle(a)tresys.com, mbeckerle.dfdl(a)gmail.com, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: dfdl-wg(a)ogf.org, Stephanie Fetzer/Charlotte/IBM@IBMUS OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg

1 0

OGF DFDL WG Call Minutes 2013-08-28
by Steve Hanson 28 Aug '13

28 Aug '13

Please find minutes from the above two calls at http://redmine.ogf.org/dmsf_files/13107?download=. Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Action 217 - scannable - Re: clarification needed for scan and scannable
by Mike Beckerle 28 Aug '13

28 Aug '13

I propose this new erratum, and corresponding edits to erratum 2.9. In the draft r14.3 (to be circulated soon), I have OPEN comment bubbles to review the impact of this change, but I have edited this stuff in, as you really have to see it in action to see if it "works for you". *Errata 2.155* Sections 3, 7.3.1, 7.3.2, 12.3.5. Scan, scannable, scannable-as-text These terms all added to/changed in the glossary. Definitions removed from the prose. Scannable now means able to scan, which is natural. More specific term scannable-as-text used when we want the recursive requirement of uniform encoding. Errata 2.9 updated to use term scannable-as-text. On Wed, Aug 14, 2013 at 5:45 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Action 217 raised to decide new terminology for the regex scanning > encoding requirement. > > Regards > > Steve Hanson > Architect, IBM Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: Tim Kimber/UK/IBM@IBMGB, > Cc: Steve Hanson/UK/IBM@IBMGB > Date: 30/07/2013 01:01 > Subject: Re: clarification needed for scan and scannable > ------------------------------ > > > > I reviewed the spec. I have to reverse my prior statement. What you > describe is definitely currently allowed by the spec. The 'scannable' > restriction is reserved for lengthKind='pattern'. > > In my view this is a terrible idea, but it is where it is. I.e., that the > lengthKind 'delimited' doesn't require 'scannable' characteristic > throughout everything that is contained within it. It will not allow us to > really assist users in identifying what are very likely to be mistakes in > their schemas, because what we call 'delimited' is much too permissive. > > I am not going to argue to change this at the current time. This is just > due to the need to converge on the standard. My preference would be that > lengthKind 'delimited' requires scannability (as uniform text), and that > some new keyword be used to mean the current algorithm as specified. > > I do suggest that we rename the characteristic scannability to > scannable-as-text, in order to make it clearer what the requirement is, and > clarify that this is a requirement on the schema component. > > I suspect that lengthKind='delimited' will perhaps someday (DFDL v2.0?) be > deprecated and/or augmented by other keywords that are more specific such > as "delimitedText" which means scanning for delimiters over only text in > one encoding (the 'scannable' restriction), and other keywords like > "delimitedBinary" or "delimitedMixed" meaning formats which admit the more > powerful and complex things. > > My argument is simple: if people have something like delimiters in > distinct encodings appearing in their schema, it is most-likely due to a > human error (they missed one place when they were editing the encoding > property), rather than something as obscure as this > delimiters-in-a-different-encoding being a real characteristic of the data. > An SDE here will help them find this error. > > Furthermore, if you truly want a string to be terminated by either an > ebcdic comma (6B) or an ascii linefeed (0A), then you have a few > alternatives before you have to go to the now optional feature of > 'rawbyte', and which are consistent with scannability. > > First, use encoding ascii, and specify terminators of %LF;, and 'k' (or > %#x6B; since 'k' is 6B ascii), in which case the string is assumed to > contain only ascii code units. In that case, if ebcdic 'a' (x81 - illegal > in ascii) is encountered in the data, you will either get an error or a > unicode replacement character depending on encodingErrorPolicy. If you know > your data will only ascii-legal code units, then this is a good solution. > > I would note that your example (previously in the thread) did not specify > an encoding for the innermost string elements. Did you intend for those to > be ebcdic or ascii? > > Alternatively you can specify encoding ebcdic-cp-us, specify terminator of > comma (codepoint x6B) and %#x0A; which is the RPT control code in ebcdic > but not an illegal character. In that case the string can contain any legal > ebcdic code point. However, if code unit x70 is encountered (corresponds to > an ascii 'p', but unmapped in ebcdic-cp-us), you will either get an error, > or a unicode replacement character depending on encodingErrorPolicy. If you > know your data will contain only legal ebcdic code units, then this is a > good solution. > > Finally, you can specify encoding iso-8859-1, and terminators %#x6B; (or > 'k' which is 6B in iso-8859-1) and %#x0A; (linefeed in iso-8859-1). Then > any code units at all will be legal as all bytes have corresponding > character codepoints in iso-8859-1. If you have no idea what the data is, > but just want some sort of string out of it, and know only that the > terminators are these bytes, then this is a good solution. If your data > contains, for example, packed-decimal numbers, then this is the only way to > safely scan past it as a string, because both ebcdic and ascii have > unmapped code points from the code units that could appear in > packed-decimal number bytes. > > All the above are consistent with implementations that attempt to convert > data from code-units (bytes) to Unicode codepoints using a character set > decoder, and then apply any scanning/searching only in the Unicode realm, > and work in one pass over the data. > > I currently think that to implement both rawbytes and encodingErrorPolicy > it will require two passes over the data. I expect this overhead to be > negligible, so I'm not worried about it really. > > > ...mike > > On Mon, Jul 29, 2013 at 7:18 AM, Tim Kimber <*KIMBERT(a)uk.ibm.com*<KIMBERT(a)uk.ibm.com>> > wrote: > Hi Mike, > > Good - that was almost exactly the reply that I was expecting. I now > understand exactly where you are coming from, and how we arrived at this > position. > > First, a few statements that I think are true. I want to establish some > basic ground rules before we decide how to go forward: > a) it is desirable for DFDL ( more accurately, a valid subset of DFDL ) to > be implementable using well known parsing techniques. I think that pretty > much means regular expressions and BNF-style grammars. That implies that it > might be possible to implement a DFDL parser using one of the well-known > parser-generator technologies like Yacc/Bison/JavaCC/Antlr. I'm not > claiming that it *is* possible, but I think it would be a good thing if it > was. > b) It is technically feasible to implement a DFDL parser that can handle > the mixed-encoding example using regex technology. However, it would not be > a very efficient implementation because the scanner would have to scan the > data once per encoding, and it would have to do that for every character > after the end of field1. > c) It is possible to produce an efficient scanner that handles mixed > encodings. Such a scanner cannot use regex technology for scanning - in > fact, I think the only efficient implementation is to convert all > terminating markup into byte sequences and then perform all scanning in the > byte domain. This is what the IBM implementation does. > > The scenario in my example is not entirely far-fetched - it is conceivable > that the encoding might change mid-way through a document, and I think > Steve came up with a real-world format that required this ( I have a hazy > memory of discussing this a couple of years ago ). The requirement for the > raw byte entity is not for this case - it is for the case where the > delimiters are genuinely not characters ( e.g. a UTF-16 data stream > terminated by a single null byte ). However, it is not easy to come up with > realistic examples where the raw byte entity could not be translated into a > character before the processor uses it. I think that's where some of the > confusion has arisen. > > We have already agreed to make the raw byte entity an optional feature. We > should consider disallowing mixed encodings when lengthKind is delimited. > If we cannot do that then I agree with Mike that the descriptions and > definitions in the spec need to be made a bit clearer. > > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: *kimbert(a)uk.ibm.com* <kimbert(a)uk.ibm.com> > Tel. 01962-816742 > Internal tel. 37246742 > > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com> > > > To: Tim Kimber/UK/IBM@IBMGB, > Cc: Steve Hanson/UK/IBM@IBMGB > Date: 26/07/2013 22:49 > Subject: Re: clarification needed for scan and scannable > ------------------------------ > > > > Clearly we have to rework the description and definitions because my > understanding of the current spec would say your schema is not valid > because it is not scannable. Everything in the scope of the enclosing > element which is delimited must be scannable and that means uniform > encoding. > It is exactly to rule out this mixed encoding ambiguity that we have the > restriction. > > I have no idea how to implement delimiters without this restriction. > > I think the case you have here is what raw bytes are for. Though I am no > longer clear on how to implement rawbytes either. > > On Jul 25, 2013 6:40 PM, "Tim Kimber" <*KIMBERT(a)uk.ibm.com*<KIMBERT(a)uk.ibm.com>> > wrote: > Suppose I have this DFDL XSD: > > <*xs:element* name="documentRoot" dfdl:encoding="US-ASCII" dfdl:lengthKind > ="delimited" dfdl:terminator="]]]"> > <*xs:complexType*> > <*xs:sequence*> > <*xs:element* name="delimitedRecord" maxOccurs="unbounded" > dfdl:encoding="US-ASCII" dfdl:lengthKind="delimited"dfdl:terminator > ="%LF;"> > <*xs:complexType*> > <*xs:sequence* dfdl:separator="," dfdl:separatorSuppressionPolicy > ="suppressedAtEndLax" dfdl:encoding="EBCDIC-US"> > <*xs:element* name="field1" type="xs:string" dfdl:lengthKind= > "delimited"> > <*xs:element* name="field2" type="xs:string" dfdl:lengthKind= > "delimited" minOccurs="0"> > <*xs:element* name="field3" type="xs:string" dfdl:lengthKind= > "delimited" minOccurs="0"> > <*/xs:sequence*> > <*/xs:complexType*> > <*/xs:element*> > <*/xs:sequence*> > <*/xs:complexType*> > <*/xs:element*> > > > ...which will parse some data like this: > field1Value,field2Value,field3Value > field1Value, > field1Value,field2Value]]] > ...except that the commas that delimit the fields are in EBCDIC and the > linefeeds that delimit the records are in ASCII . > > I have purposely constructed this example with separatorSuppressionPolicy > set to 'suppressedAtEndLax'. This means that field1 and field2 could both > be terminated by either of > a) an EBCDIC comma or > b) an ASCII line feed > > This is a very artificial example, but it is valid DFDL and should not > produce schema definition error. If we use the term 'scan' when discussing > lengthKind='delimited' then we need to be careful how to define the term > 'scannable' - otherwise we might appear to prohibit things that are > actually valid. I think this is the same point that Steve was making. > > Most implementations will do the same as Daffodil and will use a regex > engine to implement all types of scanning, including > lengthKind='delimited'. It's the most natural solution, and it can be made > to work as long as > a) the implementation takes extra care if/when the encoding changes within > a component and > b) the implementation either does not support the raw byte entity, or only > supports it when it can be translated to a valid character in the > component's encoding. > > There is huge scope for confusion in this area - most readers of the DFDL > specification will assume that delimited scanning can be implemented using > regex technology + EBNF grammar. It may be possible ( I'm not claiming that > it is btw ), but the grammar would be decidedly non-trivial. so the DFDL > specification should take extra care to be unambiguous when discussing how > to scan for terminating markup. > > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: *kimbert(a)uk.ibm.com* <kimbert(a)uk.ibm.com> > Tel. 01962-816742 > Internal tel. 37246742 > > > > > From: Mike Beckerle <*mbeckerle.dfdl(a)gmail.com*<mbeckerle.dfdl(a)gmail.com> > > > To: Steve Hanson/UK/IBM@IBMGB, > Cc: Tim Kimber/UK/IBM@IBMGB > Date: 25/07/2013 19:04 > Subject: clarification needed for scan and scannable > ------------------------------ > > > > > If a user has read the Glossary and noted the definition of 'scannable' > then when he sees the terms 'scanning' and 'scanned' in 12.3.2 he may think > that this implies data being read by lengthKind must be 'scannable', and it > is not so. That's how I read the spec, hence my original comment. I find > it confusing to define 'scan' and then not to define 'scannable' as 'able > to be *scan*ned'. > Ok, so i am rethinking how we express what we mean by scan. > > scan - verb. The action of attempting to find something in characters. > Implies decoding the characters from code units to character codes. > > A scan can succeed on real data, or fail (no match, not found), but both > are normal behaviors for a scan. > > However, this is predicated on the data at least being meaningfully > decoded into characters of a single character set encoding. This is because > our regex technology isn't expected to be able to shift encodings mid > match/scan, nor is it expected to be able to jump over/around things that > aren't textual so as not to misinterpret them as code units of characters. > > So saying a schema component is scannable - means the schema expresses the > assumption that the data is expected to be all textual/characters in a > uniform encoding so that it is meaningful to talk about scanning it with > the kind of regex technology we have available today. That is, there's no > need to consider encoding changes mid match, nor what data to jump > over/around. > > This is the sense of "scan" -able that we use the term to mean. Our > expectation as expressed in the schema, is that so long as the data decodes > without error the scan will either succeed or fail normally. > > Actual data is scannable if it meets this expectation. > > Contrast with what happens for an assert with testKind='pattern'. In that > case we scan, but DFDL processors aren't required to provide any guarantee > that the data is scannable, which means the designer of that regex pattern > must really know the data, and know what they are doing, as to avoid the > issues of decode-errors. > > Why do we do this? Because we don't want to have to refactor a complex > type element into two sequences one of which is the stuff at the front that > an assert with test pattern examines, and the other of which is the stuff > after that. In some situations it may be very awkward or impossible to > separate out the part at the front that we need to scrutinize. Instead we > just allow you to put the assert at the beginning, and it is up to the > regex designer to know that the first few fields it looks at the data for, > are scannable (in the sense of the expectation of them being text), or if > not scannable in terms of that expectation from the schema, then at least > that the actual contents of them will not have data that interpreted as > character code points, will cause decode errors. In other words the writer > of a test pattern has to know that either the data is scannable (schema > says so), or that the actual data will actually be scan-able, in that > decode errors won't occur if it is decoded. > > This also provides a backdoor by which one can use regex technology to > match against binary data. But then you really have to understand the data > so as to understand exactly what code units will be found in it, so one can > understand what the regex technology will do. One can even play games with > encoding="iso-8859-1", where no decoding errors are possible, so as to > guarantee no decode errors when scanning binary data that really shouldn't > be thought of as text. > > The part of the data that a test pattern of an assert actually looks at > needs to be scannable by schema expectation, or scannable in actual data > contents. > > If you write a test pattern, expecting to match things in what will turn > out to be the first and second elements of a complex type, but the encoding > is different for the second element, then your test pattern isn't going to > work (you may get false matches, false non-matches, or may get decode > errors) because the assumption that the data is scannable for that first > and second element, is violated. > > Summary: can we just say > > scan - verb > scannable - a schema component is scannable if ... current definition. > Data is scannable with respect to a specific character set encoding if, > when it is interpreted as a sequence of code units, they always decode > without error. > > There are two ways to make "any data" scannable for certain. One - set > encodingErrorPolicy to 'replace'. Now all data will decode without error > because you get substitution characters insteead. > > Two - change encoding to iso-8859-1 or other supported encoding where > every byte is a legal code unit. > > If you are in neither situation One or Two above, then you have to say the > data is scannable, but with respect to a specific encoding. So saying the > data is scannable as ASCII means that it will not contain any bytes that > interpreted as ASCII code units will be illegal. For this example, that > means all the bytes have values from 0 to 127 (high bit not set). If you > know that will be true of your data, then even if it is binary data you can > say it is scannable as ASCII. > > The representation of Packed Decimal numbers are not scannable as ascii, > for example, because any high-nibble digit 8 or greater will cause the byte > containing it to have a value 128 or higher, for which there is no > corresponding ascii code unit. > > Packed Decimal numbers are also not scannable as UTF-8, because many > packed decimal bytes will not be legal utf-8 code unit values, nor will > they be in the ordered arrangements UTF-8 requires. > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > > -- > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | * > www.tresys.com* <http://www.tresys.com/> > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 3