- dfdl-wg - lists.ogf.org

Fw: stop bit encoding
by Steve Hanson 23 Sep '14

23 Sep '14

Agreed to defer to future DFDL 1.1. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/09/2014 17:37 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 16/07/2014 08:51 Subject: Re: [DFDL-WG] stop bit encoding That's correct. This is the example of stop-bit encoding that I have referred to in the past. It's actually the FAST (FIX Adapted for STreaming) protocol, and not the standard FIX format. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 15/07/2014 18:34 Subject: [DFDL-WG] stop bit encoding Sent by: dfdl-wg-bounces(a)ogf.org The FIX protocols (financial interchange) make use of stop bit encoding. I have heard of other places using this also. This is modern, current, non-legacy usage. This is the length encoding where the most-significant bit of a byte is used to determine whether there are more ASCII characters or not. The last byte will have the most-significant bit of 1, prior bytes most-significant bit of 0. Minimum length in this length kind is 1. It seems we are missing this lengthKind, which applies only to text in the ASCII/US-ASCII encoding. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF spec updates - planning for errata vs. editorial vs. respins
by Mike Beckerle 23 Sep '14

23 Sep '14

Below are the relevant 3 paragraphs from: http://www.ogf.org/Public_Comment_Docs/Documents/2007-10/draft-gwdc-gfsg-c1… Our existing errata process divides into editorial things (which we just fix), minor tech issues, major tech issues which is consistent with the below. Note that the discussion suggests not a separate errata document, but that the errata must be added to the spec itself - and the notion that they are listed on the cover sheet means they are anticipating only a relatively small set of such errata compared what has happened since 2013 Sept. To me this suggests such errata are not suitable for anything one can anticipate as a "parked issue" for the future. Such things must be in a new revision of the standard. Editorial fixes applied within the first month after publication will generally not be publicly announced to the OGF community. Any other changes to a document, whether major or minor, will be announced using the same mechanism as for a newly published document (i.e., email to the OGF community, posting to the OGF Web site, etc.). Whenever a document is updated, even for minor editorial updates, the document header will be adjusted to reflect the date the document was updated. The date of the update should appear in the upper right side of the document's first page, beneath any prior date. For very minor updates, especially those within the first month after publication, no additional information needs to be added to the document. A publicly accessible archive of all old versions of the document is maintained, and accessible through the same means as current documents. For all other updates, minor and major technical fixes, an errata report must be added to the document itself. Recommended practice is to put an errata report labeled as “Document Change History” on the first page of the document, directly under the copyright statement and before the abstract. A brief report on what was changed is sufficient, along with the date. If a more discussion is desired, the Document Change History can refer to a later section where the update is discussed in detail. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Cancelled: DFDL Working Group Call
by Steve Hanson 23 Sep '14

23 Sep '14

Steve on vacation

1 0

Fw: A gap in the behaviour for when escapeKind=escapeBlock
by Steve Hanson 23 Sep '14

23 Sep '14

Agreed on processing error. Erratum raised, tracked here: http://redmine.ogf.org/issues/230 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/09/2014 15:40 ----- From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 09/09/2014 15:57 Subject: Re: [DFDL-WG] A gap in the behaviour for when escapeKind=escapeBlock I think that is still ambiguous. I prefer a processing error - data is not compliant with escape scheme. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Andrew Edwards/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 09/09/2014 15:44 Subject: Re: [DFDL-WG] A gap in the behaviour for when escapeKind=escapeBlock Sent by: dfdl-wg-bounces(a)ogf.org Now that's a really good catch. Is it sufficient to say the EEC is escaped if it is last in the data to avoid creating the escaped block end? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Mon, Sep 8, 2014 at 12:05 PM, Andrew Edwards <andy.edwards(a)uk.ibm.com> wrote: While working in the escapeBlock area, I seem to have found a gap in the definition for DFDL's behaviour when using escape blocks. >From section 13.2.1, relevant extracts are: When 'escapeBlock': On unparsing the entire data are escaped by adding dfdl:escapeBlockStart to the beginning and dfdl:escapeBlockEnd to the end of the data. The data is either always escaped or escaped when needed as specified by dfdl:generateEscapeBlock. If the data is escaped and contains the dfdl:escapeBlockEnd then first character of each appearance of the dfdl:escapeBlockEnd is escaped by the dfdl:escapeEscapeCharacter. and On parsing the dfdl:escapeBlockStart string must be the first characters in the (trimmed) data in order to activate the escape scheme. The dfdl:escapeBlockStart string is removed from the beginning of the data. Until a matching dfdl:escapeBlockEnd string (that is, one not preceded by the dfdl:escapeEscapeCharacter) is found in the data, any in-scope terminating delimiter encountered in the data is not interpreted as such, and any dfdl:escapeEscapeCharacters are removed when they precede an dfdl:escapeBlockEnd string. Now consider a a model where: escapeBlockStart="start" escapeBlockEnd="end" escapeEscapeCharacter="#" Then take a logical value of: A hash is a # When we serialize the value, we wrap the value with the escapeBlockStart and escapeBlockEnd, and we preceed any instance of the escapeBlockEnd within the data with an escapeEscapeCharacter. This then gives us the physical value "startA hash is a #end". If we were to parse that data, we see the "#end" as an escaped escapeBlockEnd and report that there is no escapeBlockEnd. The gap in the behavioural definition seems to be that the specification makes no claim to do anything to escape an instance of an escapeEscapeCharacter when serializing; There is nothing to catch the case of an escapeEscapeCharacter that isn't escaping an escapeBlockEnd but ends up doing it by circumstance. Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Fw: OR operator in DFDL Expressions - what does IBM DFDL do?
by Steve Hanson 23 Sep '14

23 Sep '14

Mike IBM DFDL compiles XPath expressions into an instruction set. When it processes the instruction set, if it encounters an OR then the first true operand causes the expression to return true. However the rest of the instruction set is processed, but the results (including any errors) are ignored. Same for AND with first false operand. That's our Java behaviour. But it looks like our C behaviour is different in that when the rest of the instruction set is processed, any errors are not ignored !! Which is taking 'implementation-dependent' a little too literally :) XPath 2.0 spec here http://www.w3.org/TR/xpath20/#id-errors-and-opt says that the results of expressions are to a degree implementation-dependent, which is admitting that two implementations might return different results. In DFDL we already acknowledge that behaviour of implementations might be different, in the paragraph of section 23 that talks about dynamic versus static XPTY0004 errors. I don't think we should force implementations to implement their XPath processor to guarantee portability, because that has a knock-on effect on other criteria like performance, but we can certainly publish guidelines for schema authors so that they can author portable schemas. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ---- Reply ---- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 17/09/2014 09:05 Subject: [DFDL-WG] OR operator in DFDL Expressions Sent by: dfdl-wg-bounces(a)ogf.org This is really a question about how IBM's DFDL works, but I have to introduce the topic: XPath 2.0 says that implementations have freedom about whether these expressions cause errors or not: true() or error() = true() Let's call this the sequential semantics true() or error() = error Let's call this the parallel semantics error() or true() = true Let's call this the bizarre semantics There are analogous cases for 'and' with false() Saxon, which dates from XPath 1.0, implements the sequential semantics, which is required by XPath 1.0, and so that's what the various TDML tests and such that we have in the Daffodil project have come to depend on. Quite often we have things like dfdl:occursIndex() = 1 or ../r[dfdl:occursIndex() - 1]/flag where that 2nd operand is effectively an error if the first operand is true. Strictly by the new XPath 2.0 rules, the only portable way to write this expression is with an if-then-else. But I am reluctant to change all these tests we have. Part of me says the most conservative thing is the parallel semantics - because it prevents you from writing an OR statement like the one I have above that depends on the sequentiality. It is probably more important that the initial implementations are consistent so that schemas are more likely to interoperate. What does IBM DFDL do for the semantics of the OR expressions? Thanks Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2014-09-23
by Steve Hanson 22 Sep '14

22 Sep '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13356?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: DFDL Working Group Call (23 Sep 14:00 GDT in DE3S09)
by Steve Hanson 22 Sep '14

22 Sep '14

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg Moving to 14:00 UK (9:00 US Eastern) for just this week

1 0

DFDL Experience document 'Bit Order' is in public comment
by Steve Hanson 18 Sep '14

18 Sep '14

http://redmine.ogf.org/news/67 Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Catch 22.for EVDP & NVDP ?
by Steve Hanson 17 Sep '14

17 Sep '14

<xs:element name="choiceGroup" dfdl:lengthKind="implicit"> <xs:complexType> <xs:choice dfdl:choiceLengthKind="implicit" dfdl:initiatedContent="yes"> <xs:element name="myNum1" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:textNumberPattern="#0" dfdl:initiator="num:" dfdl:terminator="~" /> > <xs:element name="myText1" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="2" dfdl:initiator="text:" dfdl:terminator="~" /> </xs:choice> </xs:complexType> </xs:element> The above model is a choice of two elements, each of which is fixed length and has an initiator and terminator. Because the elements are fixed length, I must set dfdl:emptyValueDelimiterPolicy (EVDP) to be 'none' **. I also want to discriminate using the initiator, so I set dfdl:initiatedContent 'yes' on the choice, so I must set EVDP to 'initiator' or 'both' ++. (Same for NVDP if nillable). Catch 22. ** 'none' because the only valid empty representation is that there is nothing there at all. The element is fixed length X, so if the delimiters are present then the content must be length X and can't be zero length. ++ 'initiator' or 'both' because an initiator must always be present when dfdl:initiatedContent 'yes'. The implication is that I can't use dfdl:initiatedContent for this choice. While I can switch off nillable to avoid the error with NVDP, there is no other way of avoiding the EVDP error. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

referencing local element declarations from path expressions between two schema documents
by Mike Beckerle 17 Sep '14

17 Sep '14

I am looking for clarification about how path steps are to be interpreted with respect to the issue of qualified/unqualified element names, and default or no-default namespaces. If there is a precedent we can follow set by other tools which embed paths into XSD (e.g., schematron?) or even just the selector/paths things in XSD key/uniqueness constraints then I'd like to understand it. These are some of the conundrums: Two Schema Files: A, and B Schema file A has target namespace AA, and binds the AA namespace to prefix aa, and the BB namespace to prefix bb Schema file B has target namespace BB, and the same prefix bindings. Schema B has an element named b1 which has two local element declarations inside it named c1 and c2. Schema A has an element named a1 which contains an element reference to bb:b1. a1 has a DFDL annotation (e.g., dfdl:assert) which has a path expression in it. this path contains bb:b1/c1 When is the path correct? 1a) when schema B has elementFormDefault='unqualified' (that's the default), and schema A does NOT have a default namespace. Why? the default namespace - if it existed for schema A - would implicitly qualify the c1 step of the path, but since c1 is a local element in schema B with element form unqualified, it does not require (and cannot have) a namespace specified in path steps for c1. 1b) when schema B has elementFormDefault='qualified', and schema A has xmlns='BB'. Why? Because in this case the path bb:b1/c1 is equivalent to bb:b1/bb:c1, and since c1 will have a namespace-qualified name in schema B, that path will properly describe the path to the c1 element. 1c) Variation: when schema B has no target namespace (which I believe makes elementFormDefault irrelevant), and schema A has no default namespace. Because then c1 can't be qualfied as it lives in no namespace, and schema A has no implied qualifiers. When is the path incorrect? 2a) when schema A has xmlns="BB". Why? Because the default namespace means the bb:b1/c1 is equivalent to {BB}b1/{BB}c1, but there is no {BB}/c1 since c1 is just a local element declaration, and B has elementFormDefault unqualified. 2b) when schema B has elementFormDefault='qualified', and schema A has no default namespace. In that case the step c1 needs a namespace qualifier either by default, or explicitly via a prefix, but it doesn't have one. So I am interested in opinions on whether the above analysis is correct or not, and especially in precedent for how this stuff works in other tools. I have made a big assumption above which is that if xmlns="BB", then any path step inside something carrying that default binding is interpreted as qualified by that namespace. I picked this because it seemed reasonable, and it seems one can always modify one's schema to properly address something in another schema without having to change that other schema. I.e., it seems to have good composition properties. Thanks for any thoughts. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1