- dfdl-wg - lists.ogf.org

Actions 290 & 291 - variable width characters and utf16Width 'variable'
by Mike Beckerle 08 Nov '16

08 Nov '16

I verified that Java's representation of unicode codepoints above U+FFFF is (a) an int - for a unicode character code handled outside of a string. (b) a pair of surrogate codepoints when represented in a java String There are a variety of methods now that take or return an int which can be up to U+10FFFF, and which interact with either 1 or 2 character codepoints of a String. For utf-16: A character requiring more than 16 bits is represented as 2 code units of a surrogate pair, and each of those becomes a Java character. When utf16Width is 'variable' then this surrogate pair counts as 1 unicode character for length in 'characters' purposes. This is the feature I believe should be optional in DFDL. For utf-8: A character requiring more than 16 bits is also represented as 2 code units of a surrogate pair. However, we have no property for indicating this surrogate pair counts as 1 unicode character for length in 'characters' units purposes. For utf-32: Same issue. A single codepoint in utf-32 may have to be represented as a surrogate pair in a java string. However, we have no property for indicating this surrogate pair counts as 1 unicode character. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2016-11-08
by Steve Hanson 08 Nov '16

08 Nov '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13545?download= Next call Nov 22nd Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL Working Group call
by Steve Hanson 08 Nov '16

08 Nov '16

Confirmed: OGF DFDL Working Group call 08/11/2016 - (Repeats) Location: IBM: Hursley DE3S09 Steve Hanson has confirmed this meeting Repeats This entry repeats Required: mbeckerle(a)tresys.com, mbeckerle.dfdl(a)gmail.com Optional: Alex Wood1/UK/IBM@IBMGB, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, jorge.marizan(a)gmail.com, Mark Frost/UK/IBM FYI: Calls for 2016. OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 Netherlands Toll-Free 0-800-363-6036 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg DFDL Getting Started: http://www.ibm.com/developerworks/library/se-dfdl/ DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2016-11-08
by Steve Hanson 08 Nov '16

08 Nov '16

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13544?download= Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

clarification - lengthKind pattern IS allowed for hexBinary?
by Mike Beckerle 31 Oct '16

31 Oct '16

We have a restriction in Daffodil that doesn't allow lengthKind pattern for hexBinary, but I recently reviewed the spec and can find no such restriction. Did I miss the restriction somewhere else? I find no occurrences of 'pattern' nor 'hexbinary' in the current errata document. I would say it is inconsistent to disallow lengthKind pattern for hexBinary, after all, we allow lengthKind 'delimited' which scans the binary bits/bytes, and that means dfdl:encoding is needed for hexBinary when lengthKind is 'delimited', so we're already allowing the other things we would need to support lengthKind 'pattern' also. We do have a workaround which is to use a string with encoding iso-8859-1, but that just proves the point that really there is no reason why hexBinary doesn't allow this. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Re: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group
by Steve Hanson 11 Oct '16

11 Oct '16

New issue raised - https://redmine.ogf.org/issues/322 - words added Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 14/09/2016 08:47 Subject: Re: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group ...and also missed this one. Next call ... meantime if you agree with my conclusion then please feel free to compile some extra words for the spec. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 07/09/2016 10:28 Subject: Re: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group I agree. Btw the language was added by several erratum so is fairly new. Perhaps what is missing is some words in the spec that state the resolved set of format annotation properties for a simple element describe SimpleElement in the data syntax grammar, and so on. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 06/09/2016 20:47 Subject: Re: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group Perhaps this is a minor clarification I need. Perhaps this is obvious. The language you excerpted probably pre-dates the data syntax grammar in the spec. What grammar regions does point (4) " dfdl:sequence or dfdl:choice or dfdl:group following property scoping rules" apply to? dfdl:group for group references are non-issue since those turn into sequence or choice. I think the answer is perhaps, obvious in retrospect.: it is the Sequence or Choice regions, and for simple types it is SimpleElement (not SimpleContent) for complex elements it is ComplexElement (not ComplexContent). That way it is clear that in every case, framing, alignment regions etc., are included. So for sequences, what textually appears above/before the dfdl:setVariable statement is framing properties of the sequence (in short form, or in long form if the dfdl:sequence element is placed above the setVariable statemnts). Those can use runtime-valued expressions which refer to the variable, and will see the variable after the set has executed. This is disconcerting only because of the lexical ordering mis-match with the order of execution. However, the same is true for asserts and discriminators using expressions. Those execute after the setVariables, no matter where they appear in lexical order vis a vis the setVariable statement annotations. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 6, 2016 at 12:58 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: Mike >From section 9.5: For sequences, choices and group refs: 1. dfdl:discriminator or dfdl:assert(s) with testKind 'pattern' (parsing only) 2. dfdl:newVariableInstance(s) - in lexical order, innermost schema component first 3. dfdl:setVariable(s) - in lexical order, innermost schema component first 4. dfdl:sequence or dfdl:choice or dfdl:group following property scoping rules 5. dfdl:discriminator or dfdl:assert(s) with testKind 'expression' (parsing only) This says that a setVariable on a group is processed before the format annotations of the group, so your example would be allowed. For discriminators and asserts, when they are executed depends of the testKind. If 'pattern' they are executed first. If 'expression' they are executed last. All seems clear to me - am I missing some subtlety somewhere? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 02/09/2016 16:56 Subject: [DFDL-WG] clarification needed: setVariable - executes before or after framing of the element/model group Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> The spec defines where setVariable executes as a statement relative to element processing and model-group processing. Question is: what does element and model group processing mean: the value region of the element, the content region of the element, or the whole element including all framing? So, for an element, spec says that setVariables run after the element. Rhetorical question: Does "after" mean after the terminator (including any expression for the terminator) has completed? Yes, because otherwise a element could contain the value of its own terminator, and that wouldn't work for lengthKind='delimited' as you would need the value to get the terminator to parse the value. So 'after' has to mean after the terminator, meaning all framing, and any other expression that can affect the length or interpretation in forming the element. For a model group, the spec says that setVariables run before the model group. So, can a sequence carry a set-variable that sets a variable value that is then used in the initiator property of that very sequence? E.g., <xs:sequence dfdl:initiator="{ $init }"> <xs:annotation><xs:appinfo ...> <dfdl:setVariable ref="init">{ ../someplace/else }</dfdl:setVariable> </xs:appinfo></xs:annotation> ..... reset of sequence </xs:sequence> If this is not allowed for the initiator, what about for the terminator or separator, or outputNewLine or encoding or any other property that affects the syntax of the model group? What about discriminators/asserts. Do those run before the sequence's initiator is matched, or after? E.g., can a discriminator indicate "true", but then the initiator fails to match? When unparsing, you don't execute discriminator/assert at all, so no guidance there. Maybe this is clear already, but I am not sure if setVariables should run before any framing of a model group, or after the framing, but before the content region, or before the value region. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2016-10-11
by Steve Hanson 11 Oct '16

11 Oct '16

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13544?download= Next call Oct 25th Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2016-10-11
by Steve Hanson 10 Oct '16

10 Oct '16

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13544?download= Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Announcement: Daffodil 2.0.0 Release Candidate 1 - Release to Test
by Steve Lawrence 29 Sep '16

29 Sep '16

We are pleased to announce the first release candidate of Daffodil 2.0.0! Notable changes since Daffodil 1.1.0 include unparse achieving full feature parity with parse, support for additional unparse only properties such as outputValueCalc, fillByte, truncateSpecifiedLengthString, etc., numerous bug fixes in parse, and improved parse performance. The release candidate has been tagged as 2.0.0-rc1 in the git repo: https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil… The release is also available for downloaded from the Daffodil Artifacts page: https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&versio… Daffodil will now enter the test phase, which we expect to last approximately 3 weeks, followed by the official release of Daffodil 2.0.0. Please give the release candidate a test and let us know if you find any issues. To report bugs or ask any questions, the development team is available on the Daffodil users mailing list: http://oss.tresys.com/mailman/listinfo/daffodil-users as well as XMPP HipChat: http://hipchat.ncsa.illinois.edu/gvZdmJHmq or via the JIRA Daffodil bug tracker: https://opensource.ncsa.illinois.edu/jira/projects/DFDL/summary On behalf of the Daffodil team, - Steve

3 2

OGF DFDL WG Call 2016-09-27 - cancelled
by Steve Hanson 27 Sep '16

27 Sep '16

Steve not available so cancelling. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0