July 2013 - dfdl-wg - lists.ogf.org

Fw: Unordered Groups : new draft
by Steve Hanson 22 Jul '13

22 Jul '13

Hopefully we can close on this week's WG call. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/07/2013 13:13 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> Date: 18/07/2013 16:07 Subject: Re: Unordered Groups : new draft Tim That looks great. I've made a few minor editorial changes so it can be dropped straight into the errata document and then the spec with no further tweaks. Also added that occurrences of the same member need not be contiguous, and replaced 'real infoset' with 'infoset implied by the original unordered group'. You didn't incorporate Mike's review comment? "If a member element is found to have the empty representation then the parsing of that element must use the original value of minOccurs. In this example, element "b" has minOccurs="0" and if it is found with the empty representation then it must not be defaulted." should perhaps be: "If a member element is found to have the empty representation, then the decision about whether a default value is to be computed is based on the original value of minOccurs, and the count of how many other occurrences of that member element have been accumulated so far by the parsing. In this example, element "b" has minOccurs="0" and if it is found with the empty representation then it must not be defaulted." Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com>, Date: 18/07/2013 11:38 Subject: Unordered Groups : new draft I've addressed Steve last round of comments, and improved the wording of the introductory paragraph. [attachment "Section 14.3 - Unordered Sequence Groups.doc" deleted by Steve Hanson/UK/IBM] regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

proposed dfdl:hexBinary function for hex constants in schemas
by Mike Beckerle 22 Jul '13

22 Jul '13

Separated out from prior proposal where it was mixed in with another topic. dfdl:hexBinary($arg as xs:anyAtomicType?) as xs:hexBinary? This function provides an extension to the behavior of the xs:hexBinary constructor. The argument can be a string, in which case the behavior is as the xs:hexBinary constructor. The argument can also be a long, unsignedLong, or any subtype thereof, and in that case a string containing hexadecimal digits is produced. The hexadecimal digits produced that are letters are always uppercase. The number of hex digits in the resulting string is a function of the input type. If byte or unsigned byte, exactly 2 hex digits are produced, for short and unsignedShort, 4 hex digits, and so on. The hex digits correspond to a big-endian representation of a twos-complement binary representation of the argument value. So: dfdl:short(xs:concat('x', dfdl:hexBinary(xs:short(208)))) eq xs:short(208) is true, and a corresponding tautology holds for all the other DFDL functions that construct integers from hexadecimal, if you replace the 208 above with a value in range for the corresponding numeric type. If the argument is a numeric literal, then the smallest signed integer type (long, int, short, byte) is selected that can contain the value, and the number of hexadecimal digits produced corresponds to that type. So: dfdl:hexBinary(208) eq 'D0' dfdl:hexBinary(-2084) eq 'F7FF' -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

3 3

Re: [DFDL-WG] DFDL regular expressions and Unicode - conformance
by Steve Hanson 22 Jul '13

22 Jul '13

Jonathan No need for us to contact ICU, as Andy indicates below ICU and Java both claim conformance. Here's the words from errata 3.29. Please can you rephrase to combine the conformance requirement and the restrictions, so that we end up with a form you are happy with, then we can update the errata? A DFDL regular expression is defined by a set of valid pattern characters. For portability, a DFDL regular expression pattern is restricted to the inclusive subset of the ICU regular expression [ICURE] and the Java(R) 7 regular expression [JAVARE] with the Unicode flags UNICODE_CASE and UNICODE_CHARACTER_CLASS turned on. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, "Cranford, Jonathan W." <jcranford(a)mitre.org> Date: 11/07/2013 14:19 Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode Hi Jonathan, Sorry for the delay; first week back in the office... As you've noted, errata 3.29 describes what DFDL regexes are supported. Specifically, it is a subset of Java 7's java.util.regex ( http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html) and ICU's regular expression support ( http://userguide.icu-project.org/strings/regexp) both of which conform with level 1 of Unicode technical standard #18 It looks like there are 2 stages to checking conformance: Logical - do the available regex constructs provide conformance to the technical standard. This is probably just a couple of hours of reading the Unicode standard rules and cross-checking the constructs in each matching engine. Actual - do Java 7 and ICU really match properly for each of the conformance statements. This can take an ever increasing amount of time testing various sets of data and regex patterns, and it risks the only reward being that we find bugs in Java 7 or ICU. Minimum would be 3 or 4 days of test generation. Does that answer the issue? Andy Andy Edwards - IBM Integration Bus - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE3 V17 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Steve Hanson/UK/IBM 08/07/2013 11:08 To "Cranford, Jonathan W." <jcranford(a)mitre.org>, cc "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Subject Re: [DFDL-WG] DFDL regular expressions and Unicode Jonathan I've copied Andy who added regexs support into IBM DFDL recently. He might have an idea as to the effort involved in stating conformance. We will discuss your other two emails on next DFDL-WG call or so. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 06/07/2013 00:56 Subject: Re: [DFDL-WG] DFDL regular expressions and Unicode Sent by: dfdl-wg-bounces(a)ogf.org Update: I just found errata 3.29, which answers this question, I think. >From the description in the errata, and looking at the documentation for java 7 regular expressions, it looks like DFDL regular expressions conform to level 1 of Unicode Regular expressions (UTS#18). I still think there would be value in stating such conformance in the DFDL spec, but I suppose that would take some legwork for someone to actually confirm the conformance of ICU and Java7 to level 1. Very respectfully, -- Jonathan Cranford >-----Original Message----- >From: Cranford, Jonathan W. >Sent: Friday, July 05, 2013 1:36 PM >To: dfdl-wg(a)ogf.org >Subject: DFDL regular expressions and Unicode > >I've been going through the spec recently, and I have a few questions about DFDL >regular expressions. > >Rather than put them into one long email, I'll break them up into separate emails. > >First question: What level of conformance to Unicode Technical Standard #18 >UNICODE > REGULAR EXPRESSIONS do DFDL regular expressions claim? > > For example, > * XML Schema regular expressions are "targeted at support of 'Level 1' >features" > (http://www.w3.org/TR/xmlschema-2/#dt-ccesN) > * Java 1.4 regular expressions "implement its second level of support" > ( http://docs.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.html) > * Perl 5.18 seems to implement most of Level 1 > ( http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression- >Support-Level) > > I think the conformance level should be specified in the DFDL spec so that it is >clear to schema > designers what a regular expression would really match against. Details > like case conversion and canonical equivalence make a difference when > matching against a Unicode string. > >Thanks in advance, > >-- >Jonathan W. Cranford <jcranford(a)mitre.org> >Senior Information Systems Engineer >The MITRE Corporation (http://www.mitre.org) -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg

2 2

Why do occursCountKind=implicit and separatorSuppressionPolicy=never collide with unbounded?
by Garriss Jr., James P. 19 Jul '13

19 Jul '13

The Content Type header can have multiple parameters, like so: Content-Type: text/plain; charset=UTF-8; name=file.txt I have defined the Content Type like so in DFDL: <xsd:element name="ContentType" dfdl:initiator="Content-Type:%SP;" dfdl:terminator="%CR;%LF;"> <xsd:complexType> <xsd:sequence dfdl:separator=";"> <xsd:element name="MimeType" type="xsd:string"/> <xsd:element name="Parameter" type="ParameterType" minOccurs="0" maxOccurs="unbounded" dfdl:occursCountKind="implicit"/> </xsd:sequence> </xsd:complexType> </xsd:element> When I run this against a Content Type with 0 or 1 parameters, everything is good. But when I run it with a Content Type header with two parameters, Daffodil throws this error: 2013-07-18 12:10:13:856 edu.illinois.ncsa.daffodil.compiler.ProcessorFactory Error[Schema Definition Error: separatorSuppressionPolicy='never' with occursCountKind='implicit' required bounded maxOccurs. Schema context: element.Parameter. Location line 23 column 180 in It seems that an element can't have separatorSuppressionPolicy='never' and occursCountKind='implicit' if it's unbounded. Why is this an issue? TIA

2 3

Fw: DFDL String Literal type
by Steve Hanson 18 Jul '13

18 Jul '13

We discussed the correct XML schema type for DFDL String Literal on the last WG call. I read up on xs:NMTOKEN - not appropriate as it is basically a name so does not allow the full range of characters we need. Then I looked at restricting xs:token, but I could not work out from the XML Schema 1.0 spec how whitespace facets were handled when other facets were present. So I asked Sandy, and got the very useful clarification below. Please review for next call. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 17/07/2013 16:01 ----- From: Sandy Gao/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Date: 17/07/2013 13:33 Subject: Re: DFDL String Literal type Hi Steve, Yes, that should work. All other facet checking, including pattern, happens *after* whitespace handling. This was made clearer in Schema 1.1, where "whitespace" is called a pre-lexical facet, and "pattern" etc. are called lexical facets. Thanks, Sandy Gao Source Code Monitoring (SCMon) IBM Canada sandygao(a)ca.ibm.com From: Steve Hanson/UK/IBM@IBMGB To: Sandy Gao/Toronto/IBM@IBMCA, Date: 2013-07-17 06:07 AM Subject: DFDL String Literal type Hi Sandy Please can I ask your advice on use of the whitespace facet in conjunction with the pattern facet? This is in order to model the correct data type for a DFDL String Literals. This is defined as: DFDL String Literal DFDL String Literals represent a sequence of literal bytes or characters which appear in the data stream. This presents the following challenges - the literal characters in the data stream might not be in the same encoding as the DFDL schema - it may be necessary to specify a literal character which is not valid in an XML document - it may be necessary to specify one or more raw byte values A DFDL string literal can describe any of the following types of literal data in any combination: - a single literal character in any encoding - a string of literal characters in any encoding - a bi-directional character string - one or more characters from a set of related characters ( e.g. end-of-line characters) - a literal byte value A DFDL string literal is therefore able to describe any arbitrary sequence of bytes and characters. Empty Strings: Empty string is not allowed as a DFDL string literal value unless explicitly stated otherwise in the description of a property. In this case the use of empty string provides some property specific behavior different from simply using the empty string as a value. When the empty string is to be used as a value, the entity %ES; must be used in the corresponding DFDL string literal. Whitespace: When whitespace must be used as part of a property value, the DFDL string literal must use entities (such as %WSP;) to represent the whitespace. (This allows a property to represent lists of DFDL string literals by using literal spaces to separate list elements.) The nearest match to an XSDL built-in type is xs:token, but we require the additional constraint that no whitespace can appear. My thought is to define a restriction of xs:token that applies a pattern facet to disallow use of #x20, given that the whitespace 'collapse' implied by xs:token would have replaced #x9, #xA, #xD with #x20, collapsed contiguous #x20, and trimmed leading/trailing #x20. Does that sound right? Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 5

Policy Driven DFDL?
by Garriss Jr., James P. 17 Jul '13

17 Jul '13

Suppose you have an XSLT stylesheet with a bunch of global variables. Suppose that the value of those variables must be determined outside of the XSLT. How do you get the values into it? You might use parameters, and pass in the values when you call the transform() function in your code. But you could also use the document() function from within XSLT, read in an XML "policy" file that has all the values, then store the values in the variables. You might call this approach policy driven XSLT. Can we do policy driven DFDL? Yes, I know that we have parameters, but is there a way to read in a "policy" file (whether XML or not), read the values, then set the variables to those values? TIA

3 2

OGF DFDL WG Call Minutes 2013-07-16
by Steve Hanson 16 Jul '13

16 Jul '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/13095?download= Errata document is now at v013 here: http://redmine.ogf.org/dmsf_files/8104?download= Action 140 errata addendum at v001 here: http://redmine.ogf.org/dmsf_files/13017?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2013-07-16
by Steve Hanson 16 Jul '13

16 Jul '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13095?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] List of desired tutorials?
by Cranford, Jonathan W. 12 Jul '13

12 Jul '13

There's a list in the first tutorial (Lesson 1) at the DFDL OGF, but I don't know how up-to-date it is. http://redmine.ogf.org/dmsf/dfdl-wg?folder_id=5485 fyi, -Jonathan Cranford >-----Original Message----- >From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf >Of Garriss Jr., James P. >Sent: Friday, July 12, 2013 11:38 AM >To: dfdl-wg(a)ogf.org >Subject: [DFDL-WG] List of desired tutorials? > >Is there a list of desired (not yet written) DFDL tutorials?

1 0

List of desired tutorials?
by Garriss Jr., James P. 12 Jul '13

12 Jul '13

Is there a list of desired (not yet written) DFDL tutorials?

1 0