December 2012 - dfdl-wg - lists.ogf.org

Re: [DFDL-WG] DFDL character entities in DFDL expressions
by Mike Beckerle 05 Dec '12

05 Dec '12

Well, yes, I think we're discussing exactly how that should work. No matter what, we do need to be clear that string values created in DFDL's expression language can contain these XML-illegal characters, since they are allowed in DFDL's infoset. This means that DFDL implementations can only re-use an existing XPath implementation to create their DFDL expression language implementation to the extent that it does NOT enforce the XML-illegal characters restrictions all over the place. I am currently working with standard Saxon-B XPath and will report back. But let's be optimistic. The question then is just what is the solution to creating a string-literal including these characters. That cannot be done without some beyond-XML mechanism. DFDL has a string-literal notation for expressing these characters, so we either say that string literals in the expression language can use the DFDL character and numeric entities, or we can do something more 'library like', and provide a function which interprets the string-literal notation, and isolate the implementation concerns a bit. As a language embedded in XML schema, we already straddle the fence of two somewhat inconsistent language environments. E.g., the literals one can use as the value of the default attribute on an element declaration cannot use DFDL character entities, as this is a purely XML Schema construct. Similarly, the regular expressions one can use for the XML schema pattern facet are more restrictive than the DFDL regular expressions one can use in a dfdl:assert, or a dfdl:lengthKind='pattern'. So, it's acceptable to me to say that expressions also have some split where the dfdl-specific aspects, like the dfdl character and numeric entities notation, is isolated in a sub-construct. On Wed, Dec 5, 2012 at 6:37 AM, Andrew Coleman <andrew_coleman(a)uk.ibm.com>wrote: > No, casting a hexBinary to a string will just write out the octets - i.e. > the string will be '00'. > > XPath itself has no mechanism for interpreting entity references or > character references. Its hosting language (XQuery or XSLT/XML) provides > this. Since DFDL is XML, wouldn't that provide a mechanism? > > Regards, > - Andy > > __________________________________________ > Andrew Coleman > WebSphere Message Broker Development > IBM Hursley Park > > > > > From: Steve Hanson/UK/IBM > To: Tim Kimber/UK/IBM@IBMGB, > Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org, Mike Beckerle < > mbeckerle.dfdl(a)gmail.com>, Andrew Coleman/UK/IBM@IBMGB > Date: 05/12/2012 11:06 > Subject: Re: [DFDL-WG] DFDL character entities in DFDL expressions > ------------------------------ > > > Aren't XPath facilities sufficient here? > > outputValueCalc="{ if (fn:string-length(../s) lt 64) then > fn:concat(../s, xs:string(xs:hexBinary('00'))) else ../s }" > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > > From: Tim Kimber/UK/IBM@IBMGB > To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, > Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org > Date: 05/12/2012 10:51 > Subject: Re: [DFDL-WG] DFDL character entities in DFDL expressions > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > I think the restriction was aimed at avoiding things like this: > > outputValueCalc="{ if (fn:string-length(../s) lt 64) then > fn:concat(../s, '%#rFF;') else ../s }" > > I agree that a total ban is too restrictive. My personal preference would > be for the dfdl:string() function because it makes the usage of > DFDL-specific features obvious in the DFDL expression. But what would be > the return type of dfdl:string()? It it returned a sequence of characters > then the raw byte entity ( %#rnn; ) would still need to be disallowed. > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: kimbert(a)uk.ibm.com > Tel. 01962-816742 > Internal tel. 37246742 > > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org, > Date: 04/12/2012 23:36 > Subject: [DFDL-WG] DFDL character entities in DFDL expressions > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > > We currently have this language in the spec: > > "Within an expression, a string is never interpreted as a DFDL string > literal." > > To me this means one cannot use DFDL character entities in an expression. > > However, I need to do this: > > outputValueCalc="{ if (fn:string-length(../s) lt 64) then > fn:concat(../s, '%NUL;') else ../s }" > > Basically, I need to append a NUL on the end of the string in the output > value case. > > Unless I can put a %NUL; into an expression and have it interpreted as a > DFDL String literal, I am not sure how I can achieve this. > > At minimum I need a new DFDL function which might be an alternate string > constructor, such as dfdl:string('....') which interprets the argument as > something where the contents are to be scanned for DFDL character entities > and they are substituted so that the resulting string can contain the > characters that are disallowed in XML. (like NUL) > > -- > Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies > Tel: 781-330-0412 > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > *https://www.ogf.org/mailman/listinfo/dfdl-wg*<https://www.ogf.org/mailman/listinfo/dfdl-wg> > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

2 2

DFDL character entities in DFDL expressions
by Mike Beckerle 05 Dec '12

05 Dec '12

We currently have this language in the spec: "Within an expression, a string is never interpreted as a DFDL string literal." To me this means one cannot use DFDL character entities in an expression. However, I need to do this: outputValueCalc="{ if (fn:string-length(../s) lt 64) then fn:concat(../s, '%NUL;') else ../s }" Basically, I need to append a NUL on the end of the string in the output value case. Unless I can put a %NUL; into an expression and have it interpreted as a DFDL String literal, I am not sure how I can achieve this. At minimum I need a new DFDL function which might be an alternate string constructor, such as dfdl:string('....') which interprets the argument as something where the contents are to be scanned for DFDL character entities and they are substituted so that the resulting string can contain the characters that are disallowed in XML. (like NUL) -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

3 2

Clarification/Fix: Glossary entries about scanning
by Mike Beckerle 04 Dec '12

04 Dec '12

Definition of "scan" assumed in text throughout the spec encompasses both scanning for delimiters, and scanning for matches to regular expressions. So these glossary changes are needed to clarify that delimiter scanning is a specialized sub-class of scanning generally. Delimiter Scanning: Remove subphrase ", or simply scanning for short. Scanning also..." replace with "Delimiter scanning also ..." Scan: Remove the word "bytes". Insert after "...and terminators." this text ", or matches to regular expressions." -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

1 0

Re: [DFDL-WG] discriminators near PoU or not
by Steve Hanson 04 Dec '12

04 Dec '12

Agreed to defer to a future DFDL release. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 19/11/2012 19:24 Subject: Re: [DFDL-WG] discriminators near PoU or not Hi Mike, I hit a similar scenario today with an EDIFACT message. There was an optional array containing a bunch of initiated records. I used 'initiatedContent' for the array's sequence, but needed to add an extra discriminator to stop an error from within one of the records from causing the parser to backtrack and conclude that the array occurrence was not there. Not easy to know where to put it. One possibility is an 'all' attribute that says it resolves all points of uncertainty that are in scope? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 29/10/2012 01:25 Subject: [DFDL-WG] discriminators near PoU or not Sent by: dfdl-wg-bounces(a)ogf.org I thought this situation was worth discussing. I have a bunch of messages. They all work like this: <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> There are hundreds of those. These all are used in this way: <element name="topLevel"> <complexType> <sequence> <element name="message" minOccurs="1" maxOccurs="unbounded"> <complexType> <choice> <element ref="messageName1"/> <element ref="messageName2"/> .... </choice> </complexType> </element> </sequence> </complexType> </element> So, there are two points of uncertainty here. The choice, and the unbounded array surrounding it. For each message, it is not until the sequence carrying the initiator, which is down inside the message structure, that we know for sure we've got this kind of message. So currently each message element is designed to be used ONLY in the above context of two surrounding points of uncertainty, like so (same message format as above but with TWO discriminators added). <element name="messageName1"> <complexType> <sequence> <element ref="boilerplateElt1" minOccurs="0"/> <element ref="boilerplateElt2" minOccurs="0"/> <sequence dfdl:initiator="MSG1"> <annotation><appinfo...>  <dfdl:discriminator>{ true() }</dfdl:discriminator>  <dfdl:discriminator>{ true() }</dfdl:discriminator> </appinfo></annotation> .... message specific elements go in here... </sequence> ... optional trailing boilerplates go here </complexType> </element> I'd love to have a better solution to this issue. But in the absence of one this works and achieves what I want which is that once it hits the initiator, we discriminate both which message we have, and we discriminate that we in fact have a message. Interesting that I'd really prefer to discriminate them in the opposite order of their nesting. I could discriminate the array the the very start of the message. But I have no way to do that because the discriminators apply in inward-out nesting order, and the nearest enclosing PoU is the choice. So I have to discriminate that one first, and I can't discriminate that one until I see the initiator, which is later into the message. Possible fixes/improvements: allow a label/id on each PoU, and allow referencing that label from a discriminator to say exactly which PoU you are discriminating. This makes the outward reference implied by a discriminator explicit. You still have the context issue, but it's not implied anymore, it is explicit. (This makes the problem like expressions with "../../.." that reach up and out of a construct. They are reaching up and out, but at least you can see directly that they are doing so.) -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2012-12-04
by Steve Hanson 04 Dec '12

04 Dec '12

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/12973?download= Errata document is now at v011 (draft) here: http://redmine.ogf.org/dmsf_files/8104?download= ** No new documents will be added to GridForge ** Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

text alignment and Errata 2.95 in DFDL Errata v011
by Mike Beckerle 04 Dec '12

04 Dec '12

Steve, Please add this point to the agenda for next week. We have to revisit this errata given the 7-bit character encoding. (And, I've discovered now there are legacy 6-bit as well which do come up.) ...mikeb On Mon, Nov 26, 2012 at 10:32 PM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com>wrote: > We also need to decide on Errata 2.95 which is about whether text is > implicitly 8-bit aligned. > > Given that we have cases of text aligned in other ways (the 7-bit packed > ascii stuff), this rule seems heavy handed and inflexible. > > We could say that if alignment="implicit", then the first codepoint of any > text is implicitly aligned on an 8-bit boundary. This would apply to > delimiters, but also to any text that is being scanned with a regex. > Subsequent codepoints of the same text are not specifically aligned. This > lets us choose alignment="1", and the 7-bit packed stuff can then have > syntax in it, have regex's that it must match, etc. but if > alignment="implicit", then the normal, less error prone behavior of > skipping to a byte-boundary would be in force. > > ...mikeb > > On Mon, Nov 26, 2012 at 10:10 PM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com>wrote: > >> I reviewed this. >> >> 2.108 has Change: Format annotations.... and also Remove: Format >> Annotations. Should just have Change. >> >> 3.13 I think we missed a couple places where the previous design where we >> had replacement control is still leaking into the text. >> >> Phrase "Any character that is skipped counts as 0." should be dropped. We >> don't skip anymore. >> >> Concerning: "In the following, the match is against data where any >> character replacement has already taken place." The first sub-bullet about >> lengthKind pattern is incorrect. For lengthKind pattern I believe any >> decode error should cause the match to end, i.e., a pattern that matches >> digits only will match a string of digts followed by binary data that >> causes a decode error. Similarly, when matching literal nils, (character or >> literal value), any decode error should cause the match to fail. >> >> >> >> >> On Fri, Nov 23, 2012 at 12:41 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: >> >>> A few corrections made and added errata from Nov 20th call. Please >>> review for next call. Also now on Redmine. >>> >>> >>> >>> Regards >>> >>> Steve Hanson >>> Architect, Data Format Description Language (DFDL) >>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >>> IBM SWG, Hursley, UK* >>> **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >>> tel:+44-1962-815848 >>> ----- Forwarded by Steve Hanson/UK/IBM on 23/11/2012 17:40 ----- >>> >>> From: Steve Hanson/UK/IBM >>> To: dfdl-wg(a)ogf.org, >>> Date: 20/11/2012 14:23 >>> Subject: First draft of DFDL Errata v011 >>> ------------------------------ >>> >>> >>> Please find attached the first draft of Errata v011. >>> This incorporates all DFDL spec errata raised since v010 (June 22 2012) >>> until the last DFDL WG call (November 13 2012). >>> Change tracking: I accepted all changes from v010 so the changes you see >>> are all due to v011. >>> I would appreciate a review for correctness and omissions. >>> Please note that some of the new errata have been best handled by >>> updating existing related errata, so please read from start. >>> >>> >>> >>> Regards >>> >>> Steve Hanson >>> Architect, Data Format Description Language (DFDL) >>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >>> IBM SWG, Hursley, UK* >>> **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >>> tel:+44-1962-815848 >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >>> 3AU >>> >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >>> 3AU >>> >>> -- >>> dfdl-wg mailing list >>> dfdl-wg(a)ogf.org >>> https://www.ogf.org/mailman/listinfo/dfdl-wg >>> >> >> >> >> -- >> Mike Beckerle | OGF DFDL WG Co-Chair >> Tel: 781-330-0412 >> >> > > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > > -- Mike Beckerle | OGF DFDL WG Co-Chair | Tresys Technologies Tel: 781-330-0412

1 0

Clarification: can you escape a pad char with an escape char?
by Mike Beckerle 04 Dec '12

04 Dec '12

Did we decide whether an escape character can precede a pad character hence, making it part of the content, not the padding? E.g., suppose pad character is P, escape character is E. then can 'abcdEPPPP' be interpreted as 'abcdP' then 3 pad characters, or did we decide that padding ignores escape chars? I think we decided that padding is outside of the escape blocks, so that if [ and ] are my escape block start and end, then PPP[PabcP]PPP would be 'PabcP' as content. but I don't think we commented on regular single escape character characters and pad char. It seems consistent to me that 'abcdEPPPP' is abcdP as content. but if not, then 'abcdEPPP' stripping off the trailing padding ends with the escape character, so the escape character is escaping nothing hence the content would be 'abcd' I believe. Comments? ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

OGF DFDL WG Call Agenda 2012-12-04
by Steve Hanson 03 Dec '12

03 Dec '12

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/12973?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Research Data Alliance to launch, buy-in from funding agencies in 3 countries so far
by Steve Hanson 03 Dec '12

03 Dec '12

For discussion on next DFDL WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 03/12/2012 17:36 ----- From: "Sill, Alan" <alan.sill(a)ttu.edu> To: "David E. Martin" <martinde(a)northwestern.edu>, Steve Hanson/UK/IBM@IBMGB, Cc: "Sill, Alan" <alan.sill(a)ttu.edu> Date: 03/12/2012 15:21 Subject: Research Data Alliance to launch, buy-in from funding agencies in 3 countries so far Thought you would be interested in the following link. Launches in Gothenberg Sweden at a conference March 18-20, 2013. Topic: The purpose of the Research Data Alliance is to accelerate international data-driven innovation and discovery by facilitating research data sharing and exchange, use and re-use, standards harmonization, and discoverability. This will be achieved through the development and adoption of infrastructure, policy, practice, standards, and other deliverables. Link: http://rd-alliance.org/ Alan Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Puzzle: unparsing format involving lengthKind='pattern'
by Mike Beckerle 03 Dec '12

03 Dec '12

I have a file format where strings have this unusual discipline for termination. The string is either 64 characters long or, it is from 0 to 63 characters long, with a NUL terminator. I can parse this like so: <xs:complexType name="myStringType"> <xs:sequence> <xs:element name="s" type="xs:string" dfdl:lengthKind="pattern"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/">  <dfdl:element> <dfdl:property name="lengthPattern"><![CDATA[([^\x00]{0,63})(?=\x00)|[^\x00]{64}]]></dfdl:property> </dfdl:element> </xs:appinfo> </xs:annotation> </xs:element> <xs:element name="term" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="0" dfdl:initiator="&NUL;" dfdl:outputValueCalc="{ '' }" minOccurs="0" maxOccurs="1" dfdl:occursCountKind="expression" dfdl:occursCount="{ if(fn:string-length(../tns:s) lt 64) then 1 else 0 }" /> </xs:sequence> </xs:complexType> What I did is use a lengthKind pattern to pick off the content, excluding the NUL, and then model the NUL explicitly as the initiator of an empty element which is optionally occurring depending on the length of the string. That will work. I'd like to hide the "term" element in a hidden group, but other than that it will work for parsing. Question: How can I unparse this? I want the schema and DFDL processor to put the "term" element into the infoset by itself depending on the length of the 's' element that I will place into the infoset. So I put an outputValueCalc on there to assign it a value of empty string, but that will only happen if the occursCount expression causes the optional element to exist at all. Will this work for unparsing? Spec currently says that occursCount expression is used only when parsing, otherwise the number in the infoset are used. But I don't want to have to put this syntax-modeling element into the infoset from the application. I just want the application to create a string, and then based on whether it is 0-63 long, I want the Nul added or not. Other ways I've tried to model these strings include as a choice of two different elements. That has a similar issue that I then have to assemble my infoset for output using a length dependent element. I can't just put a string into the infoset and have it decide when unparsing which of the two choice branches is the right one. (asserts and discriminators are parse-only also.) Anyone have better ideas? ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 3