March 2013 - dfdl-wg - lists.ogf.org

OGF DFDL WG Call Agenda 2013-03-12
by Steve Hanson 12 Mar '13

12 Mar '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13034?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Optional Initiator and Terminator?
by Cranford, Jonathan W. 12 Mar '13

12 Mar '13

I don't know the answer to the DFDL portion of your question, but I can say your two workarounds aren't equivalent. The latter will accept an initiator with no terminator, and vice versa (e.g., "{ data" and "data }", while the former will not. HTH, Jonathan From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Garriss Jr., James P. Sent: Friday, March 08, 2013 12:42 PM To: dfdl-wg(a)ogf.org Subject: [DFDL-WG] Optional Initiator and Terminator? Is there a way to specify that an initiator/terminator pair is optional? IOW, these are both valid: { data } data If not, which of these workarounds is better (and why)? Choice DataWithInitiatorAndTeminator DataWithoutInitiatorAndTeminator Or Sequence OptionalInitiator (0 to 1) Data OptionalTerminator (0 to 1) These seem the same to me, but maybe there's a reason why one is better.

3 5

Can DFDL decode Base64
by Garriss Jr., James P. 11 Mar '13

11 Mar '13

Some of the content that we are parsing is Base64 encoded. I know this may be a bit of a stretch, but does DFDL decode Base64?

3 2

Optional Initiator and Terminator?
by Garriss Jr., James P. 08 Mar '13

08 Mar '13

Is there a way to specify that an initiator/terminator pair is optional? IOW, these are both valid: { data } data If not, which of these workarounds is better (and why)? Choice DataWithInitiatorAndTeminator DataWithoutInitiatorAndTeminator Or Sequence OptionalInitiator (0 to 1) Data OptionalTerminator (0 to 1) These seem the same to me, but maybe there's a reason why one is better.

1 0

Does DFDL support nesting (recursion)?
by Garriss Jr., James P. 07 Mar '13

07 Mar '13

MIME supports infinite nesting. For example, a message can have the content type multipart/mixed, which means that there will be multiple body parts, each of which can have its own content type. And, any of those body parts can have the content type multipart/mixed, which means that the part will have multiple subparts, each of which can have its own content type. Etc etc etc...and so now we have infinite nesting. Any pointers on DFDL's support for something like this? TIA

2 1

Re: [DFDL-WG] unordered sequence with constrained occurrences
by Garriss Jr., James P. 07 Mar '13

07 Mar '13

Hmmm, maybe not. I said: > The unordered sequence can be modeled with a data array Yet when implemented in MBTK, it throws a fatal error: fatal: CTDP3148E: Infinite loop at offset 3: The DFDL parser cannot process array element 'ABCarray' because maxOccurs is unbounded and the length of the previous occurrence was zero. I think what happens is that on the last pass through the array, it doesn't find a, b, or c, so it throws a fatal error. So is this a bug in MBTK? Or can DFDL not model an unordered sequence? Or am I just doing it wrong? Here's a sample DFDL schemas that illustrates the point: <?xml version="1.0" encoding="UTF-8"?> <xsd:schema xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:fmt="http://www.ibm.com/dfdl/GeneralPurposeFormat" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:import namespace="http://www.ibm.com/dfdl/GeneralPurposeFormat" schemaLocation="IBMdefined/GeneralPurposeFormat.xsd" /> <xsd:element ibmSchExtn:docRoot="true" name="ABC"> <xsd:complexType> <xsd:sequence dfdl:separator=""> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:sequence /> </xsd:appinfo> </xsd:annotation> <xsd:element dfdl:occursCountKind="implicit" maxOccurs="unbounded" minOccurs="1" name="ABCarray"> <xsd:complexType> <xsd:sequence dfdl:separator=""> <xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" fixed="a" minOccurs="0" name="a" type="xsd:string" /> <xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" fixed="b" minOccurs="0" name="b" type="xsd:string" /> <xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" fixed="c" minOccurs="0" name="c" type="xsd:string" /> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:sequence> </xsd:complexType> </xsd:element> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format ref="fmt:GeneralPurposeFormat" /> </xsd:appinfo> </xsd:annotation> </xsd:schema> Test with "abc" as sample input. From: Garriss Jr., James P. Sent: Tuesday, March 05, 2013 8:43 AM To: dfdl-wg(a)ogf.org Subject: unordered sequence with constrained occurrences Suppose text data has 3 constructs: a, b, and c. * a must occur 1 time * b can occur 0 or 1 time * c can occur any number of times, 0 or more These 3 constructs can appear in any order. So these are valid inputs: abc a bcccca But these are not: ccbcc abbc abcabc Can data like this be modeled with DFDL? The unordered sequence can be modeled with a data array, like this: Array (0 to unbounded) Sequence a (0 to 1) b (0 to 1) c (0 to 1) /Sequence /Array But I don't know how to constrain the total number of occurrences. Appreciate any ideas!

3 6

Re: [DFDL-WG] unordered sequence with constrained occurrences
by Cranford, Jonathan W. 07 Mar '13

07 Mar '13

> the more I think that unordered sequences should be part of DFDL 1.0. Did I miss something? Unordered sequence groups are still a part of DFDL 1.0, aren’t they? They might not be supported in either MBTK or Daffodil yet, but they are still in the spec, right? Just checking, Jonathan Cranford From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Garriss Jr., James P. Sent: Wednesday, March 06, 2013 11:43 AM To: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] unordered sequence with constrained occurrences Suppose I’m modeling IMF headers, many of which can have the exact same form, stuff like: From: john(a)doe.com<mailto:john@doe.com> To: jane(a)gmail.com<mailto:jane@gmail.com> Return-Path: bob(a)yahoo.com<mailto:bob@yahoo.com> Etc. Remember that these can be in any order, so they are an unordered sequence. The way that we’ve modeled these headers so far, the “From:” and “To:” and so on have been initiators; they aren’t elements. But when I use our workaround for an unordered sequence, which requires discriminators, I am in trouble. Because the thing that discriminates all of these headers is an initiator, not an element. So, it seems to me that I need to change all my headers so that the “From:” and “To:” and such are no longer initiators but elements. Does that sound right? The more I work with this workaround, the more hackish it feels, and the more I think that unordered sequences should be part of DFDL 1.0. Maybe? From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Wednesday, March 06, 2013 4:16 AM To: Garriss Jr., James P. Cc: dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>; dfdl-wg-bounces(a)ogf.org<mailto:dfdl-wg-bounces@ogf.org> Subject: Re: [DFDL-WG] unordered sequence with constrained occurrences James, The checkConstraints function is just a convenience that saves you having to duplicate constraints in an assert or discriminator. For now, just duplicate the constraint as a discriminator. This works fine as long as you can express the constraint as a DFDL expression, which with your example you can. I've tested your xsd exactly as you supplied below (without the terminator) on my latest MBTK and it parses 'abc' fine. I don't see the infinite loop error. We did have some bugs in that area where the check was being applied too strictly which we fixed. [cid:image001.gif@01CE1A6B.F8776480] I then tried with 'cba' which parsed without error, except of course that the values ended up in the wrong elements. So I added discriminators to check that the elements matched their fixed value, and 'cba' then parsed into the correct elements. <xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" fixed="b" minOccurs="0" name="b" type="xsd:string"> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator>{. eq 'b'}</dfdl:discriminator> </xsd:appinfo> </xsd:annotation> </xsd:element> [cid:image002.gif@01CE1A6B.F8776480] I then tried with more complex strings, such as 'cbabaccba', and they all parsed ok. [cid:image003.gif@01CE1A6B.F8776480] To make the infoset more symmetric, with one child per array occurrence, you can use a choice instead of a sequence. [cid:image004.gif@01CE1A6B.F8776480] Making that change then results in: [cid:image005.gif@01CE1A6B.F8776480] Here's the xsd with discriminators and choices. See if it works with your MBTK. If you are still hitting the infinite loop error then add the %NL; terminator to the array element. This will parse data of the form: c b a b a c c b a Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 From: "Garriss Jr., James P." <jgarriss(a)mitre.org<mailto:jgarriss@mitre.org>> To: "dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>" <dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>>, Date: 05/03/2013 19:15 Subject: Re: [DFDL-WG] unordered sequence with constrained occurrences Sent by: dfdl-wg-bounces(a)ogf.org<mailto:dfdl-wg-bounces@ogf.org> ________________________________ > The error message is because you don't make forward progress through the data with potentially unbounded occurrences. I think you just said, “MBTK prevents an infinite loop.” That makes sense. > If there are delimiters then model those and you might not get the error. I think you just said, “To let MBTK know when it should stop checking, you need a terminator of some sort.” That also makes sense. So I added a terminator (%NL;) here: [cid:image006.png@01CE1A6B.F8776480] Good news: That fixed the problem, so long as my input is “abc”. Bad news: This breaks if the input is any other legal value, such as “abbc” or “cba” or “b”. The problem for all of these is that my dear friend, checkConstraints, is not implemented yet, thus I can’t prevent the parser from slurping up the wrong character. I don’t know how anyone can build a non-trivial DFDL schema that involves any sort of choice without this method; I swear, it must be the single most important thing you guys have created for DFDL. Until checkConstraints is implemented, I’m not really able to test this schema with MBTK. Thanks so much for your help answering my questions, Steve! From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Tuesday, March 05, 2013 1:46 PM To: Garriss Jr., James P. Cc: dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>; dfdl-wg-bounces(a)ogf.org<mailto:dfdl-wg-bounces@ogf.org> Subject: Re: [DFDL-WG] unordered sequence with constrained occurrences James, The error message is because you don't make forward progress through the data with potentially unbounded occurrences. Is this because you are using a cut-down schema? If there are delimiters then model those and you might not get the error. Once you have processed the array you can use asserts to check the count. However IBM DFDL does not implement the count functions yet. Give me a couple of days to look at this more closely. I have a customer visit tomorrow hence the delay. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 From: "Garriss Jr., James P." <jgarriss(a)mitre.org<mailto:jgarriss@mitre.org>> To: "dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>" <dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org>>, Date: 05/03/2013 16:19 Subject: Re: [DFDL-WG] unordered sequence with constrained occurrences Sent by: dfdl-wg-bounces(a)ogf.org<mailto:dfdl-wg-bounces@ogf.org> ________________________________ Hmmm, maybe not. I said: > The unordered sequence can be modeled with a data array Yet when implemented in MBTK, it throws a fatal error: fatal: CTDP3148E: Infinite loop at offset 3: The DFDL parser cannot process array element 'ABCarray' because maxOccurs is unbounded and the length of the previous occurrence was zero. I think what happens is that on the last pass through the array, it doesn’t find a, b, or c, so it throws a fatal error. So is this a bug in MBTK? Or can DFDL not model an unordered sequence? Or am I just doing it wrong? Here’s a sample DFDL schemas that illustrates the point: <?xml version="1.0" encoding="UTF-8"?> <xsd:schema xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:fmt="http://www.ibm.com/dfdl/GeneralPurposeFormat" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:import namespace="http://www.ibm.com/dfdl/GeneralPurposeFormat" schemaLocation="IBMdefined/GeneralPurposeFormat.xsd" /> <xsd:element ibmSchExtn:docRoot="true" name="ABC"> <xsd:complexType> <xsd:sequence dfdl:separator=""> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:sequence /> </xsd:appinfo> </xsd:annotation> <xsd:element dfdl:occursCountKind="implicit" maxOccurs="unbounded" minOccurs="1" name="ABCarray"> <xsd:complexType> <xsd:sequence dfdl:separator=""> <xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" fixed="a" minOccurs="0" name="a" type="xsd:string" /> <xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" fixed="b" minOccurs="0" name="b" type="xsd:string" /> <xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" fixed="c" minOccurs="0" name="c" type="xsd:string" /> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:sequence> </xsd:complexType> </xsd:element> <xsd:annotation> <xsd:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:format ref="fmt:GeneralPurposeFormat" /> </xsd:appinfo> </xsd:annotation> </xsd:schema> Test with “abc” as sample input. From: Garriss Jr., James P. Sent: Tuesday, March 05, 2013 8:43 AM To: dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org> Subject: unordered sequence with constrained occurrences Suppose text data has 3 constructs: a, b, and c. • a must occur 1 time • b can occur 0 or 1 time • c can occur any number of times, 0 or more These 3 constructs can appear in any order. So these are valid inputs: abc a bcccca But these are not: ccbcc abbc abcabc Can data like this be modeled with DFDL? The unordered sequence can be modeled with a data array, like this: Array (0 to unbounded) Sequence a (0 to 1) b (0 to 1) c (0 to 1) /Sequence /Array But I don’t know how to constrain the total number of occurrences. Appreciate any ideas!-- dfdl-wg mailing list dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org> https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU-- dfdl-wg mailing list dfdl-wg(a)ogf.org<mailto:dfdl-wg@ogf.org> https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 2

can DFDL model this? (initiators, but no separators or terminators, plus optional elements)
by Garriss Jr., James P. 06 Mar '13

06 Mar '13

Suppose I have this input data: FirstName James LastName Garriss Hometown Raleigh Company The MITRE Corporation CRLF To the human eye, this is simple. We have four elements, each of which has an initiator. But to make things more interesting: 1. The elements are all strings, and they do not have fixed lengths, set values, or any other terminator. The only way you know them apart is by the initiator. (And this implies that the initiators cannot be part of the elements.) 2. There are no separators (spaces can be in the data). 3. The third and fourth elements are optional. So these are both valid data: FirstName John Mark LastName Smith FirstName Bob LastName Brown Company IBM How do we model this? Attempt #1: I have four elements each with a unique initiator (FirstName, LastName, Hometown, Company). The problem is that there's no way to know when the first element terminates, so everything after the "FirstName" initiator ends up in the FirstName element. Oops. Attempt #2: I got funky with the terminators. The first element has LastName as a terminator. The second element has Hometown or Company as an element. The third element has Company or %NL; as an element. And the fourth one uses %NL;. Works great, unless the optional third element isn't there. IOW, if I have this input: FirstName Bob LastName Brown Company IBM Then "IBM" winds up in Hometown element. Oops. So, what to do? I don't know. I don't know how to solve this. Hopefully you're going to teach me about some feature I don't yet know. If not, then I have a potential solution, an addition to the spec. Add this option as a terminator: "This element terminates when you find the initiator to the next element." That's probably easier said than done, but it seems to make sense in this context.

3 5

Empty element with initiator and terminator
by Garriss Jr., James P. 05 Mar '13

05 Mar '13

I have an element that can be one of two things, either: < string > Or <> So I modeled this as a choice. For the first choice, I made it a string with < and > being the initiator and terminator. Works great. For the second choice, I also made it a string with < and > being the initiator and terminator, and I set the length kind to explicit and the length to 0. That seems obvious enough. But, alas, that causes errors in parsing. Nothing I do for this seemingly simple construct works. There must be some design pattern here that I am missing. Ideas? TIA

2 3

Re: [DFDL-WG] unordered sequence with constrained occurrences
by Garriss Jr., James P. 05 Mar '13

05 Mar '13

Can this be done with one or more asserts? From: Garriss Jr., James P. Sent: Tuesday, March 05, 2013 8:43 AM To: dfdl-wg(a)ogf.org Subject: unordered sequence with constrained occurrences Suppose text data has 3 constructs: a, b, and c. * a must occur 1 time * b can occur 0 or 1 time * c can occur any number of times, 0 or more These 3 constructs can appear in any order. So these are valid inputs: abc a bcccca But these are not: ccbcc abbc abcabc Can data like this be modeled with DFDL? The unordered sequence can be modeled with a data array, like this: Array (0 to unbounded) Sequence a (0 to 1) b (0 to 1) c (0 to 1) /Sequence /Array But I don't know how to constrain the total number of occurrences. Appreciate any ideas!

1 0