Re: [DFDL-WG] unordered sequence with constrained occurrences

Yes they are in DFDL v1.0. Just haven't been implemented by anyone yet. On Wed, Mar 6, 2013 at 3:10 PM, Cranford, Jonathan W. wrote:

...

the more I think that unordered sequences should be part of DFDL 1.0.** **

** **

Did I miss something? Unordered sequence groups are still a part of DFDL 1.0, aren’t they? They might not be supported in either MBTK or Daffodil yet, but they are still in the spec, right?****

** **

Just checking,****

** **

Jonathan Cranford****

** **

*From:* dfdl-wg-bounces@ogf.org [mailto:dfdl-wg-bounces@ogf.org] *On Behalf Of *Garriss Jr., James P. *Sent:* Wednesday, March 06, 2013 11:43 AM *To:* dfdl-wg@ogf.org *Subject:* Re: [DFDL-WG] unordered sequence with constrained occurrences** **

** **

Suppose I’m modeling IMF headers, many of which can have the exact same form, stuff like:****

** **

From: john@doe.com****

To: jane@gmail.com****

Return-Path: bob@yahoo.com****

** **

Etc. Remember that these can be in any order, so they are an unordered sequence.****

** **

The way that we’ve modeled these headers so far, the “From:” and “To:” and so on have been initiators; they aren’t elements. But when I use our workaround for an unordered sequence, which requires discriminators, I am in trouble. Because the thing that discriminates all of these headers is an initiator, not an element.****

** **

So, it seems to me that I need to change all my headers so that the “From:” and “To:” and such are no longer initiators but elements.****

** **

Does that sound right?****

** **

The more I work with this workaround, the more hackish it feels, and the more I think that unordered sequences should be part of DFDL 1.0. Maybe?* ***

** **

*From:* Steve Hanson [mailto:smh@uk.ibm.com ] *Sent:* Wednesday, March 06, 2013 4:16 AM *To:* Garriss Jr., James P. *Cc:* dfdl-wg@ogf.org; dfdl-wg-bounces@ogf.org *Subject:* Re: [DFDL-WG] unordered sequence with constrained occurrences** **

** **

James,

The checkConstraints function is just a convenience that saves you having to duplicate constraints in an assert or discriminator. For now, just duplicate the constraint as a discriminator. This works fine as long as you can express the constraint as a DFDL expression, which with your example you can.

I've tested your xsd exactly as you supplied below (without the terminator) on my latest MBTK and it parses 'abc' fine. I don't see the infinite loop error. We did have some bugs in that area where the check was being applied too strictly which we fixed.

I then tried with 'cba' which parsed without error, except of course that the values ended up in the wrong elements. So I added discriminators to check that the elements matched their fixed value, and 'cba' then parsed into the correct elements.

*xsd:annotation* http://www.ogf.org/dfdl/"> *dfdl:discriminator*{. *eq* 'b'}** ** ** **

I then tried with more complex strings, such as '*cbabaccba*', and they all parsed ok.

To make the infoset more symmetric, with one child per array occurrence, you can use a choice instead of a sequence.

Making that change then results in:

Here's the xsd with discriminators and choices. See if it works with your MBTK.

If you are still hitting the infinite loop error then add the %NL; terminator to the array element. This will parse data of the form:

c b a b a c c b a

Regards

Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group http://www.ogf.org/dfdl/ IBM SWG, Hursley, UK* *smh@uk.ibm.com tel:+44-1962-815848 <+44-1962-815848>

From: "Garriss Jr., James P." To: "dfdl-wg@ogf.org" , Date: 05/03/2013 19:15 Subject: Re: [DFDL-WG] unordered sequence with constrained occurrences Sent by: dfdl-wg-bounces@ogf.org **** ------------------------------

...

The error message is because you don't make forward progress through the data with potentially unbounded occurrences.

I think you just said, “MBTK prevents an infinite loop.” That makes sense.

...

If there are delimiters then model those and you might not get the error.

I think you just said, “To let MBTK know when it should stop checking, you need a terminator of some sort.” That also makes sense. So I added a terminator (%NL;) here:

Good news: That fixed the problem, so long as my input is “abc”.

Bad news: This breaks if the input is any other legal value, such as “abbc” or “cba” or “b”.

The problem for all of these is that my dear friend, checkConstraints, is not implemented yet, thus I can’t prevent the parser from slurping up the wrong character. I don’t know how anyone can build a non-trivial DFDL schema that involves any sort of choice without this method; I swear, it must be the single most important thing you guys have created for DFDL.

Until checkConstraints is implemented, I’m not really able to test this schema with MBTK.

Thanks so much for your help answering my questions, Steve!

*From:* Steve Hanson [mailto:smh@uk.ibm.com ] * Sent:* Tuesday, March 05, 2013 1:46 PM* To:* Garriss Jr., James P.* Cc:* dfdl-wg@ogf.org; dfdl-wg-bounces@ogf.org* Subject:* Re: [DFDL-WG] unordered sequence with constrained occurrences

James,

The error message is because you don't make forward progress through the data with potentially unbounded occurrences. Is this because you are using a cut-down schema? If there are delimiters then model those and you might not get the error.

Once you have processed the array you can use asserts to check the count. However IBM DFDL does not implement the count functions yet.

Give me a couple of days to look at this more closely. I have a customer visit tomorrow hence the delay.

Regards

Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group http://www.ogf.org/dfdl/ IBM SWG, Hursley, UK* *smh@uk.ibm.com* *tel:+44-1962-815848 <+44-1962-815848>

From: "Garriss Jr., James P." To: "dfdl-wg@ogf.org" , Date: 05/03/2013 16:19 Subject: Re: [DFDL-WG] unordered sequence with constrained occurrences Sent by: dfdl-wg-bounces@ogf.org **** ------------------------------

Hmmm, maybe not. I said:

...

The unordered sequence can be modeled with a data array

Yet when implemented in MBTK, it throws a fatal error:

fatal: CTDP3148E: Infinite loop at offset 3: The DFDL parser cannot process array element 'ABCarray' because maxOccurs is unbounded and the length of the previous occurrence was zero.

I think what happens is that on the last pass through the array, it doesn’t find a, b, or c, so it throws a fatal error.

So is this a bug in MBTK? Or can DFDL not model an unordered sequence? Or am I just doing it wrong?

Here’s a sample DFDL schemas that illustrates the point:

<?xml version=*"1.0"* encoding=*"UTF-8"*?> http://www.ogf.org/dfdl/dfdl-1.0/*http://www.ogf.org/dfdl/dfdl-1.0/ *"* xmlns:fmt=*"**http://www.ibm.com/dfdl/GeneralPurposeFormat*http://www.ibm.com/dfdl/GeneralPurposeFormat *"* xmlns:ibmSchExtn=*"**http://www.ibm.com/schema/extensions*http://www.ibm.com/schema/extensions *"* xmlns:xsd=*"**http://www.w3.org/2001/XMLSchema*http://www.w3.org/2001/XMLSchema *"*> http://www.ibm.com/dfdl/GeneralPurposeFormat*http://www.ibm.com/dfdl/GeneralPurposeFormat *"* schemaLocation=*"IBMdefined/GeneralPurposeFormat.xsd"* /> xsd:complexType xsd:annotation http://www.ogf.org/dfdl/* http://www.ogf.org/dfdl/*"*> xsd:complexType xsd:annotation http://www.ogf.org/dfdl/*http://www.ogf.org/dfdl/ *"*>

Test with “abc” as sample input. * From:* Garriss Jr., James P. * Sent:* Tuesday, March 05, 2013 8:43 AM* To:* dfdl-wg@ogf.org* Subject:* unordered sequence with constrained occurrences

Suppose text data has 3 constructs: a, b, and c.

· a must occur 1 time · b can occur 0 or 1 time · c can occur any number of times, 0 or more

These 3 constructs can appear in any order.

So these are valid inputs:

abc a bcccca

But these are not:

ccbcc abbc abcabc

Can data like this be modeled with DFDL?

The unordered sequence can be modeled with a data array, like this:

Array (0 to unbounded) Sequence a (0 to 1) b (0 to 1) c (0 to 1) /Sequence /Array

But I don’t know how to constrain the total number of occurrences.

Appreciate any ideas!-- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ****

-- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg

-- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com