I had action items from GGF14 to begin building up an FAQ. Here's something about layering.

-------------------------

Layering in DFDL

The book "ASN.1 Complete" by Larmouth (ISBN 0-12-233435-3 and available
online as a pdf) discusses the importance of layer support in format descriptions.


(Begin Excerpt)

The layering concept is perhaps most commonly associated with the
International Standards Organization (ISO) and International
Telecommunications Union (ITU) "architecture" or "7-layer model"
for
Open Systems Interconnection (OSI) shown in Figure 3.  While many of
the protocols developed within this framework are not greatly used
today, it remains an interesting academic study for approaches to
protocol specification. In the original OSI concept in the late 1970s,
there would be just 6 layers providing (progressively richer) carrier
services, with a final "application layer" where each specification
supported a single endapplication, with no "holes".

It became apparent, however, over the next decade, that even in the
"application layer" people wanted to leave "holes" in their
specification for later extensions, or to provide a means of tailoring
their protocol to specific needs. For example, one of the more recent
and important protocols - Secure Electronic Transactions (SET) -
contains a wealth of fully-defined message semantics, but also
provides for a number of "holes" which can transfer "merchant
details"
which are not specified in the SET specification itself. So we have
basic messages for purchase requests and responses, inquiry requests
and responses, authorization requests and responses, and so on, but
within those messages there are “holes” for “message
extensions” - additional information specific to a particular
merchant.

It is thus important that any mechanism or notation for specifying a
protocol should be able to cater well for the inclusion of
"holes". This has been one of the more important developments
in ASN.1
in the last decade, and will be a subject of much further discussion
in this book.

"Catering well" for the inclusion of "holes" implies that
the notation
must have defined mechanisms (preferably uniformly applied to all
specifications written using that notation) to identify the contents
of a hole at communications time. (In lower layers, this is sometimes
referred to as the "protocol id" problem). Equally important,
however,
are notational means to clearly identify that a specification is
incomplete (contains a hole), together with well-defined mechanisms to
relate the (perhaps later in time) specification of the contents of
holes to the location of the holes themselves.

(End Excerpt)

The argument made here is equally true for DFDL. We need the ability to describe
a data format containing a hole or payload which another DFDL schema can then
describe the format of.

DFDL actually encounters some data formats where there are discontiguous holes.
Consider the nonVSAM VS format. (see "IBM OS/390 DFSMS: Using Data Sets"
IBM publication  SC26-7339-01, Second Edition, December 2000. (online
at: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/DGT1D411/CCONTEN
TS?SHELF=EZ239126&DN=SC26-7339-01&DT=20001014144419) In this format,
data records with actual data fields of interest are broken up into segments.
The segments are of variable size, and a record can fit in a single segment
or can span multiple segments. The segments are of 3 types, initial, middle
(there can be zero or more of these), and final. The hole that the record fits
in is assembled by putting together the partial holes from each of the segments.
Adding minor additional complexity is that in the actual format the segments
are then grouped into variable-sized blocks as an I/O transfer-unit efficiency
optimization.

A further wrinkle on holes in DFDL is the notion of encoding. Modern data formats
often contain holes (or we'll also call them payloads) which have been encoded
to allow data transfer in text-only mediums, or to compress to save space,
or to encrypt, or for various other reasons. The encoding must be decoded and
the resulting data is the payload where we then want to describe the format.
There are many examples of this, but email messages using MIME encapsulated
attachments are a classic example. We'd like to describe a file of email messages
each containing MIME encapsulated attachments where the attachments are compressed
binary data where the data is a binary data format. We'd like to describe this
file and expose the logical structure of the data that is inside the MIME
encapsulated attachments.