May 2018 - dfdl-wg - lists.ogf.org

clarification needed: unparsing and hidden group choices and choices with dfdl:choiceDispatchKey
by Mike Beckerle 13 Dec '19

13 Dec '19

So interesting issue. Comes up as part of our experience implementing the advanced expression language features: hidden groups, and computed elements. We have hidden groups in DFDL. If a xs:choice is inside a hidden group, then by definition, when unparsing, elements corresponding to some branch of the choice are not in the Infoset. DFDL spec clearly states that dfdl:outputValueCalc is not allowed on the head of a choice branch. But elements inside hidden groups must have dfdl:outputValueCalc, or a default value, otherwise there's no way for them to be filled in as part of the augmented infoset. So the only way a choice can be inside a hidden group is if the head of the choice branches have default values. This is not sufficient. I do not think it was our intention to make hidden groups incapable of holding things complex enough to contain a choice. Rather, the point is to be able to hide things that have excess complexity needed only for parsing/unparsing, so that they need not appear in the final Infoset. So if a choice is allowed inside a hidden group, then a mechanism is needed by which when unparsing a choice branch is selected based on values of other parts of the Infoset, and then the elements of that branch are computed and added to the augmented infoset. Note that without hidden groups, this problem never arises, as the Infoset always has to take a position on choice branches by having an unambiguous element in it. There are 3 cases: 1. unparsing an xs:choice where for parsing, each branch has a discriminator, or is resolved by the simple "first successful parse" rule. 2. unparsing an xs:choice with dfdl:choiceDispatchKey 3. unparsing xs:choice with dfdl:initiatedContent='yes' For all cases, I suggest we must remove the restriction on dfdl:outputValueCalc being disallowed on choice branch head. For (1). First branch where the branch can be constructed successfully is added to the augmented infoset. This is a case where an unparse error is non-fatal, and causes a form of unparser backtracking through alternatives. Branches would be attempted in schema order. For (2) I suggest dfdl:unparseChoiceDispatchKey, evaluated at unparse time only. Produces a choice branch key. That choice branch is then constructed, and default values or dfdl:outputValueCalcs are used to fill in values. (BTW: this is the driving use case we have in Daffodil for this issue.) For (3).... I have no specific solution. I think it may not actually be a separate case and has to be treated as (1) or (2). Comments/Discussion?

2 1

Re: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 features/changes
by Mike Beckerle 28 Apr '19

28 Apr '19

So if a schema doesn't use dfdl:format at all, then it doesn't hurt to add <dfdl:format DFDLVersion="2"/> to it at top level in order to express version when DFDL v2.0 features are added to it. I think the alternative would be a separate annotation element <dfdl:version value="2"/> that is explicitly for this purpose, is allowed only at top level, etc. That would eliminate the complexity of getting a DFDLVersion by way of a dfdl:ref that includes a named format. It could be required to just be at lexical top level of every schema document that uses v2.0 features. The reason not to change namespaces, even though that is what we thought we should do originally, is based on various guidance such as http://www.xml.com/pub/a/2004/07/21/design.html These comments mostly apply to versioning of individual DFDL schemas, not of the schema-defining language itself. The quintessential example of a language version changing is XMLSchema itself which now has version 1.1. This still uses the same namespace as XMLSchema 1.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Jan 23, 2018 at 11:36 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike, > > Your scheme as proposed has a couple of problems. > a) It is possible to create a schema that does not use dfdl:format at all. > Unlikely but possible. With such a schema there is no way to indicate > version. > b) What are the version implications for annotations that have nothing to > do with dfdl:format? Asserts, discriminators, variables. > > Most of the language you use refers to versions of *schemas*. That's not > the same as version of a dfdl:format. Any reason you didn't just change the > DFDL namespace URL to be *http://www.ogf.org/dfdl/dfdl-2.0/* > <http://www.ogf.org/dfdl/dfdl-2.0/>, which must be declared before any > DFDL feature can be used? I think that was always the intent of the DFDL > namespace. > > The key piece of work is validating any proposal against the scoping rules > in section 8 of the spec. > > Regards > > Steve Hanson > > IBM Hybrid Integration, Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 <+44%201962%20815848> > mob:+44-7717-378890 <+44%207717%20378890> > Note: I work Tuesday to Friday > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org > Date: 23/01/2018 16:03 > Subject: [DFDL-WG] Action Item: Versioning mechanism for DFDL v2.0 > features/changes > Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> > ------------------------------ > > > > > There are some features now under discussion which will most likely not be > part of the DFDL v1.0 standard, but a subsequent version 2.0 of the > standard. > > Implementations may have these features prior to the existence of that > draft version of the DFDL standard; hence, a mechanism for indicating the > support, and requirements that implies, on DFDL schemas is needed in order > for these v2.0 features to be introduced. > > The purpose of the versioning mechanism is to provide access to DFDL v2.0 > features, while maintaining also backward compatibility with DFDL v1.0 > schemas. > > For example, new DFDL format properties may be added to the DFDL v2.0 > version; however, those properties are not required to be specified unless > the version of the schema is specifically set to indicate it is using > version 2.0 features. Also, deprecated features should only generate > deprecation warnings if the user is requesting version 2.0 of DFDL, and > should be silent otherwise. > > Proposal: > > New attribute on the dfdl:format annotation: DFDLVersion with valid values > "1", "2". All values are reserved for future use. > > The DFDLVersion, if unspecified, defaults to "1" which means the > implementation is of the DFDL v1.0 specification. > > If the DFDLVersion is "2", then new requirements for version 2.0 of DFDL > may be enforced on the schema, and schema definition errors will result if > version 2.0 requirements are not met. DFDL Version 1.0 features that are > deprecated will generate warnings. > > Example: > > <dfdl:format DFDLVersion="2" .../> > > Example: > > <dfdl:defineFormat name="myV2Format"> > <dfdl:format DFDLVersion="2" .../> > </dfdl:defineFormat> > > The DFDLVersion is not a format property and is not scoped. It can be > added to a named format definition which is then referenced in the usual > manner for named format definitions. > > Backward Compatibility Requirement > > DFDL Implementations that provide version 2.0 functionality must also > provide DFDL v1.0 compatibility. That is, version 2.0 subsumes and extends > version 1.0. > > Mixed Version Schemas > > If a schema consists of multiple schema documents, and their respective > dfdl:format annotations do not all contain the same value for the > DFDLVersion attribute, then the schema is said to be "mixed version". This > is allowed. > > A named dfdl:format carrying DFDLVersion "2" may not be referenced from a > DFDL Schema document having DFDLVersion "1". > > The opposite is allowed: A named dfdl:format carrying DFDLVersion="1" or > lacking the DFDLVersion attribute may be referenced from a DFDL Schema > having DFDLVersion="2". > > There are really 3 ways that DFDL version 2.0 can differ from version 1.0, > and these are that new features are added, old features are deprecated, or > old features are removed. This latter is quite undesirable, but will be > considered a possibility for purpose of discussion this versioning system. > > These will commonly be in combination such as if an old feature is > deprecated with the replacement feature being added. > > These will now be discussed separately, with the emphasis on how mixed > version schemas must work. > > New Features Added > > Each new required DFDL property introduced for version 2.0 will be defined > along with what is called its "version 1.0 default value". When a mixed > version schema exists, the DFDL v1.0 parts of it behave as if any required > DFDLVersion 2 properties have their version 1.0 default value defined, at > top level of the schema (as if defined with that default value in each > DFDLVersion 1 schema document's dfdl:format annotation). > > Features Removed > > Incompatibilities are to be avoided wherever possible, and may not be > needed at all. However, there is still the possibility that some feature > will want to be entirely removed and replaced by something better, and in > that case the versioning scheme needs to be able to support this. > > In that situation, the schema definition error for version 2 is only > generated when a schema has DFDLVersion 2 specified (in dfdl:format > explicitly, or by reference to a named format carrying or referencing > DFDLVersion 2). Mixed schemas that contain schema documents entirely using > version 1 of DFDL do not generate this schema definition error. > > Features deprecated > > Deprecation warnings are generated only when a DFDLVersion="2" attribute > is present or is incorporated by reference (to a named format). A version 1 > schema that is incorporated with a version 2 DFDL schema but is not > modified in any way, does not generate deprecation warnings about version > 1.0 features that have been deprecated in version 2.0. > > (Most likely deprecations would be revising property names or value enums > for clarity.) > > Implementations may provide mechanisms for suppressing warnings. These are > implementation dependent. > > Version 1.0 Existing Deprecations and Compatibility > > During the process of standardization for DFDL v1.0 a number of properties > names were changed. DFDL version 1.0 implementations accept both the old > and new property names. > (Example: dfdl:separatorSuppressionPolicy, which replaced an older > property name.) DFDL schemas that specify DFDLVersion="2" will not accept > the older property names, and it is a schema definition error if the old > (pre DFDL v1.0) property names are used. > > Properly constructed mixed version schemas can still depend on the older > property names in the version 1.0 parts of the schema. > > ------------------------------ > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > *www.tresys.com* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tresys.com&d=DwMFaQ…> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the *OGF Intellectual Property Policy* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5…> > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ogf.org_mailman_listinfo_dfdl-2Dwg&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=AJa9ThEymJXYnOqu84mJuw&m=lDIa8xvu2n-e8 > PXWCJmajfYaXESMqvvOWpg1HMWeuGQ&s=hdO_VQ_p-1f7JfZPcSTPR1S_Y1a > nSBXJWarqPdJZdPc&e= > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >

2 6

Action 292 - Write up of hexBinary with lengthUnits 'bits'
by Mike Beckerle 20 Nov '18

20 Nov '18

The proposed change is to allow lengthUnits 'bits' for hexBinary data. This turned out to be more complex to describe than I originally suspected because of the need to deal with XSD minLength and maxLength facets, which are always measured in bytes. Those in conjunction with dfdl:lengthKind='explicit' and dfdl:lengthUnits='bits' create some minor complexities. The below changes match the Daffodil implementation of this proposed feature. These sentences in the description of dfdl:lengthUnits in Section 12.3 must change. - 'bits' may only be used for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt, and xs:unsignedLong simple types with binary representation. - 'bytes' must be used for type xs:hexBinary. The text should read - 'bits' may only be used for xs:hexBinary and for xs:boolean, xs:byte, xs:short, xs:int, xs:long, xs:unsignedByte, xs:unsignedShort, xs:unsignedInt, and xs:unsignedLong simple types with binary representation. Later in the section, the paragraph: When unparsing a simple element with binary representation, then for hexBinary the length is the number of bytes in the infoset value padded to the XSD minLength facet value using dfdl:fillByte, and for the other types the length is the minimum number of bytes to represent the value and any sign. Must change to: When unparsing a simple element with binary representation, then for types other than hexBinary the length is the minimum number of bytes to represent the value and any sign. For type hexBinary when the dfdl:lengthUnits is 'bytes' then the length is the number of bytes in the infoset value padded to the XSD minLength facet value using dfdl:fillByte. For type hexBinary when the dfdl:lengthUnits is 'bits': - First the data is padded to XSD minLength bytes as if the dfdl:lengthUnits was 'bytes'. - When dfdl:lengthKind is other than 'explicit', the length is the number of bytes times 8. - When the dfdl:lengthKind is 'explicit' then the value is further padded or truncated to fit the target length, in bits. - if the data does not have sufficient bytes to supply the target length in bits it is a processing error. - if the data is longer than the minimum number of bytes needed to supply the target length in bits, it is a processing error. - If the explicit length in bits is not a multiple of 8, then the final byte is only partially unparsed according to the current dfdl:byteOrder. Section 12.3.7.2.7 Length of Binary Opaque Elements, the first sentence must be modified from: "The dfdl:lengthUnits property must be 'bytes'. It is a schema definition error otherwise." to "The dfdl:lengthUnits property must be 'bytes' or 'bits'. It is a schema definition error otherwise. Note that even when the dfdl:lengthUnits property is 'bits', the values of the XSD minLength and XSD maxLength facets are still always interpreted as constraining the length in units of bytes. That's the end of the actual proposed language. Note about alternatives: I considered the alternative to make it a schema definition error when dfdl:lengthUnits is 'bits' type is 'xs:hexBinary', and the XSD:minLength or XSD:maxLength facets are defined. I decided to go with the description above to support the use case where a data item is, in hex, some number of bytes long for a valid XML infoset, but the representation is explicitly a partial byte smaller. E.g., in a hexBinary with 8 bytes, but fewer than 64 bits in the representation. Ex: 63 bits as the explicit target length, so 1 bit will be unused from the xs:hexBinary logical value, but the above rules insure no more than 7 bits go unused from the final byte of the hexBinary logical value. This is trying to be consistent with the notion that we do not truncate data to fit into the available length except for xs:string when properties explicitly allow it. It is also trying to be consistent with our treatment of binary integers where a xs:long value may be output into an element having room for any number of bits, and any extra bits in the logical value are ignored. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 29 May '18

29 May '18

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 29 May '18

29 May '18

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 29 May '18

29 May '18

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 29 May '18

29 May '18

1 0

Apache Daffodil (incubating) 2.1.0 released
by Mike Beckerle 16 May '18

16 May '18

The Apache Daffodil (incubating) community is pleased to announce the release of version 2.1.0. Release notes and downloads are available at: https://daffodil.apache.org/releases/2.1.0/ Daffodil is an open source implementation of the DFDL (Data Format Description Language) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Action 304: Data Streaming (was) Re: Review/Feedback wanted on proposal: Data Streaming for base64 and other Layered Transformations
by Mike Beckerle 16 May '18

16 May '18

The data layering feature previously proposed has been implemented in Daffodil. There were minor changes from prior Wiki description. The wiki page is updated to reflect what the implementation actually does. The examples on the wiki page actually run and have been incorporated into Daffodil regression tests. https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Data+Layer… This feature is also being successfully used to create iCalendar and IMF format DFDL schemas. Those are not yet released for public consumption, but the feature is being successfully used for "real" formats and is thusfar, working as designed. Of the changes since prior draft of the design note, We changed a property value name. Property dfdl:layerLengthKind='boundaryMark' was done to replace the 'terminator' property value to avoid confusion with ordinary dfdl:terminator property. Thusfar we've not run into a need for a more general mechanism for passing parameters from the schema to the layering. The boundary mark, and the length have been sufficient, but this feels like an area where something more general may be needed in the future. For example, there are really 3 different kinds of layerings all known as "base64". Rather than having one general "base64" with a parameter, we're currently requiring that each variant have its own layering transform name, e.g., "base64_MIME" is one of them (the only one implemented thusfar). ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Action 297 - places in spec for clarifications about delimited binary
by Mike Beckerle 16 May '18

16 May '18

I searched the spec for discussion of delimited binary, scanning, and escaping. I can find really only 2 places where a clarification is appropriate and would be helpful: The definition in the glosssary: *Delimiter scanning* - When parsing, the process of scanning for a specific item in the input data which either marks the end of an item or the beginning of a subsequent item. Delimiter scanning also takes into account escape schemes so as to allow the delimiters to appear within data if properly escaped. changes to (wording change in blue) *Delimiter scanning* - When parsing, the process of scanning for a specific item in the input data which either marks the end of an item or the beginning of a subsequent item. Delimiter scanning of text representations also takes into account escape schemes so as to allow the delimiters to appear within data if properly escaped. Section 12.3.2.2 Delimited Binary Data This section is extended to include: Delimited binary data may have initiators, terminators, and separators in any supported encoding however escape schemes are not available for delimited binary data. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1