- dfdl-wg - lists.ogf.org

binaryNumberRep limitations, xs:decimal and binaryDecimalVirtualPoint
by Mike Beckerle 10 Aug '22

10 Aug '22

Rather long email, apologies in advance. We are encountering many issues where DFDL is too limited in its binary number representations. 6 points are made in this email. To not bury the lead, which is Point 6, I am proposing that we make dfdl:binaryNumberRep an extensible enum, allowing QNames as values so that implementations can extend the set of supported binary number representations. E.g., dfdl:binaryNumberRep='dfdlx:onesComplement', or dfdl:binaryNumberRep='daffodil:signedASN1BERVariableLengthInteger' Point 1: I think this is just a needed clarification. But Points 1 and 2 drove the motivation for this whole discussion. So the description of dfdl:binaryDecimalVirtualPoint says it is allowed on types whose *base* is xs:decimal. That term 'base' is confusing. It can either mean allowed on types derived from xs:decimal, as all the signed and unsigned integer types are. Or it can mean literally <restriction base="xs:decimal"> .... </restriction>. That is, the base attribute of simple type restriction must be xs:decimal, or the type="xs:decimal" directly on the element. I believe the latter is the only thing that makes sense. If dfdl:binaryDecimalVirtualPoint is positive, as this type below is nonsense, as the type says it is an unsigned integer, but the binaryDecimalVirtualPoint says to divide by 100, so that 36000 would become 360.00, a non-integer. The infoset <angle>360.00</angle> makes no sense for an integer type, and would not validate treating the DFDL schema as an XSD. <element name="angle" type="with2DecimalFractionDigits"/> <simpleType name="with2DecimalFractionDigits" dfdl:binaryDecimalVirtualPoint="2">  <restriction base="xs:unsignedShort">  <maxInclusive value="360"/> </restriction> </simpleType> So based on that reasoning, I think we need to improve the clarity and say that dfdl:binaryDecimalVirtualPoint applies only to type xs:decimal alone. (Removing the word 'base' which creates the subtype confusion.) That brings me to the next related point. Point 2: Need for binaryNumberRep='unsignedBinary' We have a use case we cannot express. An angle from 0 to 360 degrees is represented by an unsigned 16 bit binary integer which if dfdl:binaryDecimalVirtualPoint works with that, can represent 000.00 to 360.00. <simpleType name="angle360" dfdl:lengthKind="explicit" dfdl:length="16" dfdl:binaryNumberRep="binary" dfdl:binaryDecimalVirtualPoint="2"> <restriction base="xs:decimal"> <minInclusive value="0"/> <maxInclusive value="360"/> </restriction> </simpleType> However, we have no way to say that the above is to be unsigned binary integer representation. dfdl:binaryNumberRep='binary' means unsigned binary if the type is unsigned, and means signed twos-complement if the type is signed, which xs:decimal is. So my definition of angle360 above is no good, as the maximum positive value is 327.67, which is insufficient. I think we need to revise dfdl:binaryNumberRep to allow for distinguishing binary unsigned from twosComplement signed types, as well as the packed types. Note that the packed types allow 'bcd' which is an unsigned representation, so there is sort of a precedent there for allowing the binaryNumberRep type to be unsigned even if the type is signed-capable. There is already a proposal to add "offsetBinary" as a signed binary integer representation. https://github.com/OpenGridForum/DFDL/issues/7 So I'm suggesting adding "unsignedBinary' as well. So the complete set (so far) would be dfdl:binaryNumberRep: - 'twosComplementBinary' (with legacy 1.0 name 'binary') - 'unsignedBinary' - 'offsetBinary' - 'packed' - 'bcd' - 'ibm4690Packed' Point 3: There are other binary integer representations that will be needed. This table comes from a format specification we use: [image: image.png] Ignore the 'Logical' column above, that's about enums. Ignore the "*" which is just about when a value must be reserved as an in-band null indicator which is the suggested such value. What is called 'Mod Twos Complement' here is what our existing proposed DFDL 2.0 feature calls 'offsetBinary'. So this table suggests the need for 'unsignedBinary' (already mentioned), but also two others: 'signPlusMagnitudeBinary', and 'onesComplementBinary'. Point 4: Zig Zag Integer representation is getting popular There's one other representation I know of which is more recent/modern called zig-zag integers, popularized by google protocol buffers, but it's a clever representation and seemingly used in many places now. *Binary Value* *Zig Zag* 000 0 001 -1 010 1 011 -2 100 2 101 -3 110 3 111 -4 Point 5: Variable Length Binary Integers There are also variable-length integer formats that are not just strings of bits. A common one I have seen is used by ASN.1 BER representation where each byte if its MSB is 1 indicates that the integer extends an additional byte, contributing 7 bits to the value. Unsigned integers are just the concatenation of these bits. Signed integers are handled after the bits are concatenated together. If the first bit of the concatenation is 1, the value is twos complement negative value. Hence, if a positive value would have a first bit of 1, then an additional byte containing 10000000 must be used as the most significant byte so that the first bit will not be 1. There is no way in DFDL to represent such a variable length integer representation and get an integer in the infoset. You have to use a hexBinary byte array. There is a need for a variable-length integer like this to support not only explicit length (used by ASN.1 BER), but implicit length as well. In this case the last byte of the variable length integer does not have the MSB set. Hence, a single byte can represent signed -64 to +63, or unsigned 0 to 127. Outside that range multiple bytes must be used, each byte contributing 7 bits. This suggests a need for several additional dfdl:binaryNumberRep enums. Point 6: Extensibility by implementations is needed here There are many other representations out there as well. I think we should have a convention where there is a core set that all DFDL representations must provide, and a convention by which DFDL implementations can provide additional support. To me, a good way to do this is to allow the enum values for dfdl:binaryNumberRep to be not only regular enums (all of which are reserved) but QName syntax, where the prefix can be for a namespace recognized by an implementation for providing an extended set of binary number representations. (Perhaps the dfdlx: prefix and namespace, or maybe we just allow implementation specific namespaces?) This means of extending enums for existing properties is not part of our existing 'experimental features' conventions, but I propose that it should be added. To me, this is a good way to generally allow property enums to be extended with experimental features in DFDL implementations, and applies to other places such as dfdl:binaryCalendarRep, and numerous other properties where we are finding a need for additional enums and want to add them as experimental features. That was long. Thanks for your consideration. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

default case needed for choice-by-dispatch
by Mike Beckerle 15 Jul '22

15 Jul '22

I know this feature was discussed at one time. The priority of this has gone up for us substantially, as the idiom in DFDL v1.0 for a choice-by-dispatch to have a default case when the choiceDispatchKey doesn't match any choiceBranchKey is far too complex. This appears to be what is needed:  <group name="discriminateTrue"> <sequence> <annontation><appinfo ...> <dfdl:discriminator>{ fn:true() }</dfdl:discriminator> </appinfo></annotation> </sequence> </group> .... <choice> <choice dfdl:choiceDispatchKey='{ ... dispatch key expr... }'> <sequence dfdl:choiceBranchKey="key1"> <group ref="p:discriminateTrue"/> ... parse the key1 branch .... </sequence> ... <sequence dfdl:choiceBranchKey="keyN"> <group ref="p:discriminateTrue"/> ... parse the keyN branch .... </sequence> </choice>  <element name="invalidKey" ....> ... </element> </choice> ..... The need for a discriminator for the outer choice on every single branch of the inner choice is tedious at best. This is needed because if you don't have this discriminator, then the backtrack will choose the other choice alternative for any failure of a branch, not just no branch key match, and one will get a diagnostic that the branch key was not found when it was found but the subsequent parse failed. This structure is too complex for DFDL schema authors to get right. A way to specify an additional choice branch which is the default to be used on no-match to the other branch keys, is needed. An explicit dfdlx:choiceBranchDefault="yes" is one way to do this. A final choice branch with no dfdl:choiceBranchKey would also work but seems too error-prone for my taste. A default case should be explicitly marked as such, even if we require it to be the last branch. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

updates to experimental features convention document: enums
by Mike Beckerle 14 Jul '22

14 Jul '22

Updates to allow enumeration properties to be extended with new experimental enum values. See https://github.com/OpenGridForum/DFDL/pull/32 Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Suggest new lengthKind 'patternMatch' where failure to match is a parse error
by Mike Beckerle 12 Apr '22

12 Apr '22

Apache Daffodil users have had quite a lot of trouble with understanding and proper use of dfdl:lengthKind 'pattern'. This is due to the fact that no match does *not* cause a parse error, but provides a successful parse with length of zero. People generally find this unintuitive given that if they wanted a zero-length match they could have defined their regex to allow a zero-length match. I have made this mistake repeatedly myself when creating DFDL schemas, and supposedly I'm an expert in DFDL. This has been so problematic that I suggest we add an additional enum for lengthKind of 'patternMatch' or maybe 'patternMatchRequired' (I'm open to suggestions for best name here) which is the same as 'pattern', except that failure to match results in a parse error instead of zero length success. I would argue that the existing 'pattern' behavior is badly designed, but it is too late to change it for DFDL v1.0. Rather, for DFDL v2.0 we should add a new correct behavior and call it 'patternMatch' and then we can deprecate the existing lengthKind 'pattern'. Has anyone else had similar difficult experience with lengthKind 'pattern' ? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 1

[ANNOUNCE] Apache Daffodil VS Code 1.0.0 Released
by Mike Beckerle 05 Apr '22

05 Apr '22

The Apache Daffodil community is pleased to announce the release of version 1.0.0 of the Daffodil VS Code Extension. This is the first official release of the Apache Daffodil VS Code Extension. Detailed release notes and downloads are available at: https://daffodil.apache.org/vscode/1.0.0/ The Apache Daffodil VS Code Extension is an open-source extension that utilizes Apache Daffodil to allow users to interactively debug a DFDL schema inside of VS Code. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team

1 0

[ANNOUNCE] Apache Daffodil 3.3.0 Released
by Mike Beckerle 22 Mar '22

22 Mar '22

The Apache Daffodil community is pleased to announce the release of version 3.3.0. Notable changes in this release include bug fixes, improvements, and a new charset. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/3.3.0/ Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team

1 0

DFDL as an ISO standard
by Steve Hanson 24 Feb '22

24 Feb '22

Alan, Wolfgang From the minutes of the last DFDL WG call in November, "OGF will write to ISO-IEC JST1 with the proposal" to make DFDL an ISO standard. Tomorrow is the first WG call since then, so just following up on that action. Regards Steve Hanson IBM Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-7717-378890 Note: I work Tuesday to Friday -----Original Message----- mob:+44-7717-378890 Note: I work Tuesday to Friday ----- Forwarded by Steve Hanson/UK/IBM on 08/02/2022 16:27 ----- From: "Steve Hanson" <smh(a)uk.ibm.com> To: "Wolfgang Ziegler" <wolfgang.ziegler(a)z-rands.de> Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)owlcyberdefense.com>, "DFDL Schemas for Commercial and Scientific Data Formats" <smhdfdl(a)gmail.com>, "Michele Zundo" <michele.zundo(a)esa.int>, "Liam O'Neill" <WILONEIL(a)uk.ibm.com>, "Sill, Alan" <Alan.Sill(a)ttu.edu> Date: 12/11/2021 13:19 Subject: [EXTERNAL] RE: [DFDL-WG] Agenda for OGF DFDL WG Call ________________________________ Hi Wolfgang Thanks for the links. This is also a useful link, as it shows the DMTF adoptions at a higher level -https://www.dmtf.org/about/register/apresourcesRegards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, ‍‍‍‍‍‍‍‍‍‍‍ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Wolfgang Thanks for the links. This is also a useful link, as it shows the DMTF adoptions at a higher level - https://www.dmtf.org/about/register/apresources<https://www.dmtf.org/about/register/apresources> Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL<http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: "Wolfgang Ziegler" <wolfgang.ziegler(a)z-rands.de> To: "Steve Hanson" <smh(a)uk.ibm.com>, dfdl-wg(a)ogf.org Cc: "Mike Beckerle" <mbeckerle(a)owlcyberdefense.com>, "DFDL Schemas for Commercial and Scientific Data Formats" <smhdfdl(a)gmail.com>, "Michele Zundo" <michele.zundo(a)esa.int>, "Liam O'Neill" <WILONEIL(a)uk.ibm.com>, "Sill, Alan" <Alan.Sill(a)ttu.edu> Date: 11/11/2021 17:00 Subject: [EXTERNAL] Re: [DFDL-WG] Agenda for OGF DFDL WG Call ________________________________ Dear Steve, dear Mike, the example for a standard developed outside of ISO and then published as an ISO standard is Open Virtualization Format Specification from DMTF https://www.dmtf.org/sites/default/files/standards/documents/DSP0243_1.1.0.…<https://www.dmtf.org/sites/default/files/standards/documents/DSP0243_1.1.0.…> as ISO standard https://www.iso.org/standard/72081.html<https://www.iso.org/standard/72081.html> You can find the description of the PAS process (among other things) here https://www.iec.ch/members_experts/refdocs/iec/Consolidated_JTC1_Supplement…<https://www.iec.ch/members_experts/refdocs/iec/Consolidated_JTC1_Supplement…> The WG5 chair is Fernando Gebara Filho <fgebara(a)amazon.com> Please send me a mail if you have more questions. Best regards Wolfgang Am 11.11.21 um 15:02 schrieb Steve Hanson: > Please find agenda for next call on GitHub > athttps://github.com/OpenGridForum/DFDL/blob/master/calls/2021-11-11_DFDL-W… > <https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-11-11_DFDL-WG-…<https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-11-11_DFDL-WG-…>> > > > Regards > Steve Hanson > > IBM Hybrid Integration, Hursley, UK > Architect, _IBM DFDL_ <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, _OGF DFDL Working Group_ <http://www.ogf.org/dfdl/<http://www.ogf.org/dfdl/>>_ > __smh(a)uk.ibm.com_ <mailto:smh@uk.ibm.com> > tel:+44-1962-815848 > mob:+44-7717-378890 > Note: I work Tuesday to Friday > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg<https://www.ogf.org/mailman/listinfo/dfdl-wg> >

2 1

dfdl:prefixLengthType with dfdl:assert on it
by Mike Beckerle 17 Feb '22

17 Feb '22

I believe the DFDL spec does not specify whether the prefixLengthType for an element of lengthKind 'prefixed' can have assertions, and if those assertions are evaluated, and when. Consider this example below. We really want the assert on the bufferPayload type to be evaluated before the length computed from the prefix is used as a length because we'd like good diagnostic behavior that points out that the prefix value is way out of realistic range. In many formats we see 4-byte length prefixes, but the range of values for specific fields is relatively tiny, often even just 1 to 100. Enforcing this via the assert expressed on the prefixLengthType's simpleType definition makes this easy. I could go either way on whether a prefixLengthType can carry a discriminator. But an assert should definitely be allowed. ... <complexType> <sequence> <element name="payload" type="xs:hexBinary" dfdl:lengthKind="prefixed" dfdl:prefixIncludesPrefixLength="no" dfdl:prefixLengthType="p8:bufferPayload"/> </sequence> </complexType> ... <simpleType name="bufferPayload"> <annotation> <appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert> </appinfo> </annotation> <restriction base="p8:unsignedLong"> <minInclusive value="0"/> <maxInclusive value="32768"/> </restriction> </simpleType>

2 3

OGF DFDL Working Group Call
by Steve Hanson 08 Feb '22

08 Feb '22

Calls for 2022 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools…

1 0

revisit: nillable complex types and %ES restriction
by Mike Beckerle 25 Jan '22

25 Jan '22

The Daffodil user community is complaining about the DFDL v1.0 restriction that nillable complex types can only have %ES; (empty data) as their nilled representation. Multiple important data standards find this restriction quite awkward. It has significant negative impact on usability because it eliminates polymorphism of paths. The path needed to refer to an element value has different ending element path steps depending on whether it is nilled or not. The value of having DFDL create an XML infoset is lost if something as basic as this cannot be done in such a way that path expressions work alike to how they would work in regular XML/XSD. Is there a significant reason why this restriction cannot be lifted? If not we would probably plan to define a dfdlx namespace property that can be used to enable this alternate behavior, and then we could prototype this new fuctionality in Daffodil and propose this for DFDL 2.0 revision.

2 1