- dfdl-wg - lists.ogf.org

updates to experimental features convention document: enums
by Mike Beckerle 14 Jul '22

14 Jul '22

Updates to allow enumeration properties to be extended with new experimental enum values. See https://github.com/OpenGridForum/DFDL/pull/32 Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Suggest new lengthKind 'patternMatch' where failure to match is a parse error
by Mike Beckerle 12 Apr '22

12 Apr '22

Apache Daffodil users have had quite a lot of trouble with understanding and proper use of dfdl:lengthKind 'pattern'. This is due to the fact that no match does *not* cause a parse error, but provides a successful parse with length of zero. People generally find this unintuitive given that if they wanted a zero-length match they could have defined their regex to allow a zero-length match. I have made this mistake repeatedly myself when creating DFDL schemas, and supposedly I'm an expert in DFDL. This has been so problematic that I suggest we add an additional enum for lengthKind of 'patternMatch' or maybe 'patternMatchRequired' (I'm open to suggestions for best name here) which is the same as 'pattern', except that failure to match results in a parse error instead of zero length success. I would argue that the existing 'pattern' behavior is badly designed, but it is too late to change it for DFDL v1.0. Rather, for DFDL v2.0 we should add a new correct behavior and call it 'patternMatch' and then we can deprecate the existing lengthKind 'pattern'. Has anyone else had similar difficult experience with lengthKind 'pattern' ? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 1

[ANNOUNCE] Apache Daffodil VS Code 1.0.0 Released
by Mike Beckerle 05 Apr '22

05 Apr '22

The Apache Daffodil community is pleased to announce the release of version 1.0.0 of the Daffodil VS Code Extension. This is the first official release of the Apache Daffodil VS Code Extension. Detailed release notes and downloads are available at: https://daffodil.apache.org/vscode/1.0.0/ The Apache Daffodil VS Code Extension is an open-source extension that utilizes Apache Daffodil to allow users to interactively debug a DFDL schema inside of VS Code. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team

1 0

[ANNOUNCE] Apache Daffodil 3.3.0 Released
by Mike Beckerle 22 Mar '22

22 Mar '22

The Apache Daffodil community is pleased to announce the release of version 3.3.0. Notable changes in this release include bug fixes, improvements, and a new charset. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/3.3.0/ Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team

1 0

DFDL as an ISO standard
by Steve Hanson 24 Feb '22

24 Feb '22

Alan, Wolfgang From the minutes of the last DFDL WG call in November, "OGF will write to ISO-IEC JST1 with the proposal" to make DFDL an ISO standard. Tomorrow is the first WG call since then, so just following up on that action. Regards Steve Hanson IBM Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-7717-378890 Note: I work Tuesday to Friday -----Original Message----- mob:+44-7717-378890 Note: I work Tuesday to Friday ----- Forwarded by Steve Hanson/UK/IBM on 08/02/2022 16:27 ----- From: "Steve Hanson" <smh(a)uk.ibm.com> To: "Wolfgang Ziegler" <wolfgang.ziegler(a)z-rands.de> Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)owlcyberdefense.com>, "DFDL Schemas for Commercial and Scientific Data Formats" <smhdfdl(a)gmail.com>, "Michele Zundo" <michele.zundo(a)esa.int>, "Liam O'Neill" <WILONEIL(a)uk.ibm.com>, "Sill, Alan" <Alan.Sill(a)ttu.edu> Date: 12/11/2021 13:19 Subject: [EXTERNAL] RE: [DFDL-WG] Agenda for OGF DFDL WG Call ________________________________ Hi Wolfgang Thanks for the links. This is also a useful link, as it shows the DMTF adoptions at a higher level -https://www.dmtf.org/about/register/apresourcesRegards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, ‍‍‍‍‍‍‍‍‍‍‍ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Wolfgang Thanks for the links. This is also a useful link, as it shows the DMTF adoptions at a higher level - https://www.dmtf.org/about/register/apresources<https://www.dmtf.org/about/register/apresources> Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL<http://www.ibm.com/developerworks/library/se-dfdl/index.html> Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/> smh(a)uk.ibm.com<mailto:smh@uk.ibm.com> tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: "Wolfgang Ziegler" <wolfgang.ziegler(a)z-rands.de> To: "Steve Hanson" <smh(a)uk.ibm.com>, dfdl-wg(a)ogf.org Cc: "Mike Beckerle" <mbeckerle(a)owlcyberdefense.com>, "DFDL Schemas for Commercial and Scientific Data Formats" <smhdfdl(a)gmail.com>, "Michele Zundo" <michele.zundo(a)esa.int>, "Liam O'Neill" <WILONEIL(a)uk.ibm.com>, "Sill, Alan" <Alan.Sill(a)ttu.edu> Date: 11/11/2021 17:00 Subject: [EXTERNAL] Re: [DFDL-WG] Agenda for OGF DFDL WG Call ________________________________ Dear Steve, dear Mike, the example for a standard developed outside of ISO and then published as an ISO standard is Open Virtualization Format Specification from DMTF https://www.dmtf.org/sites/default/files/standards/documents/DSP0243_1.1.0.…<https://www.dmtf.org/sites/default/files/standards/documents/DSP0243_1.1.0.…> as ISO standard https://www.iso.org/standard/72081.html<https://www.iso.org/standard/72081.html> You can find the description of the PAS process (among other things) here https://www.iec.ch/members_experts/refdocs/iec/Consolidated_JTC1_Supplement…<https://www.iec.ch/members_experts/refdocs/iec/Consolidated_JTC1_Supplement…> The WG5 chair is Fernando Gebara Filho <fgebara(a)amazon.com> Please send me a mail if you have more questions. Best regards Wolfgang Am 11.11.21 um 15:02 schrieb Steve Hanson: > Please find agenda for next call on GitHub > athttps://github.com/OpenGridForum/DFDL/blob/master/calls/2021-11-11_DFDL-W… > <https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-11-11_DFDL-WG-…<https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-11-11_DFDL-WG-…>> > > > Regards > Steve Hanson > > IBM Hybrid Integration, Hursley, UK > Architect, _IBM DFDL_ <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, _OGF DFDL Working Group_ <http://www.ogf.org/dfdl/<http://www.ogf.org/dfdl/>>_ > __smh(a)uk.ibm.com_ <mailto:smh@uk.ibm.com> > tel:+44-1962-815848 > mob:+44-7717-378890 > Note: I work Tuesday to Friday > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg<https://www.ogf.org/mailman/listinfo/dfdl-wg> >

2 1

dfdl:prefixLengthType with dfdl:assert on it
by Mike Beckerle 17 Feb '22

17 Feb '22

I believe the DFDL spec does not specify whether the prefixLengthType for an element of lengthKind 'prefixed' can have assertions, and if those assertions are evaluated, and when. Consider this example below. We really want the assert on the bufferPayload type to be evaluated before the length computed from the prefix is used as a length because we'd like good diagnostic behavior that points out that the prefix value is way out of realistic range. In many formats we see 4-byte length prefixes, but the range of values for specific fields is relatively tiny, often even just 1 to 100. Enforcing this via the assert expressed on the prefixLengthType's simpleType definition makes this easy. I could go either way on whether a prefixLengthType can carry a discriminator. But an assert should definitely be allowed. ... <complexType> <sequence> <element name="payload" type="xs:hexBinary" dfdl:lengthKind="prefixed" dfdl:prefixIncludesPrefixLength="no" dfdl:prefixLengthType="p8:bufferPayload"/> </sequence> </complexType> ... <simpleType name="bufferPayload"> <annotation> <appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert>{ dfdl:checkConstraints(.) }</dfdl:assert> </appinfo> </annotation> <restriction base="p8:unsignedLong"> <minInclusive value="0"/> <maxInclusive value="32768"/> </restriction> </simpleType>

2 3

OGF DFDL Working Group Call
by Steve Hanson 08 Feb '22

08 Feb '22

Calls for 2022 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools…

1 0

revisit: nillable complex types and %ES restriction
by Mike Beckerle 25 Jan '22

25 Jan '22

The Daffodil user community is complaining about the DFDL v1.0 restriction that nillable complex types can only have %ES; (empty data) as their nilled representation. Multiple important data standards find this restriction quite awkward. It has significant negative impact on usability because it eliminates polymorphism of paths. The path needed to refer to an element value has different ending element path steps depending on whether it is nilled or not. The value of having DFDL create an XML infoset is lost if something as basic as this cannot be done in such a way that path expressions work alike to how they would work in regular XML/XSD. Is there a significant reason why this restriction cannot be lifted? If not we would probably plan to define a dfdlx namespace property that can be used to enable this alternate behavior, and then we could prototype this new fuctionality in Daffodil and propose this for DFDL 2.0 revision.

2 1

clarification: lengthKind 'delimited' but no terminating delimiter in scope
by Mike Beckerle 25 Jan '22

25 Jan '22

Is this an SDE? Or allowed? Daffodil currently implements this as allowed, and it ends up behaving much like "endOfParent", i.e., a scan for delimiters, when none are specified, scans as far as it can. However, given that we have endOfParent to express this behavior, seems to me that delimited with no delimiter should be an SDE. Thoughts?

2 1

OGF DFDL WG Calls for 2022
by Steve Hanson 19 Jan '22

19 Jan '22

Due to an imminent change in my mail technology, I am deferring sending out calendar invites for our 2022 calls until February. I anticipate the first call will be 10th February. In 2021 we only had 7 calls, so for 2022 a call every 4 weeks would make sense. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0