March 2021 - dfdl-wg - lists.ogf.org

Latest OGF DFDL WG Call Minutes
by Steve Hanson 23 Mar '21

23 Mar '21

Please find minutes from the latest call at https://github.com/OpenGridForum/DFDL/wiki/DFDL-Workgroup-News Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 17 Mar '21

17 Mar '21

Please find agenda for next call on GitHub at https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-03-18_DFDL-WG-… Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

OGF DFDL Working Group Call
by Steve Hanson 17 Mar '21

17 Mar '21

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL v1.0 Specification is Final "Recommendation" - Fwd: [wg-all] New document published for DFDL-WG, #240
by Mike Beckerle 10 Mar '21

10 Mar '21

In case this announcement slipped by anyone...... This is the formal announcement that DFDL v1.0 Spec. is now a full "Recommendation" from the OGF. That's the term OGF uses for full standards, i.e., that are no longer in "proposed" status. The announcement to *wg-all(a)ogf.org* <wg-all(a)ogf.org> doesn't say it's a full recommendation. It just says "new document", but if you follow the link (*https://www.ogf.org/ogf/doku.php/documents/documents* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_ogf_doku.p…>) to the list of OGF documents, you see that GFD.240 is the top row, and has the 3 important little letters "REC" next to this document. Those indicate the status as a "RECommendation". So v1.0 is official, done. Congratulations and many thanks to all who have been contributing so much to making this happen for so long. It's kind of hard to imagine that I have been working (off and on) on DFDL since 2001, so 20 years. Given that I turned 60 this year, that's 1/3 of my life and 1/2 of my career. My full time job became DFDL only in the 2011 time frame, but that's 10 years of full time (mostly) on DFDL or DFDL-related work. Steve Hanson is the other co-chair of the DFDL Workgroup at OGF, working (off and on) on DFDL since 2004 with a full-time stretch from 2011 to 2015. The contributors to this workgroup, along with the co-chairs assembled together the collective knowledge from many data integration systems and tools, and collective knowledge of a huge amount of data experience to synthesize the DFDL standard. I believe to have real impact on the computing industry today, you have to have an open standard, long-term commercial commitment, commercial adoption/product, and an open-source implementation as well, along with enough time and attention to actually finish the work. OGF provided a patient forum for the open-standard to come to fruition. Being a small organization OGF was actually perfect for DFDL given the long time it took to complete the work. A larger standards body might have forced things to happen too fast, or the whole process might have been derailed by too many participants. IBM provided the commercial support and the first production-ready implementation called IBM DFDL, used in several of its key products, and the US DoD funded several core contributors to work on what has become the Open Source Apache Daffodil implementation. All these implementations were hard to build, and took real time and effort from many engineers. They cover very different data formats that improved the standard by making sure it was far more comprehensive than prior-generation technologies. The official PDF of the standard (er "Recommendation") is *https://www.ogf.org/documents/GFD.240.pdf* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_documents_…> Mike Beckerle OGF DFDL Workgroup Co-Chair Apache Daffodil PMC Chair Owl Cyber Defense - Principal Engineer Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5…> ---------- Forwarded message --------- From: *Greg Newby* <*gbnewby(a)petascale.org* <gbnewby(a)petascale.org>> Date: Mon, Mar 1, 2021 at 9:48 AM Subject: [wg-all] New document published for DFDL-WG, #240 To: All WG <*wg-all(a)ogf.org* <wg-all(a)ogf.org>> Dear OGF community, A new document has been published by OGF. All OGF documents may be found here: *https://www.ogf.org/ogf/doku.php/documents/documents* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_ogf_doku.p…> This document was provided by the Data Format Description Language Working Group (DFDL-WG). *GFD-R-P.240: Data Format Description Language (DFDL) v.1.0 Specification* Authors: Michael J. Beckerle, Owl Cyber Defense/Tresys and Stephen M. Hanson, IBM Abstract: This document provides a definition of a standard Data Format Description Language (DFDL). This language allows description of text, dense binary, and legacy data formats in a vendor-neutral declarative manner. DFDL is an extension to the XML Schema Description Language (XSD). This document obsoletes GFD #207 and GFD #174. - Greg Newby OGF Standards Editor -- The wg-all mailing list is an umbrella list for all OGF mailing lists. If you are subscribed to any OGF mailing list, you will receive mails to wg-all. wg-all mailing list *wg-all(a)ogf.org* <wg-all(a)ogf.org> *https://www.ogf.org/mailman/listinfo/wg-all* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li…> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

possible issue - can't use pattern to validate DFDL content containing newlines
by Mike Beckerle 04 Mar '21

04 Mar '21

Since data contains any characters, the DFDL infoset allows any characters. However, XML does not allow any characters. Furthermore, XML Schema Pattern facets are expressed using this XML Schema fragment: <xs:pattern value="...some regex pattern here ..."/> But XML attributes are normalized by XML readers/parsers. Line endings in them are converted to single spaces. So <xs:pattern value="abc def"/> is equivalent to: <xs:pattern value="abc def"/> Furthermore <xs:pattern value="abc
def"/> is also normalized to <xs:pattern value="abc def"/> As far as I can tell there is no alternate notation to this. This means, if you want to use a pattern facet to specify that a DFDL infoset string can contain A-Za-z0-9 spaces and line endings, there is no way to express this. This pattern was the example I was dealing with. <xs:pattern value="[A-Za-z0-9 
]*"/> If you look at the string for the value attribute of this pattern element, that string already has the line ending characters converted into spaces. The attribute value is "[A-Za-z0-9 ]*" which has 3 spaces before the "]". I think there is no workaround for this in XML, XSD, or DFDL. I dug into the Daffodil implementation and in the code that accesses this attribute, you don't even get a NodeSeq containing a mixture of Text and Entity nodes. You just get a single Text node. So it is pretty well hopeless without reaching under the XML parser/reader's guts. Hence, in DFDL if you want to "validate" that a DFDL string contains content that includes line-endings with a regex, you have to use dfdl:assert with failureType="recoverableError" testKind="pattern" and testPattern with the regex of interest. This is then a DFDL regex, which is a Java regex, and you can be explicit about line endings allowed. You can't do it with a pattern facet. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 1

Contributing to github
by Bradd Kadlecik 03 Mar '21

03 Mar '21

Hi, I was unable to create a branch in github to upload the experience document for indirection. My userid there is tpfbraddk. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States

2 6