- dfdl-wg - lists.ogf.org

OGF DFDL Working Group Call
by Steve Hanson 13 Apr '21

13 Apr '21

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 23 Mar '21

23 Mar '21

Please find minutes from the latest call at https://github.com/OpenGridForum/DFDL/wiki/DFDL-Workgroup-News Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 18 Mar '21

18 Mar '21

Please find agenda for next call on GitHub at https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-03-18_DFDL-WG-… Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

OGF DFDL Working Group Call
by Steve Hanson 18 Mar '21

18 Mar '21

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL v1.0 Specification is Final "Recommendation" - Fwd: [wg-all] New document published for DFDL-WG, #240
by Mike Beckerle 10 Mar '21

10 Mar '21

In case this announcement slipped by anyone...... This is the formal announcement that DFDL v1.0 Spec. is now a full "Recommendation" from the OGF. That's the term OGF uses for full standards, i.e., that are no longer in "proposed" status. The announcement to *wg-all(a)ogf.org* <wg-all(a)ogf.org> doesn't say it's a full recommendation. It just says "new document", but if you follow the link (*https://www.ogf.org/ogf/doku.php/documents/documents* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_ogf_doku.p…>) to the list of OGF documents, you see that GFD.240 is the top row, and has the 3 important little letters "REC" next to this document. Those indicate the status as a "RECommendation". So v1.0 is official, done. Congratulations and many thanks to all who have been contributing so much to making this happen for so long. It's kind of hard to imagine that I have been working (off and on) on DFDL since 2001, so 20 years. Given that I turned 60 this year, that's 1/3 of my life and 1/2 of my career. My full time job became DFDL only in the 2011 time frame, but that's 10 years of full time (mostly) on DFDL or DFDL-related work. Steve Hanson is the other co-chair of the DFDL Workgroup at OGF, working (off and on) on DFDL since 2004 with a full-time stretch from 2011 to 2015. The contributors to this workgroup, along with the co-chairs assembled together the collective knowledge from many data integration systems and tools, and collective knowledge of a huge amount of data experience to synthesize the DFDL standard. I believe to have real impact on the computing industry today, you have to have an open standard, long-term commercial commitment, commercial adoption/product, and an open-source implementation as well, along with enough time and attention to actually finish the work. OGF provided a patient forum for the open-standard to come to fruition. Being a small organization OGF was actually perfect for DFDL given the long time it took to complete the work. A larger standards body might have forced things to happen too fast, or the whole process might have been derailed by too many participants. IBM provided the commercial support and the first production-ready implementation called IBM DFDL, used in several of its key products, and the US DoD funded several core contributors to work on what has become the Open Source Apache Daffodil implementation. All these implementations were hard to build, and took real time and effort from many engineers. They cover very different data formats that improved the standard by making sure it was far more comprehensive than prior-generation technologies. The official PDF of the standard (er "Recommendation") is *https://www.ogf.org/documents/GFD.240.pdf* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_documents_…> Mike Beckerle OGF DFDL Workgroup Co-Chair Apache Daffodil PMC Chair Owl Cyber Defense - Principal Engineer Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5…> ---------- Forwarded message --------- From: *Greg Newby* <*gbnewby(a)petascale.org* <gbnewby(a)petascale.org>> Date: Mon, Mar 1, 2021 at 9:48 AM Subject: [wg-all] New document published for DFDL-WG, #240 To: All WG <*wg-all(a)ogf.org* <wg-all(a)ogf.org>> Dear OGF community, A new document has been published by OGF. All OGF documents may be found here: *https://www.ogf.org/ogf/doku.php/documents/documents* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_ogf_doku.p…> This document was provided by the Data Format Description Language Working Group (DFDL-WG). *GFD-R-P.240: Data Format Description Language (DFDL) v.1.0 Specification* Authors: Michael J. Beckerle, Owl Cyber Defense/Tresys and Stephen M. Hanson, IBM Abstract: This document provides a definition of a standard Data Format Description Language (DFDL). This language allows description of text, dense binary, and legacy data formats in a vendor-neutral declarative manner. DFDL is an extension to the XML Schema Description Language (XSD). This document obsoletes GFD #207 and GFD #174. - Greg Newby OGF Standards Editor -- The wg-all mailing list is an umbrella list for all OGF mailing lists. If you are subscribed to any OGF mailing list, you will receive mails to wg-all. wg-all mailing list *wg-all(a)ogf.org* <wg-all(a)ogf.org> *https://www.ogf.org/mailman/listinfo/wg-all* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li…> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

possible issue - can't use pattern to validate DFDL content containing newlines
by Mike Beckerle 04 Mar '21

04 Mar '21

Since data contains any characters, the DFDL infoset allows any characters. However, XML does not allow any characters. Furthermore, XML Schema Pattern facets are expressed using this XML Schema fragment: <xs:pattern value="...some regex pattern here ..."/> But XML attributes are normalized by XML readers/parsers. Line endings in them are converted to single spaces. So <xs:pattern value="abc def"/> is equivalent to: <xs:pattern value="abc def"/> Furthermore <xs:pattern value="abc
def"/> is also normalized to <xs:pattern value="abc def"/> As far as I can tell there is no alternate notation to this. This means, if you want to use a pattern facet to specify that a DFDL infoset string can contain A-Za-z0-9 spaces and line endings, there is no way to express this. This pattern was the example I was dealing with. <xs:pattern value="[A-Za-z0-9 
]*"/> If you look at the string for the value attribute of this pattern element, that string already has the line ending characters converted into spaces. The attribute value is "[A-Za-z0-9 ]*" which has 3 spaces before the "]". I think there is no workaround for this in XML, XSD, or DFDL. I dug into the Daffodil implementation and in the code that accesses this attribute, you don't even get a NodeSeq containing a mixture of Text and Entity nodes. You just get a single Text node. So it is pretty well hopeless without reaching under the XML parser/reader's guts. Hence, in DFDL if you want to "validate" that a DFDL string contains content that includes line-endings with a regex, you have to use dfdl:assert with failureType="recoverableError" testKind="pattern" and testPattern with the regex of interest. This is then a DFDL regex, which is a Java regex, and you can be explicit about line endings allowed. You can't do it with a pattern facet. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 1

Contributing to github
by Bradd Kadlecik 04 Mar '21

04 Mar '21

Hi, I was unable to create a branch in github to upload the experience document for indirection. My userid there is tpfbraddk. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States

2 6

DFDL specification v1.0 - full recommendation status
by Mike Beckerle 24 Feb '21

24 Feb '21

Per thread below, the DFDL spec was reviewed by the OGF Standards Council, and with two minor changes (which are done) they are moving to publish this as a full recommendation. Congratulations to all involved. Final links will be forthcoming. I expect there to be some press announcement about this from somewhere. ---------- Forwarded message --------- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: Tue, Feb 23, 2021 at 10:44 AM Subject: Re: Final DFDL specification - Re: DFDL Public Comment Ended To: Greg Newby <gbnewby(a)petascale.org> Attached is PDF and docx of version 1.0.8. The only changes are the redraw of figure 2, and I sorted the items in 4.2.2. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Feb 23, 2021 at 9:33 AM Greg Newby <gbnewby(a)petascale.org> wrote: > Thanks, Mike. Appreciate it! > - Greg > > On Tue, Feb 23, 2021 at 7:02 AM Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > wrote: > >> I'll get these fixed and get you a new PDF and Docx today. >> >> Thank you very much. >> >> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | >> www.owlcyberdefense.com >> Please note: Contributions to the DFDL Workgroup's email discussions are >> subject to the OGF Intellectual Property Policy >> <http://www.ogf.org/About/abt_policies.php> >> >> >> >> On Mon, Feb 22, 2021 at 11:25 AM Greg Newby <gbnewby(a)petascale.org> >> wrote: >> >>> Hi, Mike. >>> >>> The standards council reviewed this, and was able to discuss during >>> today's meeting. >>> >>> There are only two very minor inquiries: >>> >>> Image 2 (p. 20) has some artifacts due to being a bitmap-type image. If >>> you still have the source for this, we wondered whether a higher resolution >>> version might be generated. In particular, there are some greyish >>> background artifacts behind 0..1, the shaded boxes, and perhaps some other >>> locations. It looks worse in the PDF, but this is also visible in the >>> .docx. If the source is no longer easily available, it's ok to leave this >>> as-is. >>> >>> 4.2.2 information items: It was noted that the items listed here are not >>> in alphabetical order. Is that intentional? Much of it is in alpha order, >>> but not the whole list. >>> >>> These are very minor comments. Overall, the document is excellent. Well >>> written and complete, and full due attention to the public comments. >>> >>> Let me know about the two items above. No further review is needed, and >>> once you confirm/provide any final document, I will get it published & >>> announced, and the obsolescence handled, in fairly short order. >>> >>> Best, >>> Greg >>> >>> >>> >>> On Wed, Feb 17, 2021 at 10:34 AM Greg Newby <gbnewby(a)petascale.org> >>> wrote: >>> >>>> Hi, Mike. In fact the first discussion happened just today. There will >>>> be a second discussion on Monday, February 22, because due to time >>>> constraints the discussion didn't conclude. >>>> >>>> There is a very good chance that final approval will occur on Monday. >>>> It will then take a few days for me to get the document + obsoleted >>>> document into their final PDF form and online at www.orf.org >>>> >>>> Best, >>>> Greg >>>> >>>> On Wed, Feb 17, 2021 at 10:32 AM Mike Beckerle < >>>> mbeckerle.dfdl(a)gmail.com> wrote: >>>> >>>>> Hi Greg, >>>>> >>>>> Any word on when the standards council will meet about this? >>>>> >>>>> I only nag because there are some other things happening related to >>>>> DFDL implementations where some press synergy is available depending on >>>>> time frames. >>>>> >>>>> Thanks for any info >>>>> >>>>> -mikeb >>>>> >>>>> >>>>> >>>>> >>>>> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | >>>>> www.owlcyberdefense.com >>>>> Please note: Contributions to the DFDL Workgroup's email discussions >>>>> are subject to the OGF Intellectual Property Policy >>>>> <http://www.ogf.org/About/abt_policies.php> >>>>> >>>>> >>>>> >>>>> On Mon, Feb 8, 2021 at 10:01 AM Greg Newby <gbnewby(a)petascale.org> >>>>> wrote: >>>>> >>>>>> This is perfect, Mike, and very much appreciated. I've pinged the >>>>>> standards council again to set up our next meeting, and otherwise they'll >>>>>> discuss by email. >>>>>> >>>>>> Hopefully we're just a couple of weeks away from putting this online >>>>>> as a new document, and obsoleting the old. I appreciate your stewardship >>>>>> for this document's journey! >>>>>> >>>>>> Best, >>>>>> Greg >>>>>> >>>>>> >>>>>> On Wed, Feb 3, 2021 at 3:08 PM Mike Beckerle < >>>>>> mbeckerle.dfdl(a)gmail.com> wrote: >>>>>> >>>>>>> Greg, >>>>>>> >>>>>>> Attached is a changes-tracked version containing the changes made >>>>>>> based on the public review comment period, and final corrections found >>>>>>> since. >>>>>>> That one is v1.0.6. >>>>>>> >>>>>>> The attached version 1.0.7 has all changes accepted and is the clean >>>>>>> version for final publication. docx and pdf attached. >>>>>>> >>>>>>> The public review comments included several statements of support, >>>>>>> but these public comments were substantive (links): >>>>>>> >>>>>>> https://github.com/OpenGridForum/Editor/issues/16 - a large and >>>>>>> extensive review >>>>>>> https://github.com/OpenGridForum/Editor/issues/15 >>>>>>> https://github.com/OpenGridForum/Editor/issues/14 >>>>>>> https://github.com/OpenGridForum/Editor/issues/9 >>>>>>> https://github.com/OpenGridForum/Editor/issues/8 >>>>>>> >>>>>>> The first of these was based on an extensive review of the >>>>>>> specification by Don Brutzman of the US Naval Research Laboratory. >>>>>>> There are change paragraphs in the attached having to do with the >>>>>>> other 4 public comments. But with respect to this one extensive review, >>>>>>> there are many smaller changes, as he pointed out a number of >>>>>>> English-language issues (use of 1st person and 2nd person, future vs. >>>>>>> present tense, wrong mood of verbs, etc.) as well as just wording >>>>>>> improvements, all of which were corrected. Those make up the bulk of the >>>>>>> smaller changes. >>>>>>> >>>>>>> >>>>>>> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | >>>>>>> www.owlcyberdefense.com >>>>>>> Please note: Contributions to the DFDL Workgroup's email discussions >>>>>>> are subject to the OGF Intellectual Property Policy >>>>>>> <http://www.ogf.org/About/abt_policies.php> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 27, 2021 at 3:04 PM Greg Newby <gbnewby(a)petascale.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, Mike. Thanks for this. I had a calendar reminder for when >>>>>>>> public comment ended, but it was right at new years and I missed or >>>>>>>> neglected it. >>>>>>>> >>>>>>>> Indeed, your actions are exactly right: make a final version of the >>>>>>>> document, based on any input received during the public comment period. >>>>>>>> >>>>>>>> The next step is a final review by the standards council. For this, >>>>>>>> it is helpful if you can provide either a "track changes" version of the >>>>>>>> document that was reviewed, or a written summary of the nature of changes. >>>>>>>> Whichever is easier - the idea is to make it easier for the standards >>>>>>>> council to know what's changed, based on public comments received. >>>>>>>> >>>>>>>> Once the standards council approves publication, I will proceed to >>>>>>>> put that online, and update #207 as obsolete (I have the editable version >>>>>>>> of that, I believe). >>>>>>>> >>>>>>>> The standards council's next meeting isn't scheduled yet. We are >>>>>>>> trying to identify a new fortnightly meeting time, and hopefully will be >>>>>>>> meeting by the 2nd week of February. If you can provide some sort of record >>>>>>>> of changes, as described above, the final review can proceed. >>>>>>>> >>>>>>>> Thanks again, and best regards, >>>>>>>> Greg >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jan 26, 2021 at 3:30 PM Mike Beckerle < >>>>>>>> mbeckerle.dfdl(a)gmail.com> wrote: >>>>>>>> >>>>>>>>> Greg et al., >>>>>>>>> >>>>>>>>> We have ended the public comment period on DFDL earlier this >>>>>>>>> month. >>>>>>>>> >>>>>>>>> We got some good support statements, and one person gave the spec >>>>>>>>> a quite thorough review even, which was great. >>>>>>>>> >>>>>>>>> I have edited in changes per the public comments that suggested >>>>>>>>> edits/improvements. >>>>>>>>> >>>>>>>>> I think we are ready to finalize DFDL v1.0 GFD.240 as an official >>>>>>>>> OGF Recommendation, and mark the prior GFD.207 as obsolete. >>>>>>>>> >>>>>>>>> What is next? Do I just send you the pdf and ms-word docx files? >>>>>>>>> We are just proofreading my final edits and this will be ready. >>>>>>>>> >>>>>>>>> Thank you >>>>>>>>> >>>>>>>>> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | >>>>>>>>> www.owlcyberdefense.com >>>>>>>>> Please note: Contributions to the DFDL Workgroup's email >>>>>>>>> discussions are subject to the OGF Intellectual Property Policy >>>>>>>>> <http://www.ogf.org/About/abt_policies.php> >>>>>>>>> >>>>>>>>>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 19 Feb '21

19 Feb '21

Please find minutes from the latest call at https://github.com/OpenGridForum/DFDL/wiki/DFDL-Workgroup-News Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 19 Feb '21

19 Feb '21

1 0