- dfdl-wg - lists.ogf.org

Confirming expected behavior of separatorSuppressionPolicy 'never'
by Mike Beckerle 21 Apr '21

21 Apr '21

An earlier email I sent had this schema: <xs:element name="file1"> <xs:complexType> <xs:sequence dfdl:separator="," dfdl:separatorPosition="infix" dfdl:separatorSuppressionPolicy="never"> <xs:element name="given-name" type="xs:string" minOccurs="0" maxOccurs="3" /> <xs:element name="surname" type="xs:string" minOccurs="0"/> <xs:element name="phone" type="xs:string" minOccurs="0" maxOccurs="6" /> </xs:sequence> </xs:complexType> </xs:element> Given that, and a default dfd:format with occursCountKind='implicit' I expect this data: "madonna,,,,,,,,," To produce this DFDL Infoset (cast as XML): <file1> <given-name>madonna</given-name> </file1> I don't expect to see any empty elements like <given-name></given-name> because all elements are optional, so a zero-length representation for a xs:string is the Empty representation, and optional empty is never added to the infoset. Furthermore, this should "round trip" i.e., unparse back to the original input data. Am I correct here? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 1

example/use-case wanted - separatorSuppressionPolicy 'trailingEmptyStrict' and 'never'
by Mike Beckerle 20 Apr '21

20 Apr '21

Users of Apache Daffodil have asked what these property values are for, i.e., what are idiomatic uses of this property for actual formats, or in other words, what kind of data are these properties needed for? I'd like to provide them examples of this if anyone can recall the kinds of specific use cases for these property values. I've been searching through the example DFDL schemas I can find, and none use trailingEmptyStrict or never for this property. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 3

Latest OGF DFDL WG Call Minutes
by Steve Hanson 15 Apr '21

15 Apr '21

Please find minutes from the latest call at https://github.com/OpenGridForum/DFDL/wiki/DFDL-Workgroup-News Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 14 Apr '21

14 Apr '21

Please find agenda for next call on GitHub at https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-04-15_DFDL-WG-… Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

newVariableInstance direction property discussion - PCAP example
by Mike Beckerle 13 Apr '21

13 Apr '21

For our next conference call, I said I would walk through the Apache Daffodil newVariableInstance feature, and how for unparsing we had to add a direction property to variables so they can be evaluated with forward referencing for unparsing. The code I'd like to discuss is here: https://github.com/DFDLSchemas/PCAP/blob/cb9ee66aa565d982f01130918bcd785def… The defineVariable that is the best motivating example is defined at line 104, and is used at line 213. This is a simple common-subexpression elimination, but it is used to avoid redundant calls to dfdl:valueLength in the dfdl:outputValueCalc of the stored length field named InclLen. There is a second example of using newVariableInstance at unparse time in this schema also, with the variable definitions starting at line 115, and usage in the group defined starting at line 320. This usage is far more complex, and in hindsight I think a DFDL parse of PCAP data should not be creating IP addresses in the Infoset that look like "1.2.3.4", because it requires all this complexity to unparse from that back to the binary representation. However, it is a good exercise in how the newVariableInstance feature works. -mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 3

OGF DFDL Working Group Call
by Steve Hanson 13 Apr '21

13 Apr '21

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 23 Mar '21

23 Mar '21

Please find minutes from the latest call at https://github.com/OpenGridForum/DFDL/wiki/DFDL-Workgroup-News Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 17 Mar '21

17 Mar '21

Please find agenda for next call on GitHub at https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-03-18_DFDL-WG-… Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

OGF DFDL Working Group Call
by Steve Hanson 17 Mar '21

17 Mar '21

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL v1.0 Specification is Final "Recommendation" - Fwd: [wg-all] New document published for DFDL-WG, #240
by Mike Beckerle 10 Mar '21

10 Mar '21

In case this announcement slipped by anyone...... This is the formal announcement that DFDL v1.0 Spec. is now a full "Recommendation" from the OGF. That's the term OGF uses for full standards, i.e., that are no longer in "proposed" status. The announcement to *wg-all(a)ogf.org* <wg-all(a)ogf.org> doesn't say it's a full recommendation. It just says "new document", but if you follow the link (*https://www.ogf.org/ogf/doku.php/documents/documents* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_ogf_doku.p…>) to the list of OGF documents, you see that GFD.240 is the top row, and has the 3 important little letters "REC" next to this document. Those indicate the status as a "RECommendation". So v1.0 is official, done. Congratulations and many thanks to all who have been contributing so much to making this happen for so long. It's kind of hard to imagine that I have been working (off and on) on DFDL since 2001, so 20 years. Given that I turned 60 this year, that's 1/3 of my life and 1/2 of my career. My full time job became DFDL only in the 2011 time frame, but that's 10 years of full time (mostly) on DFDL or DFDL-related work. Steve Hanson is the other co-chair of the DFDL Workgroup at OGF, working (off and on) on DFDL since 2004 with a full-time stretch from 2011 to 2015. The contributors to this workgroup, along with the co-chairs assembled together the collective knowledge from many data integration systems and tools, and collective knowledge of a huge amount of data experience to synthesize the DFDL standard. I believe to have real impact on the computing industry today, you have to have an open standard, long-term commercial commitment, commercial adoption/product, and an open-source implementation as well, along with enough time and attention to actually finish the work. OGF provided a patient forum for the open-standard to come to fruition. Being a small organization OGF was actually perfect for DFDL given the long time it took to complete the work. A larger standards body might have forced things to happen too fast, or the whole process might have been derailed by too many participants. IBM provided the commercial support and the first production-ready implementation called IBM DFDL, used in several of its key products, and the US DoD funded several core contributors to work on what has become the Open Source Apache Daffodil implementation. All these implementations were hard to build, and took real time and effort from many engineers. They cover very different data formats that improved the standard by making sure it was far more comprehensive than prior-generation technologies. The official PDF of the standard (er "Recommendation") is *https://www.ogf.org/documents/GFD.240.pdf* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_documents_…> Mike Beckerle OGF DFDL Workgroup Co-Chair Apache Daffodil PMC Chair Owl Cyber Defense - Principal Engineer Please note: Contributions to the DFDL Workgroup's email discussions are subject to the *OGF Intellectual Property Policy* <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ogf.org_About_abt-5…> ---------- Forwarded message --------- From: *Greg Newby* <*gbnewby(a)petascale.org* <gbnewby(a)petascale.org>> Date: Mon, Mar 1, 2021 at 9:48 AM Subject: [wg-all] New document published for DFDL-WG, #240 To: All WG <*wg-all(a)ogf.org* <wg-all(a)ogf.org>> Dear OGF community, A new document has been published by OGF. All OGF documents may be found here: *https://www.ogf.org/ogf/doku.php/documents/documents* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_ogf_doku.p…> This document was provided by the Data Format Description Language Working Group (DFDL-WG). *GFD-R-P.240: Data Format Description Language (DFDL) v.1.0 Specification* Authors: Michael J. Beckerle, Owl Cyber Defense/Tresys and Stephen M. Hanson, IBM Abstract: This document provides a definition of a standard Data Format Description Language (DFDL). This language allows description of text, dense binary, and legacy data formats in a vendor-neutral declarative manner. DFDL is an extension to the XML Schema Description Language (XSD). This document obsoletes GFD #207 and GFD #174. - Greg Newby OGF Standards Editor -- The wg-all mailing list is an umbrella list for all OGF mailing lists. If you are subscribed to any OGF mailing list, you will receive mails to wg-all. wg-all mailing list *wg-all(a)ogf.org* <wg-all(a)ogf.org> *https://www.ogf.org/mailman/listinfo/wg-all* <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li…> Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0