February 2024 - dfdl-wg - lists.ogf.org

Action 337: ISO DFDL Workgroup Operation changes
by Mike Beckerle 20 May '24

20 May '24

This action item is about how we operate our workgroup going forward, and how that evolves now that we're an ISO/IEC standard as well as an OGF one. I believe we will need to create a workgroup document describing these new practices, particularly when they augment/vary some of the standard OGF practices. Please add to this list of issues about how we operate going forward. - Typography: I assume we have to revise the OGF Spec to match Standard ISO format (Beyond just A4 page size) so future versions come out in that conforming style. - Once we do this I don't want to be maintaining an OGF flavor and an ISO flavor of the formatting. - Does a preliminary working document of the DFDL workgroup have to be specifically marked as such (so that it is clearly preliminary)? What are these markings? - If that is done without revisions of content from current OGF content, would that replace the current ISO-provided document (which is just a new cover page on our existing OGF GFD.240 document) or does that wait for a content revision? - Who are contacts within JTC1 ? Does the OGF Workgroup contact JTC1 by way of OGF leadership or directly? (or... what is the chain of command here? Don't want to be jumping across protocol boundaries!) - Is there a TC within JTC1 that we should work more closely with? Right away? Eventually? Need contacts there. - In 6 years a PAS is supposed to become an International Standard. What does that mean? Does PAS status end? Does OGF participation end, or does the workgroup merge into a TC? Or is this just a status change assuming things are going well? - Day to day working group activity/meetings - is there guidance about notes/minutes-taking, etc. (templates)? - Such time as we have revisions to the document, how do we engage with ISO to get an update to the PAS document? Are these voted? - What about just typographical corrections? - Thoughts: Do we just use the OGF process for updates/corrections and then just submit the changes to JTC1 (some TC?) for consideration? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 1

Re: [DFDL-WG] proof text for ISO/IEC 23415
by Mike Beckerle 25 Feb '24

25 Feb '24

Lisa, The footer is cut off on every page of this PDF, making it impossible to see the page numbers. I have not so far found any problem with the content of the pages, but this problem with the page footers does require a re-spin of the PDF, and if the page sizes change the table-of-contents will need its page numbers updated. -mike Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Feb 20, 2024 at 4:32 AM Lisa Rajchel <lrajchel(a)ansi.org> wrote: > Good morning, > > > > Attached please find the proof text for ISO/IEC 23415. Please confirm, by > March 18, 2024, that the content is ready for publication. > > > > Best > > Lisa >

2 1

Fwd: proof text for ISO/IEC 23415
by Mike Beckerle 21 Feb '24

21 Feb '24

We have an action to review this proof, no later than March 18th. I uploaded it to our gitlab ISO-Process folder. https://github.com/OpenGridForum/DFDL/blob/master/ISO-process/Data%20Format… Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> ---------- Forwarded message --------- From: Lisa Rajchel <lrajchel(a)ansi.org> Date: Tue, Feb 20, 2024 at 4:32 AM Subject: proof text for ISO/IEC 23415 To: Sill, Alan <Alan.Sill(a)ttu.edu> Cc: Wolfgang Ziegler <wolfgang.ziegler(a)z-rands.de>, Norbert Bensalem < norbert_bensalem(a)fr.ibm.com>, Jens Jensen <jens.jensen(a)stfc.ac.uk>, Greg Newby <gbnewby(a)petascale.org>, Steve Hanson <smhdfdl(a)gmail.com>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Good morning, Attached please find the proof text for ISO/IEC 23415. Please confirm, by March 18, 2024, that the content is ready for publication. Best Lisa

1 0

dfdl:extraEscapedCharacters - should be able to be expression
by Mike Beckerle 17 Feb '24

17 Feb '24

A user has noticed that for DFDL escape schemes based on escape characters, the properties dfdl:escapeCharacter and dfdl:escapeEscapeCharacter can both be runtime-valued expressions, but not dfdl:extraEscapedCharacters. This is not consistent. A major motivation for having these properties be runtime expressions is EDI formats, and the example motivating the need for dfdl:extraEscapedCharacters to be an expression also comes from EDIFACT. See this Daffodil ticket with a link to the email thread discussing this: https://issues.apache.org/jira/browse/DAFFODIL-2876 Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Agenda for Workgroup Call on 2024-02-07
by Mike Beckerle 08 Feb '24

08 Feb '24

We do have some topics for this month's call. Agenda: see https://github.com/OpenGridForum/DFDL/blob/master/calls/2024/2024-02-07_DFD… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Action 338: pathological DFDL behavior vs transformations in DFDL
by Mike Beckerle 07 Feb '24

07 Feb '24

Bradd, There's this example schema that illustrates DFDL being used to transform a pair of lists (which are hidden in a hidden group) into a list of pairs (which is the visible infoset). https://github.com/OpenDFDL/examples/tree/master/pairsTransform I'm curious if this illustrates the nature of the transformations you describe that zTPF users are doing or if there is more to it. The output is not the same "shape" as the parsed input data, but it is linearly related in size. -mike Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Example schema that illustrates pathological DFDL behaviors
by Mike Beckerle 07 Feb '24

07 Feb '24

I have been concerned for a while that combinations of the features in DFDL v1.0 can be used in ways that are just far too powerful for what is needed for data format expression. For example, if the input is n bits, how large should the corresponding infoset be when the data is parsed using a DFDL schema? Clearly at least O(n) since there are n bits and each bit could create an element in the infoset. But should it be able to be O(n^2) ? I can think of no use case for this. Intuition about data formats suggests that the infoset should always be linearly related in size to the size of the input data, and that the constant factor on that should be relatively small. Furthermore, the time taken to parse data should be linearly related to the size of the data. I think lots of people are thinking DFDL parsing should be implementable in a circuit, and the memory requirements are proportional to the data size. It doesn't make sense to try to make hardware implementations of DFDL if that's not going to be the case. Turns out, using DFDL v1.0 using dfdl:occursCountKind='expression' and dfdl:inputValueCalc, can create an infoset from parsing that is super-exponentially larger than the input data. Yes, it is finite, and bounded to some function of the size of the input data. But we don't even have notations for this kind of numeric growth. The schema I created consumes a 3 bit integer as the data. The size of the infoset grows like this: - input 000 -> infoset size 1 - input 001 -> infoset size 1 - input 010 -> infoset size 256 - input 011 -> infoset size is over 10,000 elements - ... - input 111 -> infoset size is larger than the number of atoms in the universe. There are TDML tests for this schema that illustrate this (except that last one) But it is always finite! No recursion is required to do this. With infinite space and time available it WILL terminate. I believe the language remains theoretically decidable. The README in this schema project discusses this and other pathological cases, and suggests things we can do to restrict DFDL so that this kind of pathological behavior is not allowed. Really we want the largest infoset that can be created from - n bits of data - a schema of size s to be O(sn) in size. I have started an argument in the README about why O(sn) is the right target. Basically there are practical schemas that can meet O(sn) as a lower bound. I believe this can be achieved by adding more forward-progress requirements to DFDL - all arrays should have to make forward progress when parsing, not just unbounded arrays, and reused groups/types should have to make forward progress at points of reuse. By doing that, we ensure that to make a giant infoset, you have to be consuming a giant number of bits. Right now this stuff is just living in a personal github repo, but I'll give it an official better home either in DFDLSchemas or as part of Apache Daffodil, or something later. If this is of interest, please take a look at this github pull request below. ---------- Forwarded message --------- From: Mike Beckerle <notifications(a)github.com> Date: Wed, Jan 24, 2024 at 3:50 PM Subject: [mbeckerle/superexp] Example schema that illustrates pathological DFDL behaviors (PR #1) To: mbeckerle/superexp <superexp(a)noreply.github.com> Cc: Mike Beckerle <mbeckerle(a)apache.org>, Your activity < your_activity(a)noreply.github.com> Illustrates DFDL issues where parsing small data can lead to explosively larger parse output. Super exponentially larger. The expressive power of DFDL should be restricted so that this is not possible. See the README.md for discussion of the issue. ------------------------------ You can view, comment on, or merge this pull request online at: https://github.com/mbeckerle/superexp/pull/1 Commit Summary - 46dbae6 <https://github.com/mbeckerle/superexp/pull/1/commits/46dbae6b2dc5acd76d6954…> First version of this example schema File Changes (9 files <https://github.com/mbeckerle/superexp/pull/1/files>) - *A* .gitattributes <https://github.com/mbeckerle/superexp/pull/1/files#diff-618cd5b83d62060ba3d…> (5) - *M* .gitignore <https://github.com/mbeckerle/superexp/pull/1/files#diff-bc37d034bad56458379…> (7) - *M* README.md <https://github.com/mbeckerle/superexp/pull/1/files#diff-b335630551682c19a78…> (96) - *A* build.sbt <https://github.com/mbeckerle/superexp/pull/1/files#diff-5634c415cd8c8504fdb…> (30) - *A* project/build.properties <https://github.com/mbeckerle/superexp/pull/1/files#diff-cbc85b9df7818a480f2…> (1) - *A* src/exp.dfdl.xsd <https://github.com/mbeckerle/superexp/pull/1/files#diff-9751c75e50c64f92bee…> (84) - *A* src/superexp.dfdl.xsd <https://github.com/mbeckerle/superexp/pull/1/files#diff-2c057f270cab45f3f05…> (72) - *A* test/TestSuperexp.scala <https://github.com/mbeckerle/superexp/pull/1/files#diff-c6b466d6536f0f670f9…> (35) - *A* test/TestSuperexp.tdml <https://github.com/mbeckerle/superexp/pull/1/files#diff-f1b4e40b448ec3ab530…> (117) Patch Links: - https://github.com/mbeckerle/superexp/pull/1.patch - https://github.com/mbeckerle/superexp/pull/1.diff — Reply to this email directly, view it on GitHub <https://github.com/mbeckerle/superexp/pull/1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALUDAZF7ZM5XWD72HZ4Z5DYQ…> . You are receiving this because you are subscribed to this thread.Message ID: <mbeckerle/superexp/pull/1(a)github.com>

1 1

Action 336: ISO announcement and branding
by Mike Beckerle 07 Feb '24

07 Feb '24

Please add to the list of questions for short term. If this looks ok then we'll forward the questions to the ISO JTC1 secretary to get info. - When can we update OGF web sites, WIkipedia, blogs/news, X/Twitter, and other web content to announce and reflect our ISO status? - Are there branding guidelines for how one does this in conformance with ISO practices/policy? E.g., must use specific phrasing, must cite specific trademarks or such? Are there official logos we are supposed to use? (Or rules about when we can use those or are not to use them, etc.) - Can/should we update the cover of the OGF DFDL spec (GFD.240) on the OGF web site to reflect the ISO PAS status? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0