- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] proof text for ISO/IEC 23415
by Mike Beckerle 25 Feb '24

25 Feb '24

Lisa, The footer is cut off on every page of this PDF, making it impossible to see the page numbers. I have not so far found any problem with the content of the pages, but this problem with the page footers does require a re-spin of the PDF, and if the page sizes change the table-of-contents will need its page numbers updated. -mike Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Feb 20, 2024 at 4:32 AM Lisa Rajchel <lrajchel(a)ansi.org> wrote: > Good morning, > > > > Attached please find the proof text for ISO/IEC 23415. Please confirm, by > March 18, 2024, that the content is ready for publication. > > > > Best > > Lisa >

2 1

Fwd: proof text for ISO/IEC 23415
by Mike Beckerle 21 Feb '24

21 Feb '24

We have an action to review this proof, no later than March 18th. I uploaded it to our gitlab ISO-Process folder. https://github.com/OpenGridForum/DFDL/blob/master/ISO-process/Data%20Format… Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> ---------- Forwarded message --------- From: Lisa Rajchel <lrajchel(a)ansi.org> Date: Tue, Feb 20, 2024 at 4:32 AM Subject: proof text for ISO/IEC 23415 To: Sill, Alan <Alan.Sill(a)ttu.edu> Cc: Wolfgang Ziegler <wolfgang.ziegler(a)z-rands.de>, Norbert Bensalem < norbert_bensalem(a)fr.ibm.com>, Jens Jensen <jens.jensen(a)stfc.ac.uk>, Greg Newby <gbnewby(a)petascale.org>, Steve Hanson <smhdfdl(a)gmail.com>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Good morning, Attached please find the proof text for ISO/IEC 23415. Please confirm, by March 18, 2024, that the content is ready for publication. Best Lisa

1 0

dfdl:extraEscapedCharacters - should be able to be expression
by Mike Beckerle 17 Feb '24

17 Feb '24

A user has noticed that for DFDL escape schemes based on escape characters, the properties dfdl:escapeCharacter and dfdl:escapeEscapeCharacter can both be runtime-valued expressions, but not dfdl:extraEscapedCharacters. This is not consistent. A major motivation for having these properties be runtime expressions is EDI formats, and the example motivating the need for dfdl:extraEscapedCharacters to be an expression also comes from EDIFACT. See this Daffodil ticket with a link to the email thread discussing this: https://issues.apache.org/jira/browse/DAFFODIL-2876 Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Agenda for Workgroup Call on 2024-02-07
by Mike Beckerle 08 Feb '24

08 Feb '24

We do have some topics for this month's call. Agenda: see https://github.com/OpenGridForum/DFDL/blob/master/calls/2024/2024-02-07_DFD… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Action 338: pathological DFDL behavior vs transformations in DFDL
by Mike Beckerle 07 Feb '24

07 Feb '24

Bradd, There's this example schema that illustrates DFDL being used to transform a pair of lists (which are hidden in a hidden group) into a list of pairs (which is the visible infoset). https://github.com/OpenDFDL/examples/tree/master/pairsTransform I'm curious if this illustrates the nature of the transformations you describe that zTPF users are doing or if there is more to it. The output is not the same "shape" as the parsed input data, but it is linearly related in size. -mike Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Example schema that illustrates pathological DFDL behaviors
by Mike Beckerle 07 Feb '24

07 Feb '24

I have been concerned for a while that combinations of the features in DFDL v1.0 can be used in ways that are just far too powerful for what is needed for data format expression. For example, if the input is n bits, how large should the corresponding infoset be when the data is parsed using a DFDL schema? Clearly at least O(n) since there are n bits and each bit could create an element in the infoset. But should it be able to be O(n^2) ? I can think of no use case for this. Intuition about data formats suggests that the infoset should always be linearly related in size to the size of the input data, and that the constant factor on that should be relatively small. Furthermore, the time taken to parse data should be linearly related to the size of the data. I think lots of people are thinking DFDL parsing should be implementable in a circuit, and the memory requirements are proportional to the data size. It doesn't make sense to try to make hardware implementations of DFDL if that's not going to be the case. Turns out, using DFDL v1.0 using dfdl:occursCountKind='expression' and dfdl:inputValueCalc, can create an infoset from parsing that is super-exponentially larger than the input data. Yes, it is finite, and bounded to some function of the size of the input data. But we don't even have notations for this kind of numeric growth. The schema I created consumes a 3 bit integer as the data. The size of the infoset grows like this: - input 000 -> infoset size 1 - input 001 -> infoset size 1 - input 010 -> infoset size 256 - input 011 -> infoset size is over 10,000 elements - ... - input 111 -> infoset size is larger than the number of atoms in the universe. There are TDML tests for this schema that illustrate this (except that last one) But it is always finite! No recursion is required to do this. With infinite space and time available it WILL terminate. I believe the language remains theoretically decidable. The README in this schema project discusses this and other pathological cases, and suggests things we can do to restrict DFDL so that this kind of pathological behavior is not allowed. Really we want the largest infoset that can be created from - n bits of data - a schema of size s to be O(sn) in size. I have started an argument in the README about why O(sn) is the right target. Basically there are practical schemas that can meet O(sn) as a lower bound. I believe this can be achieved by adding more forward-progress requirements to DFDL - all arrays should have to make forward progress when parsing, not just unbounded arrays, and reused groups/types should have to make forward progress at points of reuse. By doing that, we ensure that to make a giant infoset, you have to be consuming a giant number of bits. Right now this stuff is just living in a personal github repo, but I'll give it an official better home either in DFDLSchemas or as part of Apache Daffodil, or something later. If this is of interest, please take a look at this github pull request below. ---------- Forwarded message --------- From: Mike Beckerle <notifications(a)github.com> Date: Wed, Jan 24, 2024 at 3:50 PM Subject: [mbeckerle/superexp] Example schema that illustrates pathological DFDL behaviors (PR #1) To: mbeckerle/superexp <superexp(a)noreply.github.com> Cc: Mike Beckerle <mbeckerle(a)apache.org>, Your activity < your_activity(a)noreply.github.com> Illustrates DFDL issues where parsing small data can lead to explosively larger parse output. Super exponentially larger. The expressive power of DFDL should be restricted so that this is not possible. See the README.md for discussion of the issue. ------------------------------ You can view, comment on, or merge this pull request online at: https://github.com/mbeckerle/superexp/pull/1 Commit Summary - 46dbae6 <https://github.com/mbeckerle/superexp/pull/1/commits/46dbae6b2dc5acd76d6954…> First version of this example schema File Changes (9 files <https://github.com/mbeckerle/superexp/pull/1/files>) - *A* .gitattributes <https://github.com/mbeckerle/superexp/pull/1/files#diff-618cd5b83d62060ba3d…> (5) - *M* .gitignore <https://github.com/mbeckerle/superexp/pull/1/files#diff-bc37d034bad56458379…> (7) - *M* README.md <https://github.com/mbeckerle/superexp/pull/1/files#diff-b335630551682c19a78…> (96) - *A* build.sbt <https://github.com/mbeckerle/superexp/pull/1/files#diff-5634c415cd8c8504fdb…> (30) - *A* project/build.properties <https://github.com/mbeckerle/superexp/pull/1/files#diff-cbc85b9df7818a480f2…> (1) - *A* src/exp.dfdl.xsd <https://github.com/mbeckerle/superexp/pull/1/files#diff-9751c75e50c64f92bee…> (84) - *A* src/superexp.dfdl.xsd <https://github.com/mbeckerle/superexp/pull/1/files#diff-2c057f270cab45f3f05…> (72) - *A* test/TestSuperexp.scala <https://github.com/mbeckerle/superexp/pull/1/files#diff-c6b466d6536f0f670f9…> (35) - *A* test/TestSuperexp.tdml <https://github.com/mbeckerle/superexp/pull/1/files#diff-f1b4e40b448ec3ab530…> (117) Patch Links: - https://github.com/mbeckerle/superexp/pull/1.patch - https://github.com/mbeckerle/superexp/pull/1.diff — Reply to this email directly, view it on GitHub <https://github.com/mbeckerle/superexp/pull/1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALUDAZF7ZM5XWD72HZ4Z5DYQ…> . You are receiving this because you are subscribed to this thread.Message ID: <mbeckerle/superexp/pull/1(a)github.com>

1 1

Action 336: ISO announcement and branding
by Mike Beckerle 07 Feb '24

07 Feb '24

Please add to the list of questions for short term. If this looks ok then we'll forward the questions to the ISO JTC1 secretary to get info. - When can we update OGF web sites, WIkipedia, blogs/news, X/Twitter, and other web content to announce and reflect our ISO status? - Are there branding guidelines for how one does this in conformance with ISO practices/policy? E.g., must use specific phrasing, must cite specific trademarks or such? Are there official logos we are supposed to use? (Or rules about when we can use those or are not to use them, etc.) - Can/should we update the cover of the OGF DFDL spec (GFD.240) on the OGF web site to reflect the ISO PAS status? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Updated invitation with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Wed Feb 7, 2024 11am - 12pm (EST) (dfdl-wg@ogf.org)
by Mike Beckerle 25 Jan '24

25 Jan '24

This event has been updated with a note: "I need to change this one call to a different day. I've chosen to try moving it to Wednesday Feb 7. " Changed: time DFDL Workgroup - DFDL Spec Issues and Actions Wednesday Feb 7, 2024 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer frostmar(a)uk.ibm.com Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional View all guest info https://calendar.google.com/calendar/event?action=VIEW&eid=amJnNXNxYzdwZmlo… Reply for dfdl-wg(a)ogf.org and view more details https://calendar.google.com/calendar/event?action=VIEW&eid=amJnNXNxYzdwZmlo… Your attendance is optional. ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. To stop receiving future updates for this event, decline this event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

DFDL == ISO/IEC DIS 23415
by Mike Beckerle 22 Jan '24

22 Jan '24

Links: https://www.iso.org/standard/87444.html says we're in under development status https://www.iso.org/obp/ui/en/#iso:std:iso-iec:23415:dis:ed-1:v1:en Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

ISO JTC1 ballot approved !
by Mike Beckerle 22 Jan '24

22 Jan '24

Looks like ISO JTC1 has approved our PAS application. I am not sure what next steps are. Presumably we'll have to create a version of the spec in ISO preferred formatting style, and our branding may need to change some. ---------- Forwarded message --------- From: Wolfgang Ziegler <wolfgang.ziegler(a)z-rands.de> Date: Mon, Jan 22, 2024 at 3:50 AM Subject: Re: Any word from ISO JTC1? To: <mbeckerle(a)apache.org> Cc: GFSG Group <gfsg(a)ogf.org> Dear Mike, we finally received the notification from the JTC1 secretary. "Sorry for the late notification. The DIS ballot was approved 100%. I can now ask ISO to publish. I will let you know if I need anything from you." Congratulations! We'll get in contact with you, if ISO needs something additional for the publishing process. Best regards Wolfgang Am 09.01.24 um 22:50 schrieb Wolfgang Ziegler: > Dear Mike, > > Happy new year! > > We had a brief discussion in our GFSG meeting yesterday also and > I’ve sent a mail to Lisa and their colleagues asking for updates. > > I’ve got no response yet but I’ll inform you once I have more information. > > Best regards > > Wolfgang > > -- > Dr. Wolfgang Ziegler > z-rands > > Im tiefen Bruch 23, D-53773 Hennef, Germany > > Tel: +49 2242 80371 <tel:+49%202242%2080371>; Mobil: +49 151 50445303 > <tel:+49%20151%2050445303> > > StandICT - Supporting European Experts Presence in International Standardisation > Activities in ICT > www.standict.eu <http://www.standict.eu/>, www.linkedin.com/in/standict > <http://www.linkedin.com/in/standict>, > twitter: @Stand_ICT > >> Am 09.01.2024 um 22:13 schrieb Mike Beckerle <mbeckerle(a)apache.org>: >> >> >> Happy new year! >> >> Any news from the ISO/JTC1 folks? Any status we can get? >> >> Thanks >> >> Mike Beckerle >> Apache Daffodil PMC |daffodil.apache.org <http://daffodil.apache.org/> >> OGFDFDLWorkgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl >> <http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl> >> Owl Cyber Defense | www.owlcyberdefense.com < http://www.owlcyberdefense.com/> >> >> -- Dr. Wolfgang Ziegler z-rands www.z-rands.de Im tiefen Bruch 23, D-53773 Hennef, Germany Tel: +49 2242 80371; Mobil: +49 151 50445303 StandICT - Supporting European Experts Presence in International Standardisation Activities in ICT www.standict.eu, www.linkedin.com/in/standict, twitter: @Stand_ICT

1 0