April 2025 - dfdl-wg - lists.ogf.org

Idea: Peek layer
by Mike Beckerle 02 Jun '25

02 Jun '25

I've been dealing with lots of data formats lately which involve lookahead needs, and I'm trying to come up with a cleaner, easier, and less ad-hoc way to deal with them in DFDL to propose for the future. I wanted to bounce this idea off the workgroup before prototyping. Idea: Property dfdlx:peek="yes/no". Property on a sequence model group. Compatible with a hidden group ref. Unlike other properties, which are disallowed on a sequence with dfdl:hiddenGroupRef, this could be allowed on hidden sequences. That's actually a primary use case for this. If "no", no behavior change (what things do now). If "yes", the parse happens, including set variable assignments, infoset creation, etc. Then at the end of the sequence if the parse is successful the position in the data stream is reset to where it was at the start of the peek sequence. If the parse fails you backtrack to the enclosing PoU as normal and everything about the peek (when inside the PoU) is discarded. This allows you to learn by parsing something in the data more than once. Once to discover something which goes into the parser infoset (hidden or not), and into single-assignments to DFDL variables. The second time can parse making use of this learning. This is sort of like backtracking at a PoU, but you don't undo anything except the position in the data stream. On unparsing, all data written while unparsing the infoset for a sequence with dfdl:peek="yes" is discarded. Or maybe we can just say the infoset corresponding to a sequence with dfdl:peek="yes" is not unparsed at all. Implementations could put a limit on how far ahead you can peek. But a minimum of say, 512 bytes or maybe a bit bigger makes sense I think. That would be enough for every use case I have. I believe current restrictions in DFDL to ensure forward progress when parsing are sufficient to make it impossible to delay parsing forever with this. I.e., parsing can take a long time, but it still has to terminate (at least in theory, if there is enough memory for a big infoset). I think this dfdlx:peek has some nice properties. Pro: This is the most important thing: No specialized constructs for looking ahead. Just use DFDL to learn about the data, save it in variables or a piece of infoset that you can navigate with expressions to utilize the knowledge. Pro: Composition properties are good. Nothing new to learn. I can think of no impact on backtracking or any other aspects. Pro: Pretty cheap to implement, so long as the amount you can peek ahead is reasonably bounded. Pro: Synergistic with existing things like newVariableInstance and hidden groups to capture learning from a peek ahead. Con: Huge generality. Peeking ahead with a sequence with really rich sub-structure, PoUs and backtracking inside it, etc. That's all enabled by this feature, but none of the use cases I have need anything like that level of generality. This is one of those things where the stuff people will invent with it are unanticipated. All thoughts / musings are welcome. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 1

Reminder for Agenda Topics for Workgroup call on 2024-06-13
by Mike Beckerle 05 May '25

05 May '25

Please send along any agenda topics you wish discussed on our next call which is next week. (If you are already aware you will be not attending, send that along as well) I do have one agenda topic to cover which is ISO document creation. I'll circulate a separate email introducing the topic.

1 4

Call for Agenda: Call this week: Thurs April 10, 11am EDT.US
by Mike Beckerle 09 Apr '25

09 Apr '25

If there are specific agenda topics we should have the call, so let me know by Wed afternoon if you have a topic for discussion. Steve Hanson has already notified that he cannot attend this week's call. Despite my best efforts, I have not been able to make progress on action items for this workgroup this month, so I have nothing to report. (I am always so determined to make progress right after our calls, and then ... reality hits, and I get bogged down in other work.) Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 1

Canceled event with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Apr 10, 2025 11am - 12pm (EDT) (dfdl-wg@ogf.org)
by Mike Beckerle 09 Apr '25

09 Apr '25

This event has been canceled with a note: "April meeting cancelled. " DFDL Workgroup - DFDL Spec Issues and Actions Thursday Apr 10, 2025 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0