- dfdl-wg - lists.ogf.org

Reminder: DFDL Workgroup call Thursday June 12, 11am EDT.US
by Mike Beckerle 09 Jun '25

09 Jun '25

This is a reminder that we're having a DFDL Workgroup call this Thursday June 11, 11am EDT.US. Please send items you'd like on the agenda, and let me know if you plan to attend. I have this topic for discussion: * Peek feature - related Issue 66 - https://github.com/OpenGridForum/DFDL/issues/66 (about a lookAhead function) An idea has been proposed to generalize this to a more flexible peek/look-ahead capability. This is a requested feature for DFDL 2.0 allowing a parser to learn from examining the data once, and then use that knowledge in a subsequent parse over the same data. Details: - annotation on a xs:sequence - a parser to speculate forward and when it reaches the end of the sequence, resetting the position to the start of the sequence, but retaining information from the parse. - the retained information certainly includes DFDL variables set during the peek-ahead parse. - more controversial is the ability to retain the infoset, effectively allowing one to parse the same data multiple times. - does nothing on unparse. The annotated sequence is simply skipped by the unparser. - seems very easy to implement. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Idea: Peek layer
by Mike Beckerle 02 Jun '25

02 Jun '25

I've been dealing with lots of data formats lately which involve lookahead needs, and I'm trying to come up with a cleaner, easier, and less ad-hoc way to deal with them in DFDL to propose for the future. I wanted to bounce this idea off the workgroup before prototyping. Idea: Property dfdlx:peek="yes/no". Property on a sequence model group. Compatible with a hidden group ref. Unlike other properties, which are disallowed on a sequence with dfdl:hiddenGroupRef, this could be allowed on hidden sequences. That's actually a primary use case for this. If "no", no behavior change (what things do now). If "yes", the parse happens, including set variable assignments, infoset creation, etc. Then at the end of the sequence if the parse is successful the position in the data stream is reset to where it was at the start of the peek sequence. If the parse fails you backtrack to the enclosing PoU as normal and everything about the peek (when inside the PoU) is discarded. This allows you to learn by parsing something in the data more than once. Once to discover something which goes into the parser infoset (hidden or not), and into single-assignments to DFDL variables. The second time can parse making use of this learning. This is sort of like backtracking at a PoU, but you don't undo anything except the position in the data stream. On unparsing, all data written while unparsing the infoset for a sequence with dfdl:peek="yes" is discarded. Or maybe we can just say the infoset corresponding to a sequence with dfdl:peek="yes" is not unparsed at all. Implementations could put a limit on how far ahead you can peek. But a minimum of say, 512 bytes or maybe a bit bigger makes sense I think. That would be enough for every use case I have. I believe current restrictions in DFDL to ensure forward progress when parsing are sufficient to make it impossible to delay parsing forever with this. I.e., parsing can take a long time, but it still has to terminate (at least in theory, if there is enough memory for a big infoset). I think this dfdlx:peek has some nice properties. Pro: This is the most important thing: No specialized constructs for looking ahead. Just use DFDL to learn about the data, save it in variables or a piece of infoset that you can navigate with expressions to utilize the knowledge. Pro: Composition properties are good. Nothing new to learn. I can think of no impact on backtracking or any other aspects. Pro: Pretty cheap to implement, so long as the amount you can peek ahead is reasonably bounded. Pro: Synergistic with existing things like newVariableInstance and hidden groups to capture learning from a peek ahead. Con: Huge generality. Peeking ahead with a sequence with really rich sub-structure, PoUs and backtracking inside it, etc. That's all enabled by this feature, but none of the use cases I have need anything like that level of generality. This is one of those things where the stuff people will invent with it are unanticipated. All thoughts / musings are welcome. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 1

Canceled event: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu May 8, 2025 11am - 12pm (EDT) (dfdl-wg@ogf.org)
by Mike Beckerle 05 May '25

05 May '25

This event has been canceled. DFDL Workgroup - DFDL Spec Issues and Actions Thursday May 8, 2025 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

Reminder for Agenda Topics for Workgroup call on 2024-06-13
by Mike Beckerle 05 May '25

05 May '25

Please send along any agenda topics you wish discussed on our next call which is next week. (If you are already aware you will be not attending, send that along as well) I do have one agenda topic to cover which is ISO document creation. I'll circulate a separate email introducing the topic.

1 4

Call for Agenda: Call this week: Thurs April 10, 11am EDT.US
by Mike Beckerle 09 Apr '25

09 Apr '25

If there are specific agenda topics we should have the call, so let me know by Wed afternoon if you have a topic for discussion. Steve Hanson has already notified that he cannot attend this week's call. Despite my best efforts, I have not been able to make progress on action items for this workgroup this month, so I have nothing to report. (I am always so determined to make progress right after our calls, and then ... reality hits, and I get bogged down in other work.) Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 1

Canceled event with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Apr 10, 2025 11am - 12pm (EDT) (dfdl-wg@ogf.org)
by Mike Beckerle 09 Apr '25

09 Apr '25

This event has been canceled with a note: "April meeting cancelled. " DFDL Workgroup - DFDL Spec Issues and Actions Thursday Apr 10, 2025 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

Reminder - DFDL Workgroup Call this week: Thursday March 13
by Mike Beckerle 11 Mar '25

11 Mar '25

This is a call for specific agenda/discussion topics. Note: US.EST has just changed to US.EDT, so we are now UTC-04:00 The call will be at 11am US.EDT

1 0

Fwd: from a daffodil DFDL discussion on AI/LLMs
by Mike Beckerle 21 Feb '25

21 Feb '25

Thought this might be of interest to the workgroup, or we might spawn a thread from this in the dfdl workgroup. ---------- Forwarded message --------- From: Mike Beckerle <mbeckerle(a)apache.org> Date: Fri, Feb 21, 2025 at 1:22 PM Subject: AI/LLMs - was: .... To: <users(a)daffodil.apache.org> Consistent with my experience. There's nowhere near enough DFDL materials on github or stack overflow (yet) for a LLM to really create DFDL from a spec, and I've had little luck uploading format specs and asking it questions about them. Just asking GPT4o about DFDL is quite interesting though. Here's a chat I did: https://chatgpt.com/share/67b8bf26-1814-800f-95d6-d08ac63e14b8 I actually tried to use GPT4o to help with generating a property index into the PDF so that one could quickly jump to properties. It can't even list the properties in DFDL, but asking this was very interesting though. DFDL has a property named "finalTerminatorCanBeMissing". If you read that once, grokked the concept, but couldn't remember the exact name, well "terminatorCanBeRequired" is not a bad guess, and that's what GPT4o's guess was, which shows that it does have this stuff organized somehow as concepts, not a bunch of snippets to regurgitate. When I followed up with: " I haven't heard of the dfdl:terminatorCanBeRequired before. How does that work?" it corrected its mistake, and did get the name dfdl:finalTerminatorCanBeMissing this time. So the challenge is, if you know DFDL well, you can spot its mistakes, but if you are a naive user trying to learn DFDL, it gets quite a bit right, but the mistakes it does make are perhaps problematic. I believe I can prompt GPT4.0 to create a DFDL schema for me. Since I know DFDL, this is not faster than typing for me. But if you don't know DFDL, you can ask it to do lots of things that would save time like 'move that local simple type definition to a separate file and name it "intType2"'. And it is going to do that sort of thing just fine, particularly because that is mostly just using knowledge of XSD. This is all pretty cool. On Sat, Feb 15, 2025 at 7:37 AM Claude Mamo <claude.mamo(a)gmail.com> wrote: > I don't know if it's possible to capture the DICOM format in DFDL > > > Hah, a research team generated DICOM DFDL schemas from LLMs: > > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > The schema from gemini particularly stands out ;) > > ... > >>

1 0

testing please ignore
by Mike Beckerle 13 Feb '25

13 Feb '25

testing if just dfdl-wg(a)ogf.org now works, or if the 'lists' part is needed as in dfdl-wg(a)lists.ogf.org.

1 0

Reminder: DFDL Workgroup Call this week: Thursday Jan 13. 11:00 - 12:00 EST.US (UTC-5)
by Mike Beckerle 13 Feb '25

13 Feb '25

Draft agenda here: https://github.com/OpenGridForum/DFDL/blob/master/calls/2025/2025-02-13_DFD… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 3