February 2025 - dfdl-wg - lists.ogf.org

Reminder for Agenda Topics for Workgroup call on 2024-06-13
by Mike Beckerle 05 May '25

05 May '25

Please send along any agenda topics you wish discussed on our next call which is next week. (If you are already aware you will be not attending, send that along as well) I do have one agenda topic to cover which is ISO document creation. I'll circulate a separate email introducing the topic.

1 4

Idea: Peek layer
by Mike Beckerle 21 Feb '25

21 Feb '25

I've been dealing with lots of data formats lately which involve lookahead needs, and I'm trying to come up with a cleaner, easier, and less ad-hoc way to deal with them in DFDL to propose for the future. I wanted to bounce this idea off the workgroup before prototyping. Idea: Property dfdlx:peek="yes/no". Property on a sequence model group. Compatible with a hidden group ref. Unlike other properties, which are disallowed on a sequence with dfdl:hiddenGroupRef, this could be allowed on hidden sequences. That's actually a primary use case for this. If "no", no behavior change (what things do now). If "yes", the parse happens, including set variable assignments, infoset creation, etc. Then at the end of the sequence if the parse is successful the position in the data stream is reset to where it was at the start of the peek sequence. If the parse fails you backtrack to the enclosing PoU as normal and everything about the peek (when inside the PoU) is discarded. This allows you to learn by parsing something in the data more than once. Once to discover something which goes into the parser infoset (hidden or not), and into single-assignments to DFDL variables. The second time can parse making use of this learning. This is sort of like backtracking at a PoU, but you don't undo anything except the position in the data stream. On unparsing, all data written while unparsing the infoset for a sequence with dfdl:peek="yes" is discarded. Or maybe we can just say the infoset corresponding to a sequence with dfdl:peek="yes" is not unparsed at all. Implementations could put a limit on how far ahead you can peek. But a minimum of say, 512 bytes or maybe a bit bigger makes sense I think. That would be enough for every use case I have. I believe current restrictions in DFDL to ensure forward progress when parsing are sufficient to make it impossible to delay parsing forever with this. I.e., parsing can take a long time, but it still has to terminate (at least in theory, if there is enough memory for a big infoset). I think this dfdlx:peek has some nice properties. Pro: This is the most important thing: No specialized constructs for looking ahead. Just use DFDL to learn about the data, save it in variables or a piece of infoset that you can navigate with expressions to utilize the knowledge. Pro: Composition properties are good. Nothing new to learn. I can think of no impact on backtracking or any other aspects. Pro: Pretty cheap to implement, so long as the amount you can peek ahead is reasonably bounded. Pro: Synergistic with existing things like newVariableInstance and hidden groups to capture learning from a peek ahead. Con: Huge generality. Peeking ahead with a sequence with really rich sub-structure, PoUs and backtracking inside it, etc. That's all enabled by this feature, but none of the use cases I have need anything like that level of generality. This is one of those things where the stuff people will invent with it are unanticipated. All thoughts / musings are welcome. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Fwd: from a daffodil DFDL discussion on AI/LLMs
by Mike Beckerle 21 Feb '25

21 Feb '25

Thought this might be of interest to the workgroup, or we might spawn a thread from this in the dfdl workgroup. ---------- Forwarded message --------- From: Mike Beckerle <mbeckerle(a)apache.org> Date: Fri, Feb 21, 2025 at 1:22 PM Subject: AI/LLMs - was: .... To: <users(a)daffodil.apache.org> Consistent with my experience. There's nowhere near enough DFDL materials on github or stack overflow (yet) for a LLM to really create DFDL from a spec, and I've had little luck uploading format specs and asking it questions about them. Just asking GPT4o about DFDL is quite interesting though. Here's a chat I did: https://chatgpt.com/share/67b8bf26-1814-800f-95d6-d08ac63e14b8 I actually tried to use GPT4o to help with generating a property index into the PDF so that one could quickly jump to properties. It can't even list the properties in DFDL, but asking this was very interesting though. DFDL has a property named "finalTerminatorCanBeMissing". If you read that once, grokked the concept, but couldn't remember the exact name, well "terminatorCanBeRequired" is not a bad guess, and that's what GPT4o's guess was, which shows that it does have this stuff organized somehow as concepts, not a bunch of snippets to regurgitate. When I followed up with: " I haven't heard of the dfdl:terminatorCanBeRequired before. How does that work?" it corrected its mistake, and did get the name dfdl:finalTerminatorCanBeMissing this time. So the challenge is, if you know DFDL well, you can spot its mistakes, but if you are a naive user trying to learn DFDL, it gets quite a bit right, but the mistakes it does make are perhaps problematic. I believe I can prompt GPT4.0 to create a DFDL schema for me. Since I know DFDL, this is not faster than typing for me. But if you don't know DFDL, you can ask it to do lots of things that would save time like 'move that local simple type definition to a separate file and name it "intType2"'. And it is going to do that sort of thing just fine, particularly because that is mostly just using knowledge of XSD. This is all pretty cool. On Sat, Feb 15, 2025 at 7:37 AM Claude Mamo <claude.mamo(a)gmail.com> wrote: > I don't know if it's possible to capture the DICOM format in DFDL > > > Hah, a research team generated DICOM DFDL schemas from LLMs: > > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > The schema from gemini particularly stands out ;) > > ... > >>

1 0

testing please ignore
by Mike Beckerle 13 Feb '25

13 Feb '25

testing if just dfdl-wg(a)ogf.org now works, or if the 'lists' part is needed as in dfdl-wg(a)lists.ogf.org.

1 0

Reminder: DFDL Workgroup Call this week: Thursday Jan 13. 11:00 - 12:00 EST.US (UTC-5)
by Mike Beckerle 13 Feb '25

13 Feb '25

Draft agenda here: https://github.com/OpenGridForum/DFDL/blob/master/calls/2025/2025-02-13_DFD… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 3