- dfdl-wg - lists.ogf.org

Fwd: from a daffodil DFDL discussion on AI/LLMs
by Mike Beckerle 21 Feb '25

21 Feb '25

Thought this might be of interest to the workgroup, or we might spawn a thread from this in the dfdl workgroup. ---------- Forwarded message --------- From: Mike Beckerle <mbeckerle(a)apache.org> Date: Fri, Feb 21, 2025 at 1:22 PM Subject: AI/LLMs - was: .... To: <users(a)daffodil.apache.org> Consistent with my experience. There's nowhere near enough DFDL materials on github or stack overflow (yet) for a LLM to really create DFDL from a spec, and I've had little luck uploading format specs and asking it questions about them. Just asking GPT4o about DFDL is quite interesting though. Here's a chat I did: https://chatgpt.com/share/67b8bf26-1814-800f-95d6-d08ac63e14b8 I actually tried to use GPT4o to help with generating a property index into the PDF so that one could quickly jump to properties. It can't even list the properties in DFDL, but asking this was very interesting though. DFDL has a property named "finalTerminatorCanBeMissing". If you read that once, grokked the concept, but couldn't remember the exact name, well "terminatorCanBeRequired" is not a bad guess, and that's what GPT4o's guess was, which shows that it does have this stuff organized somehow as concepts, not a bunch of snippets to regurgitate. When I followed up with: " I haven't heard of the dfdl:terminatorCanBeRequired before. How does that work?" it corrected its mistake, and did get the name dfdl:finalTerminatorCanBeMissing this time. So the challenge is, if you know DFDL well, you can spot its mistakes, but if you are a naive user trying to learn DFDL, it gets quite a bit right, but the mistakes it does make are perhaps problematic. I believe I can prompt GPT4.0 to create a DFDL schema for me. Since I know DFDL, this is not faster than typing for me. But if you don't know DFDL, you can ask it to do lots of things that would save time like 'move that local simple type definition to a separate file and name it "intType2"'. And it is going to do that sort of thing just fine, particularly because that is mostly just using knowledge of XSD. This is all pretty cool. On Sat, Feb 15, 2025 at 7:37 AM Claude Mamo <claude.mamo(a)gmail.com> wrote: > I don't know if it's possible to capture the DICOM format in DFDL > > > Hah, a research team generated DICOM DFDL schemas from LLMs: > > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > https://github.com/narfindustries/llm-tests-langsec/blob/main/results/1.0/D… > > The schema from gemini particularly stands out ;) > > ... > >>

1 0

testing please ignore
by Mike Beckerle 13 Feb '25

13 Feb '25

testing if just dfdl-wg(a)ogf.org now works, or if the 'lists' part is needed as in dfdl-wg(a)lists.ogf.org.

1 0

Reminder: DFDL Workgroup Call this week: Thursday Jan 13. 11:00 - 12:00 EST.US (UTC-5)
by Mike Beckerle 13 Feb '25

13 Feb '25

Draft agenda here: https://github.com/OpenGridForum/DFDL/blob/master/calls/2025/2025-02-13_DFD… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 3

Minutes from 2025-01-09 DFDL Workgroup Call
by Mike Beckerle 09 Jan '25

09 Jan '25

I've written these up here: https://github.com/OpenGridForum/DFDL/blob/master/calls/2025/2025-01-09_DFD… Comments and anything I missed welcome. You can edit the document, or reply to this email and I'll incorporate.

2 2

Clarification needed: multiple assert statements evaluation order
by Mike Beckerle 09 Jan '25

09 Jan '25

Given this DFDL annotation on an xs:sequence <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:assert test="..." /> <dfdl:discriminator test="..." /> <dfdl:assert test="..." /> </xs:appinfo> The spec seems silent about the evaluation order among these 3 statement annotations. Let's call them A, B, C. First, the spec makes it clear that the discriminator B could be evaluated earlier than either assertion, and even before some of the sequence content. This is to allow optimization by a DFDL implementation. The spec is also clear that even if the parse of the sequence content fails, discriminator B is evaluated (with the infoset being the state at the time of the failure). Again this is to ensure the behavior matches that of an optimizing DFDL implementation. But let's assume an implementation does no such optimization. I believe these evaluation orders are legal for the 3 statements after the sequence content has been parsed: A, B, C - if A fails, we do know B must still be evaluated. B, A, C - this is the minimum sort of hoist/optimization, doing the discriminator before the asserts. Questions: 1) Are any other orders of evaluation allowed? 2) If evaluation of A fails, do we still evaluate assertion C? (My hope is the answer here is no, because that allows consecutive asserts to build on each other's assumptions. But the spec is unclear.) 3) Can users depend on the failure of A to generate a message output? (ex: if the assert has a message attribute, can we state that this message will somehow be exhibited or logged by the implementation, unless the failure is suppressed by backtracking at a point of uncertainty) If A fails, the spec does say that B must still be evaluated. But if A fails, will C be evaluated? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 3

Canceled event with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Dec 12, 2024 11am - 12pm (EST) (dfdl-wg@ogf.org)
by Mike Beckerle 09 Dec '24

09 Dec '24

This event has been canceled with a note: "Neither Steve Hanson nor Mike Beckerle are available for this meeting. Next call January 9th. " DFDL Workgroup - DFDL Spec Issues and Actions Thursday Dec 12, 2024 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

Clarification needed on dfdl:decimalSigned
by Mike Beckerle 18 Nov '24

18 Nov '24

This property appears in exactly 3 places in the DFDL spec. * Section 13.5 where it is defined * Appendix G: 36.1 Parser - property precedence tables * Appendix G: 36.2 Unparser - property precedence tables. The definition in Section 13.5 says Indicates whether an xs:decimal element is signed. See 13.6.2 Converting logical numbers to/from text representation and 13.7.1 Converting Logical Numbers to/from Binary to see how this affects the presence of the sign in the data stream. 'yes' means that the xs:decimal element is signed 'no' means that the xs:decimal element is not signed Neither Section 13.6.2 nor 13.7.1 mention the decimalSigned property. We need to add language which clarifies this. I believe this is roughly what is needed: In Section 13.6.2, add this material at the start of the section: *The signed number types are byte, short, int, long, integer, float, double* *The unsigned number types have names beginning with "unsigned", as well as nonNegativeInteger.* *For type xs:decimal the decimal is considered signed if dfdl:decimalSigned="yes" and unsigned if dfdl:decimalSigned="no". * In Section 13.7.1 we need to add this: *When dfdl:binaryNumberRep="binary", then dfdl:decimalSigned controls whether the underlying representation is interpreted as a twos-complement signed binary integer or an unsigned binary integer. This value is then scaled using dfdl:binaryDecimalVirtualPoint to convert to/from the infoset value. * *When dfdl:binaryNumberRep is 'bcd', no sign is possible and the dfdl:decimalSigned property is ignored. * *When dfdl:binaryNumberRep is 'packed' a sign nibble is required and the dfdl:decimalSigned property is ignored.* *(rationale: packed but unsigned is the same as BCD)* *When dfdl:binaryNumberRep is "ibm4690Packed" the dfdl:decimalSigned property is used to determine if negative values are allowed, i.e., the packed digits may be preceded by a negative sign nibble.* *If dfdl:decimalSigned is "no" then no sign nibble is expected or allowed. * Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 5

clarification needed: prefixLengthType has min/max facets prefixIncludesPrefixLength="yes"
by Mike Beckerle 14 Nov '24

14 Nov '24

(Note: email sent to dfdl-wg at lists.ogf.org, not just ogf.org per prior practice.) Sometime back, in order to allow facets to sanity check prefix length values, we specified that if facets are expressed on a prefixLengthType they would always be checked and if they do not hold it is a processing error. This correction was part of the most recent GFD.240 However, we did not specify if the facet check is done on the prefix length value before or after it is adjusted when prefixIncludesPrefixLength="yes". So for example, suppose there is a single-byte prefix on a string and dfdl:prefixIncludesPrefixLength is "yes". Then the minimum well-formed value of the prefix is "1" which indicates a length 0 string with a single byte prefix. Suppose I want to add XSD facets to sanity check this prefix value: <simpleType name="prefix1Byte"> <restriction type="xs:unsignedByte"> <maxInclusive value="99"/> </restriction> </simpleType> So, if this type is used as the dfdl:prefixLengthType of a string with dfdl:lengthKind='prefixed', and dfdl:prefixIncludesPrefixLength="yes" then does the check for 99 measure the length of the string, or the value of the prefix, which is the length of the string plus one? My thought is that there is a quasi-element corresponding to this single byte prefix. That quasi-element's value is what is constrained by this maxInclusive facet. So I'm claiming the facet value 99 should be compared against the stored value in the prefix quasi-element, that is before any value adjustment is done for dfdl:prefixIncludesPrefixLength. Which is to say, suppose you switched to dfdl:lengthKind="explicit", and made an actual element of the single unsiged byte "prefix" field. Then the adjusting for the length of that prefix length would be done in the dfdl:length expression for the string, and the dfdl:outputValueCalc for the single byte prefix field. Which is to say that facets constraining the value of that one-byte element would apply to the value directly, not after any adjustment. I think lengthKind prefixed should work exactly the same as when a real length element and outputValueCalc are used. One should not have to adjust facets on the type of the length field when when it is the type of the length element, no should one have to adjust the facets if you convert and use lengthKind 'prefixed' with the dfdl:prefixLengthType, the facets should mean the same thing. Any thoughts/comments? Note: daffodil PR about this: https://github.com/apache/daffodil/pull/1334 JIRA Issue: https://issues.apache.org/jira/browse/DAFFODIL-2660

2 2

Agenda for call on 2024-11-14
by Mike Beckerle 14 Nov '24

14 Nov '24

See: https://github.com/OpenGridForum/DFDL/blob/master/calls/2024/2024-11-14_DFD… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

DFDL-WG: Call for Agenda Topics for Meeting/Call on Nov 14
by Mike Beckerle 07 Nov '24

07 Nov '24

Call for agenda items/topics for November Workgroup Call. I have two minor clarification issues that have come up in email: * decimalSigned * prefixLengthType Please send other agenda items you would like to discuss. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0