- dfdl-wg - lists.ogf.org

clarification - argument expression to fn:count() function
by Mike Beckerle 11 Jan '24

11 Jan '24

The description of fn:count in section 18.5.2.5 says fn:count() can be called on a node-sequence. It does not seem to require an array/optional path. This seems to be suggesting that if you have say.... <element name="b" type="xs:int"/> <element name="a" type="xs:int"/> <element name="b" type="xs:int" minOccurs="0"/> That one could write fn:count(b) and it should return 1 or 2. The Daffodil project calls this a "query-style" expression, as opposed to just a basic path expression. Furthermore, in the first paragraph of section 18.5.2.5 it says "(Note that DFDL v1.0 does not support sequences of length > 1 as the final results of expressions.)" which suggests that some expressions can constructively return node sequences of length > 1 as intermediate results. Presumably fn:count(b) above is such a situation, as the expression 'b' returns 2 nodes. The above all suggests to me that fn:count(a) would be legal where 'a' is a scalar. This is just always 1 of course. But Section 35 says: Expression value is not single node § Most DFDL expression contexts require an expression to identify a single node, not an array (aka sequence of nodes). There are a few exceptions such as the fn:count(…) function, where the path expression must be to an array or optional element. o Expression value is not array element or optional element. § Some DFDL expression contexts require an array or an optional element. § Example: The fn:count(...) function argument must be to an array or optional element. It is a Schema Definition Error if the argument expression is otherwise. Experience at the Daffodil project is that allowing fn:count argument expression to be a non-array non-optional element, where fn:count would always return 1, just hides errors that are very hard to find, and this situation comes up often as a schema is written. Usually the expression to fn:count is initially correct with an array/optional as the argument, but element nesting evolves, and the paths need updating, but end up referring not to the array/optional element, but that name is now of a scalar enclosing element of the array, so the fn:count is always 1, and the schema is incorrect because the expression is not doing what is intended, but no error is detected. This is then quite hard to isolate and fix. A concrete example of this experience is you start with a schema like: <element name="record" maxOccurs="unbounded"> <complexType> <sequence> .... elements of the record But then you need the valueLength of the whole array of all the records, to store the length for unparsing, so you revise this to: <element name="record"> <complexType> <sequence> <element name="item" maxOccurs="unbounded"/> <complexType> <sequence> .... elements of each record 'item'. And now, paths you had like fn:count(foo/bar/record) are no longer to an array, they are to a scalar, so always return 1. This is decidedly unhelpful in a large schema. It is far better if fn:count(foo/bar/record) becomes an SDE because record is now scalar. So the clarification I'm seeking is whether section 35 was just missed when updates were made about this node-sequence stuff, or if it is reasonable to implement the restrictions in Section 35. I am biased. I want the restrictions in Section 35, but this was muddy enough that I thought we should get a clarification first. Daffodil already doesn't implement any query-style expressions so the fn:count(b) example above would be an SDE in Daffodil. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 1

Future feature? allow pattern facet on numbers when textNumberRep="standard" and representation="text"
by Mike Beckerle 20 Dec '23

20 Dec '23

It has come up often now that DFDL cannot be strict enough about text number formats because our ICU-based textNumberPattern isn't strict enough or expressive enough of subtle syntax variations. I suggest this could be fixed by just allowing the XSD pattern facet to be used on numeric types when they are known textual and standard (not zoned). For example dfdl:textNumberPattern="00.####" will allow the number "99." to be accepted. There's currently no way to say "when it's an integer, there cannot be a decimal point". People are resistant to the notion that this requires a complex type with a bunch of different elements with different textNumberFormats so that you have an '<int>99</int>' or <dec>99.9</dec> element. They really don't want there to be different paths to this value in the infoset just because of this format issue about the decimal point. It's a painful loss of polymorphism in these path expressions. Instead of a simple path expression to obtain such a value you end up with if (fn:exists(path/int)) then path/int else path/dec Note that DFDL's expression language has no let statement, so in the above if "path" is actually "a/b/c/d/e/f/g" i.e., a typical deep path (which commonly have much longer path steps than my single-letters), then that path is going to be repeated 3 times in the expression. This is pretty unpleasant. Rather than come up with a bunch of ICU mods to tighten up all the places it is lax, and to add features for suppressed decimal points, etc. we could just allow the pattern facet on textual numbers. Then the pattern facet could be "\d\d|\d\d\.\d{1,4}" which would achieve the goal of enforcing the precise pattern desired if you validate after parsing and before unparsing. It would not prevent conversion of the text to the corresponding numeric type, but it would allow an additional tighter check on what the text was. Regular XML Schema allows the pattern facet on all the numeric types, so we would be eliminating what is currently a DFDL restriction, on condition of only when the numeric types have standard text representation. Thoughts? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

3 5

Canceled event with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Dec 14, 2023 11am - 12pm (EST) (dfdl-wg@ogf.org)
by Mike Beckerle 14 Dec '23

14 Dec '23

This event has been canceled with a note: "No topics for agenda have been submitted and no email discussion has indicated a need to meet, so today's meeting is cancelled. " DFDL Workgroup - DFDL Spec Issues and Actions Thursday Dec 14, 2023 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer frostmar(a)uk.ibm.com Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. To stop receiving future updates for this event, decline this event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

Reminder - Call for agenda topics for next DFDL WG call on 2023-12-14 T 16:00:00 Z
by Mike Beckerle 07 Dec '23

07 Dec '23

The next call is scheduled for Thursday Dec 14, at 11am US.ET (UTC-5) Please let me know, or discuss on this email list, any agenda topics for discussion. Our hope is that we hear back from ISO JTC1 before this about our PAS application, but if not, and there are no agenda topics, then I will cancel the call . Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Canceled event with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Nov 9, 2023 11am - 12pm (EST) (dfdl-wg@ogf.org)
by Mike Beckerle 08 Nov '23

08 Nov '23

This event has been canceled with a note: "No discussion topics for the meeting have come up, so I'm cancelling the Nov 9th 2023 DFDL Workgroup Call. We're still all just awaiting word from ISO JTC1 I guess. " DFDL Workgroup - DFDL Spec Issues and Actions Thursday Nov 9, 2023 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer frostmar(a)uk.ibm.com Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. To stop receiving future updates for this event, decline this event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

[ANNOUNCE] Apache Daffodil 3.6.0 Released
by Mike Beckerle 06 Nov '23

06 Nov '23

The Apache Daffodil community is pleased to announce the release of version 3.6.0. This release includes a number of bug fixes. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/3.6.0/ Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team

1 0

ChatGPT writes DFDL Schemas (well almost)
by Mike Beckerle 31 Oct '23

31 Oct '23

Ok, there's some mistakes, but I was frankly astounded that ChatGPT (4) can even start to write DFDL schemas: Check out: https://chat.openai.com/share/2f5e33ed-89a9-4aa9-a9cf-f6532c57f1b7 I'd say that's a good start. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 1

Question about unparsing with dfdl:textNumberPattern "#.##" vs. "0.##"
by Mike Beckerle 26 Oct '23

26 Oct '23

These textNumberPatterns behave the same as far as I can tell. I'd expect them to be different when unparsing. When unparsing decimal number value 0.12 I'd expect.... textNumberPattern="0.##" to create a 4 character representation "0.12" because one digit left of the decimal point is required. But I expect... textNumberPattern="#.##" to create a 3-character representation ".12" because the digit to the left of the decimal point is optional and the whole integer part is zero and so should be suppressed. Isn't that the point of "#" versus "0" as a digit indicator in the pattern? That's not what happens though. I get "0.12" from either textNumberPattern pattern. Is this a bug? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 1

Cancelled 2023-10-12 dfdl workgroup call
by Mike Beckerle 11 Oct '23

11 Oct '23

No topics were raised in advance.

1 0

Canceled event: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Oct 12, 2023 11am - 12pm (EDT) (dfdl-wg@ogf.org)
by mbeckerle.dfdl＠gmail.com 11 Oct '23

11 Oct '23

This event has been canceled. DFDL Workgroup - DFDL Spec Issues and Actions Thursday Oct 12, 2023 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer mbeckerle.dfdl(a)gmail.com mbeckerle.dfdl(a)gmail.com Guests mbeckerle.dfdl(a)gmail.com - organizer Mark Frost mbeckerle(a)gmail.com - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. To stop receiving future updates for this event, decline this event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0