- dfdl-wg - lists.ogf.org

ICU and dfdl:textNumberPattern issue
by Mike Beckerle 15 Jan '24

15 Jan '24

We've run into an issue with dfdl:textNumberPattern, which is an ICU number pattern. I'll discuss here, and then suggest this is a fix needed in DFDL generally, but we should discuss that hypothesis. The motivating example is fixed length 5 character integer text data. The data ranges from -9999 to 99999. Note that the minus sign uses up one of the 5 characters that can be a digit for positive values. Consider the value -123 and textNumberPattern of "00000;-0". The value unparses as -00123 which is length 6 so too long. The padding feature of ICU number patterns can be used to "fix" this. Consider textNumberPattern="*0####0". The "*0" notation means to use 0 as the padding character to replace the "#" when needed. Now the value 123 unparses as 00123 but ... here's the problem.... -123 unparses as 0-123. Notice how the zero padding is before the minus sign when we wanted it to appear after. This problem is caused by ICU taking nearly all the information from the positive part of the textNumberPattern. The negative part of the pattern, if it exists, is used only to define the affix (prefix or suffix or both) that indicate negative values. The problem is that positive numbers commonly have no affix, so the position of padding characters relative to the affix cannot always be determined from the positive pattern alone. Hence, if textNumberPattern specifies a pad character before the number pattern and without a positive prefix, then ICU defaults to a pad position of PAD_BEFORE_PREFIX with no way to change it with just the pattern. This behavior is reasonable for most cases, like when the pad character is a space. However, if the pad character in textNumberPattern is '0', then negative numbers are padded with a '0' before the negative sign. So we get the errant behavior where a pattern of "*0####0" unparses -123 to "0-123". This is very unlikely to be what the user wants with this pattern. Now suppose the positive pattern required a prefix "+" sign. The textNumberPattern of "+*0####0" works properly because ICU determines that the padding is PAD_AFTER_PREFIX from the positive pattern where the "*0" is after the "+" prefix. The proposed fix to this issue that we're implementing in Daffodil is this: If both negative and positive patterns define padding on the same affix, and the positive pattern has an empty string for that affix, then we use the pad position from the negative pattern. In all other cases, the pad character in the negative pattern is ignored following usual ICU behavior. For example, a textNumberPattern of "*0####0;-*00" formats a negative number with zero padding after the hyphen, whereas normal ICU behavior would ignore the negative pattern and zero pad before the hyphen. I would suggest this is something that is a needed fix in general for DFDL. The workaround when you have the fixed length number use case (-9999 to 99999) is very ugly, treating the minus sign as an initiator and creating separate elements for positive and negative values, or punting on this integer and treating the whole thing as a string. Arguably, this might be considered an ICU bug, but ICU's API can be used to specify the PAD_AFTER_PREFIX behavior, so it's not like ICU doesn't let you achieve the needed behavior, it's just not something that can be achieved using only the ICU pattern string. But ICU maintainers may or may not consider this to be a bug. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Confirming DFDL Workgroup call today (2024-01-11)
by Mike Beckerle 11 Jan '24

11 Jan '24

Hope to see you there. I will, hopefully, send out agenda in advance.

1 1

spec bug: Figure 4
by Mike Beckerle 11 Jan '24

11 Jan '24

Section 13.6.1.1, Figure 4: dfdl:textNumberPattern BNF Syntax has a line: vinteger := ('#'* '0'* 'V' '0'* '0') | (integer 'V') prefix := '\u0000'..'\uFFFD' - specialCharacters This is missing a line ending before "prefix", which is a separate production of this BNF grammar. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

clarification - argument expression to fn:count() function
by Mike Beckerle 11 Jan '24

11 Jan '24

The description of fn:count in section 18.5.2.5 says fn:count() can be called on a node-sequence. It does not seem to require an array/optional path. This seems to be suggesting that if you have say.... <element name="b" type="xs:int"/> <element name="a" type="xs:int"/> <element name="b" type="xs:int" minOccurs="0"/> That one could write fn:count(b) and it should return 1 or 2. The Daffodil project calls this a "query-style" expression, as opposed to just a basic path expression. Furthermore, in the first paragraph of section 18.5.2.5 it says "(Note that DFDL v1.0 does not support sequences of length > 1 as the final results of expressions.)" which suggests that some expressions can constructively return node sequences of length > 1 as intermediate results. Presumably fn:count(b) above is such a situation, as the expression 'b' returns 2 nodes. The above all suggests to me that fn:count(a) would be legal where 'a' is a scalar. This is just always 1 of course. But Section 35 says: Expression value is not single node § Most DFDL expression contexts require an expression to identify a single node, not an array (aka sequence of nodes). There are a few exceptions such as the fn:count(…) function, where the path expression must be to an array or optional element. o Expression value is not array element or optional element. § Some DFDL expression contexts require an array or an optional element. § Example: The fn:count(...) function argument must be to an array or optional element. It is a Schema Definition Error if the argument expression is otherwise. Experience at the Daffodil project is that allowing fn:count argument expression to be a non-array non-optional element, where fn:count would always return 1, just hides errors that are very hard to find, and this situation comes up often as a schema is written. Usually the expression to fn:count is initially correct with an array/optional as the argument, but element nesting evolves, and the paths need updating, but end up referring not to the array/optional element, but that name is now of a scalar enclosing element of the array, so the fn:count is always 1, and the schema is incorrect because the expression is not doing what is intended, but no error is detected. This is then quite hard to isolate and fix. A concrete example of this experience is you start with a schema like: <element name="record" maxOccurs="unbounded"> <complexType> <sequence> .... elements of the record But then you need the valueLength of the whole array of all the records, to store the length for unparsing, so you revise this to: <element name="record"> <complexType> <sequence> <element name="item" maxOccurs="unbounded"/> <complexType> <sequence> .... elements of each record 'item'. And now, paths you had like fn:count(foo/bar/record) are no longer to an array, they are to a scalar, so always return 1. This is decidedly unhelpful in a large schema. It is far better if fn:count(foo/bar/record) becomes an SDE because record is now scalar. So the clarification I'm seeking is whether section 35 was just missed when updates were made about this node-sequence stuff, or if it is reasonable to implement the restrictions in Section 35. I am biased. I want the restrictions in Section 35, but this was muddy enough that I thought we should get a clarification first. Daffodil already doesn't implement any query-style expressions so the fn:count(b) example above would be an SDE in Daffodil. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 1

Future feature? allow pattern facet on numbers when textNumberRep="standard" and representation="text"
by Mike Beckerle 20 Dec '23

20 Dec '23

It has come up often now that DFDL cannot be strict enough about text number formats because our ICU-based textNumberPattern isn't strict enough or expressive enough of subtle syntax variations. I suggest this could be fixed by just allowing the XSD pattern facet to be used on numeric types when they are known textual and standard (not zoned). For example dfdl:textNumberPattern="00.####" will allow the number "99." to be accepted. There's currently no way to say "when it's an integer, there cannot be a decimal point". People are resistant to the notion that this requires a complex type with a bunch of different elements with different textNumberFormats so that you have an '<int>99</int>' or <dec>99.9</dec> element. They really don't want there to be different paths to this value in the infoset just because of this format issue about the decimal point. It's a painful loss of polymorphism in these path expressions. Instead of a simple path expression to obtain such a value you end up with if (fn:exists(path/int)) then path/int else path/dec Note that DFDL's expression language has no let statement, so in the above if "path" is actually "a/b/c/d/e/f/g" i.e., a typical deep path (which commonly have much longer path steps than my single-letters), then that path is going to be repeated 3 times in the expression. This is pretty unpleasant. Rather than come up with a bunch of ICU mods to tighten up all the places it is lax, and to add features for suppressed decimal points, etc. we could just allow the pattern facet on textual numbers. Then the pattern facet could be "\d\d|\d\d\.\d{1,4}" which would achieve the goal of enforcing the precise pattern desired if you validate after parsing and before unparsing. It would not prevent conversion of the text to the corresponding numeric type, but it would allow an additional tighter check on what the text was. Regular XML Schema allows the pattern facet on all the numeric types, so we would be eliminating what is currently a DFDL restriction, on condition of only when the numeric types have standard text representation. Thoughts? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

3 5

Canceled event with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Dec 14, 2023 11am - 12pm (EST) (dfdl-wg@ogf.org)
by Mike Beckerle 14 Dec '23

14 Dec '23

This event has been canceled with a note: "No topics for agenda have been submitted and no email discussion has indicated a need to meet, so today's meeting is cancelled. " DFDL Workgroup - DFDL Spec Issues and Actions Thursday Dec 14, 2023 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer frostmar(a)uk.ibm.com Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. To stop receiving future updates for this event, decline this event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

Reminder - Call for agenda topics for next DFDL WG call on 2023-12-14 T 16:00:00 Z
by Mike Beckerle 07 Dec '23

07 Dec '23

The next call is scheduled for Thursday Dec 14, at 11am US.ET (UTC-5) Please let me know, or discuss on this email list, any agenda topics for discussion. Our hope is that we hear back from ISO JTC1 before this about our PAS application, but if not, and there are no agenda topics, then I will cancel the call . Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Canceled event with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Nov 9, 2023 11am - 12pm (EST) (dfdl-wg@ogf.org)
by Mike Beckerle 08 Nov '23

08 Nov '23

This event has been canceled with a note: "No discussion topics for the meeting have come up, so I'm cancelling the Nov 9th 2023 DFDL Workgroup Call. We're still all just awaiting word from ISO JTC1 I guess. " DFDL Workgroup - DFDL Spec Issues and Actions Thursday Nov 9, 2023 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer frostmar(a)uk.ibm.com Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. To stop receiving future updates for this event, decline this event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

[ANNOUNCE] Apache Daffodil 3.6.0 Released
by Mike Beckerle 06 Nov '23

06 Nov '23

The Apache Daffodil community is pleased to announce the release of version 3.6.0. This release includes a number of bug fixes. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/3.6.0/ Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team

1 0

ChatGPT writes DFDL Schemas (well almost)
by Mike Beckerle 31 Oct '23

31 Oct '23

Ok, there's some mistakes, but I was frankly astounded that ChatGPT (4) can even start to write DFDL schemas: Check out: https://chat.openai.com/share/2f5e33ed-89a9-4aa9-a9cf-f6532c57f1b7 I'd say that's a good start. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 1