March 2024 - dfdl-wg - lists.ogf.org

Action 337: ISO DFDL Workgroup Operation changes
by Mike Beckerle 20 May '24

20 May '24

This action item is about how we operate our workgroup going forward, and how that evolves now that we're an ISO/IEC standard as well as an OGF one. I believe we will need to create a workgroup document describing these new practices, particularly when they augment/vary some of the standard OGF practices. Please add to this list of issues about how we operate going forward. - Typography: I assume we have to revise the OGF Spec to match Standard ISO format (Beyond just A4 page size) so future versions come out in that conforming style. - Once we do this I don't want to be maintaining an OGF flavor and an ISO flavor of the formatting. - Does a preliminary working document of the DFDL workgroup have to be specifically marked as such (so that it is clearly preliminary)? What are these markings? - If that is done without revisions of content from current OGF content, would that replace the current ISO-provided document (which is just a new cover page on our existing OGF GFD.240 document) or does that wait for a content revision? - Who are contacts within JTC1 ? Does the OGF Workgroup contact JTC1 by way of OGF leadership or directly? (or... what is the chain of command here? Don't want to be jumping across protocol boundaries!) - Is there a TC within JTC1 that we should work more closely with? Right away? Eventually? Need contacts there. - In 6 years a PAS is supposed to become an International Standard. What does that mean? Does PAS status end? Does OGF participation end, or does the workgroup merge into a TC? Or is this just a status change assuming things are going well? - Day to day working group activity/meetings - is there guidance about notes/minutes-taking, etc. (templates)? - Such time as we have revisions to the document, how do we engage with ISO to get an update to the PAS document? Are these voted? - What about just typographical corrections? - Thoughts: Do we just use the OGF process for updates/corrections and then just submit the changes to JTC1 (some TC?) for consideration? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 1

Re: [DFDL-WG] [EXTERNAL] Fwd: proof text for ISO/IEC 23415
by Mike Beckerle 19 Apr '24

19 Apr '24

Lisa, We found some additional issues with the previously sent ISO DFDL PDF. Besides the problem with the footers being cut off (already reported) we found: - Links do not work: - Table-of-contents entries do not let you navigate to the sections. - Internal links like "[XML]" to the references section do not work. - Hyperlink text in the references sections do not work. - The page margins are inconsistent. Viewed with 2 pages on screen side by side, the left pages have narrow left margin in comparison to right pages. - Tables 46 and 47 in Sections 14.2.1 and 14.2.1 break across page boundaries. This makes them very hard to interpret, and while some large tables can split across pages normally, for these specific tables, the page boundary makes it very hard to understand them. Manual breaks should be inserted to avoid this. - Table 48 in Section 14.2.3 does not have a bad-break currently. But this table is also one that a page-break across it needs to be avoided. Please forward these to the right parties who are preparing the ISO/IEC version. Thank you Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Fri, Mar 8, 2024 at 9:42 AM Lisa Rajchel <lrajchel(a)ansi.org> wrote: > Mike, > > I have sent your comment to ISO and now it is their hands. No one has > asked for anything more from you. I will keep you posted. > > Best > Lisa > > Sent with BlackBerry Work > (www.blackberry.com) > > > *From: *Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > *Date: *Friday, Mar 08, 2024 at 9:23 AM > *To: *Lisa Rajchel <lrajchel(a)ansi.org> > *Subject: *[EXTERNAL] Fwd: [DFDL-WG] proof text for ISO/IEC 23415 > > Lisa, > > I am following up on this need for another spin on the DFDL 23415 official > ISO version. > > As far as I know this is out of our hands, meaning the workgroup on DFDL. > However, we have a workgroup meeting this week, I am wondering if there are > any actions we need or could take to help get the 23415 through the process > and fully published. > > Please advise > > Thank you > > > Mike Beckerle > OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl > Apache Daffodil PMC | daffodil.apache.org > Owl Cyber Defense | www.owlcyberdefense.com > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the OGF Intellectual Property Policy > <http://www.ogf.org/About/abt_policies.php> > > > > ---------- Forwarded message --------- > From: Steve Hanson <smhdfdl(a)gmail.com> > Date: Sun, Feb 25, 2024 at 5:10 AM > Subject: Re: [DFDL-WG] proof text for ISO/IEC 23415 > To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > Cc: Lisa Rajchel <lrajchel(a)ansi.org>, DFDL-WG <dfdl-wg(a)ogf.org> > > > It looks like the footer is being cut off in order to accommodate the > three lines of licence text in every header. > > Regards > Steve > > On Sat, Feb 24, 2024 at 10:39 PM Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > wrote: > >> Lisa, >> >> The footer is cut off on every page of this PDF, making it impossible to >> see the page numbers. >> >> I have not so far found any problem with the content of the pages, but >> this problem with the page footers does require a re-spin of the PDF, and >> if the page sizes change the table-of-contents will need its page numbers >> updated. >> >> -mike >> >> Mike Beckerle >> OGF DFDL Workgroup Co-Chair | >> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl >> Apache Daffodil PMC | daffodil.apache.org >> Owl Cyber Defense | www.owlcyberdefense.com >> Please note: Contributions to the DFDL Workgroup's email discussions are >> subject to the OGF Intellectual Property Policy >> <http://www.ogf.org/About/abt_policies.php> >> >> >> >> On Tue, Feb 20, 2024 at 4:32 AM Lisa Rajchel <lrajchel(a)ansi.org> wrote: >> >>> Good morning, >>> >>> >>> >>> Attached please find the proof text for ISO/IEC 23415. Please confirm, >>> by March 18, 2024, that the content is ready for publication. >>> >>> >>> >>> Best >>> >>> Lisa >>> >> -- >> dfdl-wg mailing list >> dfdl-wg(a)lists.ogf.org >> https://lists.ogf.org/mailman/listinfo/dfdl-wg >> > > > This e-mail is intended exclusively for the individual or entity to which > it is addressed or otherwise directed. This e-mail (and its attachments) > may contain confidential, proprietary or privileged information, and > unauthorized disclosure or use of such information is prohibited absent > express authorization of the sender. If you receive this e-mail in error, > please notify the sender and delete this e-mail from your system. >

1 3

Updated invitation with note: DFDL Workgroup - DFDL Spec Issues and Actions @ Thu Apr 18, 2024 11am - 12pm (EDT) (dfdl-wg@ogf.org)
by Mike Beckerle 29 Mar '24

29 Mar '24

This event has been updated with a note: "Rescheduled just this meeting. " Changed: time DFDL Workgroup - DFDL Spec Issues and Actions Thursday Apr 18, 2024 ⋅ 11am – 12pm Eastern Time - New York Location Google Meet https://www.google.com/maps/search/Google+Meet?hl=en Join with Google Meet https://meet.google.com/ifg-hknz-ysw?hs=224 Please send email to open discussions and add agenda for this meeting. If there are no specific topics I will cancel the meeting. Organizer Mike Beckerle mbeckerle.dfdl(a)gmail.com Guests Mike Beckerle - organizer frostmar(a)uk.ibm.com Mike Beckerle - optional smhdfdl(a)gmail.com - optional jens.jensen(a)stfc.ac.uk - optional alan.sill(a)ttu.edu - optional mbeckerle(a)owlcyberdefense.com - optional braddk(a)us.ibm.com - optional dfdl-wg(a)ogf.org - optional wiloneil(a)uk.ibm.com - optional wolfgang.ziegler(a)z-rands.de - optional View all guest info https://calendar.google.com/calendar/event?action=VIEW&eid=amJnNXNxYzdwZmlo… Reply for dfdl-wg(a)ogf.org and view more details https://calendar.google.com/calendar/event?action=VIEW&eid=amJnNXNxYzdwZmlo… Your attendance is optional. ~~//~~ Invitation from Google Calendar: https://calendar.google.com/calendar/ You are receiving this email because you are an attendee on the event. To stop receiving future updates for this event, decline this event. Forwarding this invitation could allow any recipient to send a response to the organizer, be added to the guest list, invite others regardless of their own invitation status, or modify your RSVP. Learn more https://support.google.com/calendar/answer/37135#forwarding

1 0

Reschedule Next Call (was April 11) - Mike Beckerle can't make it
by Mike Beckerle 29 Mar '24

29 Mar '24

I just realized I am on vacation that week. I would like to reschedule for April 18th. I will move the calendar invite, and we'll have the call or not depending on whether others accept the invite. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

xs:dateTime constructor - spec bug - erratum on the erratum
by Mike Beckerle 29 Mar '24

29 Mar '24

This will probably require an erratum. On 2023-06-13 we applied errata to the DFDL spec, and one of these errata fixes is actually incorrect. The correction says: Section 18.5.1 the xs:dateTime function takes 2 arguments, an xs:date and xs:time. This is not consistent with the XPath constructor function of that name, which takes a single argument: - xs:dateTime($arg as xs:anyAtomicType?) as xs:dateTime? DFDL spec also describes an fn:dateTime function - same signature - two arguments, in that same section (table 57). This is correct and consistent with XPath. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Re: [DFDL-WG] ICU and dfdl:textNumberPattern issue
by Mike Beckerle 14 Mar '24

14 Mar '24

Reporting back to the DFDL WG on how this issue got resolved on the Daffodil project. The issue came up in the NITF data format. The schema is on github DFDLSchemas site. Here is the long issue: https://github.com/DFDLSchemas/NITF/issues/19 read intro then skip to this comment: https://github.com/DFDLSchemas/NITF/issues/19#issuecomment-1889110915 Also related: https://issues.apache.org/jira/browse/DAFFODIL-2870 which is is a open issue to warn about any ICU negative pattern chars that are ignored. Also related: https://github.com/apache/daffodil/pull/1139 which is a related fix. On Thu, Mar 14, 2024 at 11:55 AM Steve Hanson <smhdfdl(a)gmail.com> wrote: > Mike, this is the thread about the negative pattern length issue. > > On Mon, Jan 15, 2024 at 11:10 AM Steve Hanson <smhdfdl(a)gmail.com> wrote: > >> I seem to recall hitting the same problem with X12 fixed length numerics >> which could be positive or negative. If I get time before our next call >> I'll re-familiarise myself with the X12 issue and what I did to work round >> it. >> >> Regards >> Steve >> >> On Thu, Jan 11, 2024 at 6:25 PM Mike Beckerle <mbeckerle(a)apache.org> >> wrote: >> >>> We've run into an issue with dfdl:textNumberPattern, which is an ICU >>> number pattern. I'll discuss here, and then suggest this is a fix needed in >>> DFDL generally, but we should discuss that hypothesis. >>> >>> The motivating example is fixed length 5 character integer text data. >>> The data ranges from -9999 to 99999. Note that the minus sign uses up one >>> of the 5 characters that can be a digit for positive values. >>> >>> Consider the value -123 and textNumberPattern of "00000;-0". The value >>> unparses as -00123 which is length 6 so too long. >>> >>> The padding feature of ICU number patterns can be used to "fix" this. >>> Consider textNumberPattern="*0####0". The "*0" notation means to use 0 as >>> the padding character to replace the "#" when needed. >>> Now the value 123 unparses as 00123 but ... here's the problem.... -123 >>> unparses as 0-123. Notice how the zero padding is before the minus sign >>> when we wanted it to appear after. >>> >>> This problem is caused by ICU taking nearly all the information from the >>> positive part of the textNumberPattern. The negative part of the pattern, >>> if it exists, is used only to define the affix (prefix or suffix or both) >>> that indicate negative values. >>> >>> The problem is that positive numbers commonly have no affix, so the >>> position of padding characters relative to the affix cannot always be >>> determined from the positive pattern alone. >>> >>> Hence, if textNumberPattern specifies a pad character before the number >>> pattern and without a positive prefix, then ICU defaults to a pad position >>> of PAD_BEFORE_PREFIX with no way to change it with just the pattern. >>> >>> This behavior is reasonable for most cases, like when the pad character >>> is a space. However, if the pad character in textNumberPattern is '0', then >>> negative numbers are padded with a '0' before the negative sign. So we get >>> the errant behavior where a pattern of "*0####0" unparses -123 to "0-123". >>> This is very unlikely to be what the user wants with this pattern. >>> >>> Now suppose the positive pattern required a prefix "+" sign. The >>> textNumberPattern of "+*0####0" works properly because ICU determines that >>> the padding is PAD_AFTER_PREFIX from the positive pattern where the "*0" is >>> after the "+" prefix. >>> >>> The proposed fix to this issue that we're implementing in Daffodil is >>> this: If both negative and positive patterns define padding on the same >>> affix, and the positive pattern has an empty string for that affix, then we >>> use the pad position from the negative pattern. In all other cases, the pad >>> character in the negative pattern is ignored following usual ICU behavior. >>> >>> For example, a textNumberPattern of "*0####0;-*00" formats a negative >>> number with zero padding after the hyphen, whereas normal ICU behavior >>> would ignore the negative pattern and zero pad before the hyphen. >>> >>> >>> I would suggest this is something that is a needed fix in general for >>> DFDL. >>> >>> >>> The workaround when you have the fixed length number use case (-9999 to >>> 99999) is very ugly, treating the minus sign as an initiator and creating >>> separate elements for positive and negative values, or punting on this >>> integer and treating the whole thing as a string. >>> >>> >>> Arguably, this might be considered an ICU bug, but ICU's API can be used >>> to specify the PAD_AFTER_PREFIX behavior, so it's not like ICU doesn't let >>> you achieve the needed behavior, it's just not something that can be >>> achieved using only the ICU pattern string. But ICU maintainers may or may >>> not consider this to be a bug. >>> >>> >>> Mike Beckerle >>> Apache Daffodil PMC | daffodil.apache.org >>> OGF DFDL Workgroup Co-Chair | >>> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl >>> Owl Cyber Defense | www.owlcyberdefense.com >>> >>> >>> -- >>> dfdl-wg mailing list >>> dfdl-wg(a)lists.ogf.org >>> https://lists.ogf.org/mailman/listinfo/dfdl-wg >>> >>

2 1

Resolution of the ICU-related "#.##" textNumberPattern with 0.12
by Mike Beckerle 14 Mar '24

14 Mar '24

I cannot find the original email thread about this, but for records/posterity... The issue is described in this JIRA ticket: https://issues.apache.org/jira/browse/DAFFODIL-2859 That was closed as "Won't Fix". In short, why does 0.12 format as "0.12" instead of ".12" when textNumberPattern is "#.##". It would seem that "#.##" would mean the leading 0 is optional. But it doesn't. ICU chooses the minimum number of integer digits to be 1 even when the textNumberPattern has "#" before the decimal point. ICU team did not resolve this either way, the bug I reported there remains open: https://unicode-org.atlassian.net/browse/CLDR-17309 But, this would be a significant change in ICU behavior, so probably cannot change in ICU without requiring a switch be thrown to enable it. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

DFDL Workgroup Meeting 2024-03-14 - note time due to US change to daylight time
by Mike Beckerle 14 Mar '24

14 Mar '24

Note: Eastern time zone of US changed to DST last weekend. We are now UTC-5. This call is at 16:00 UTC, which is 11am EDT.US. Possibly a short call, but we should probably quickly review the ISO process. Also, I notice this new use of the acronym DFDL: http://signal.ee.psu.edu/research/dfdl.html There has been long standing DFDL use by a legal-related firm in Bangladesh, which really doesn't conflict with us. But this new usage is in a technology area so it gives me some concern. Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy

2 2

*SPAM* DFDL trademarked?
by Mike Beckerle 13 Mar '24

13 Mar '24

Spam detection software, running on the system "mail.pglaf.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Does OGF have any sort of trade-mark or brand protection on the DFDL acronym? or even on "OGF DFDL"? I noticed this new use of the acronym DFDL: http://signal.ee.psu.edu/research/dfdl.html There is plenty of other use of DFDL already: Content analysis details: (5.5 points, 4.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: psu.edu] 0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60% [score: 0.4944] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [mbeckerle.dfdl[at]gmail.com] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 HTML_MESSAGE BODY: HTML included in message 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.9 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/) 1.9 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50% [cf: 100] -0.0 T_SCC_BODY_TEXT_LINE No description available. 0.0 FSL_BULK_SIG Bulk signature with no Unsubscribe 2.0 URI_DOTEDU Has .edu URI The original message was not completely plain text, and may be unsafe to open with some email clients; in particular, it may contain a virus, or confirm that your address can receive spam. If you wish to view it, it may be safer to save it to a file and open it with an editor.

2 1