- dfdl-wg - lists.ogf.org

Resolution of the ICU-related "#.##" textNumberPattern with 0.12
by Mike Beckerle 14 Mar '24

14 Mar '24

I cannot find the original email thread about this, but for records/posterity... The issue is described in this JIRA ticket: https://issues.apache.org/jira/browse/DAFFODIL-2859 That was closed as "Won't Fix". In short, why does 0.12 format as "0.12" instead of ".12" when textNumberPattern is "#.##". It would seem that "#.##" would mean the leading 0 is optional. But it doesn't. ICU chooses the minimum number of integer digits to be 1 even when the textNumberPattern has "#" before the decimal point. ICU team did not resolve this either way, the bug I reported there remains open: https://unicode-org.atlassian.net/browse/CLDR-17309 But, this would be a significant change in ICU behavior, so probably cannot change in ICU without requiring a switch be thrown to enable it. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

DFDL Workgroup Meeting 2024-03-14 - note time due to US change to daylight time
by Mike Beckerle 14 Mar '24

14 Mar '24

Note: Eastern time zone of US changed to DST last weekend. We are now UTC-5. This call is at 16:00 UTC, which is 11am EDT.US. Possibly a short call, but we should probably quickly review the ISO process. Also, I notice this new use of the acronym DFDL: http://signal.ee.psu.edu/research/dfdl.html There has been long standing DFDL use by a legal-related firm in Bangladesh, which really doesn't conflict with us. But this new usage is in a technology area so it gives me some concern. Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy

2 2

*SPAM* DFDL trademarked?
by Mike Beckerle 13 Mar '24

13 Mar '24

Spam detection software, running on the system "mail.pglaf.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Does OGF have any sort of trade-mark or brand protection on the DFDL acronym? or even on "OGF DFDL"? I noticed this new use of the acronym DFDL: http://signal.ee.psu.edu/research/dfdl.html There is plenty of other use of DFDL already: Content analysis details: (5.5 points, 4.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: psu.edu] 0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60% [score: 0.4944] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [mbeckerle.dfdl[at]gmail.com] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 HTML_MESSAGE BODY: HTML included in message 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.9 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/) 1.9 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50% [cf: 100] -0.0 T_SCC_BODY_TEXT_LINE No description available. 0.0 FSL_BULK_SIG Bulk signature with no Unsubscribe 2.0 URI_DOTEDU Has .edu URI The original message was not completely plain text, and may be unsafe to open with some email clients; in particular, it may contain a virus, or confirm that your address can receive spam. If you wish to view it, it may be safer to save it to a file and open it with an editor.

2 1

Re: [DFDL-WG] proof text for ISO/IEC 23415
by Mike Beckerle 25 Feb '24

25 Feb '24

Lisa, The footer is cut off on every page of this PDF, making it impossible to see the page numbers. I have not so far found any problem with the content of the pages, but this problem with the page footers does require a re-spin of the PDF, and if the page sizes change the table-of-contents will need its page numbers updated. -mike Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Feb 20, 2024 at 4:32 AM Lisa Rajchel <lrajchel(a)ansi.org> wrote: > Good morning, > > > > Attached please find the proof text for ISO/IEC 23415. Please confirm, by > March 18, 2024, that the content is ready for publication. > > > > Best > > Lisa >

2 1

Fwd: proof text for ISO/IEC 23415
by Mike Beckerle 21 Feb '24

21 Feb '24

We have an action to review this proof, no later than March 18th. I uploaded it to our gitlab ISO-Process folder. https://github.com/OpenGridForum/DFDL/blob/master/ISO-process/Data%20Format… Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> ---------- Forwarded message --------- From: Lisa Rajchel <lrajchel(a)ansi.org> Date: Tue, Feb 20, 2024 at 4:32 AM Subject: proof text for ISO/IEC 23415 To: Sill, Alan <Alan.Sill(a)ttu.edu> Cc: Wolfgang Ziegler <wolfgang.ziegler(a)z-rands.de>, Norbert Bensalem < norbert_bensalem(a)fr.ibm.com>, Jens Jensen <jens.jensen(a)stfc.ac.uk>, Greg Newby <gbnewby(a)petascale.org>, Steve Hanson <smhdfdl(a)gmail.com>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Good morning, Attached please find the proof text for ISO/IEC 23415. Please confirm, by March 18, 2024, that the content is ready for publication. Best Lisa

1 0

dfdl:extraEscapedCharacters - should be able to be expression
by Mike Beckerle 17 Feb '24

17 Feb '24

A user has noticed that for DFDL escape schemes based on escape characters, the properties dfdl:escapeCharacter and dfdl:escapeEscapeCharacter can both be runtime-valued expressions, but not dfdl:extraEscapedCharacters. This is not consistent. A major motivation for having these properties be runtime expressions is EDI formats, and the example motivating the need for dfdl:extraEscapedCharacters to be an expression also comes from EDIFACT. See this Daffodil ticket with a link to the email thread discussing this: https://issues.apache.org/jira/browse/DAFFODIL-2876 Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Agenda for Workgroup Call on 2024-02-07
by Mike Beckerle 08 Feb '24

08 Feb '24

We do have some topics for this month's call. Agenda: see https://github.com/OpenGridForum/DFDL/blob/master/calls/2024/2024-02-07_DFD… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Action 338: pathological DFDL behavior vs transformations in DFDL
by Mike Beckerle 07 Feb '24

07 Feb '24

Bradd, There's this example schema that illustrates DFDL being used to transform a pair of lists (which are hidden in a hidden group) into a list of pairs (which is the visible infoset). https://github.com/OpenDFDL/examples/tree/master/pairsTransform I'm curious if this illustrates the nature of the transformations you describe that zTPF users are doing or if there is more to it. The output is not the same "shape" as the parsed input data, but it is linearly related in size. -mike Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Example schema that illustrates pathological DFDL behaviors
by Mike Beckerle 07 Feb '24

07 Feb '24

I have been concerned for a while that combinations of the features in DFDL v1.0 can be used in ways that are just far too powerful for what is needed for data format expression. For example, if the input is n bits, how large should the corresponding infoset be when the data is parsed using a DFDL schema? Clearly at least O(n) since there are n bits and each bit could create an element in the infoset. But should it be able to be O(n^2) ? I can think of no use case for this. Intuition about data formats suggests that the infoset should always be linearly related in size to the size of the input data, and that the constant factor on that should be relatively small. Furthermore, the time taken to parse data should be linearly related to the size of the data. I think lots of people are thinking DFDL parsing should be implementable in a circuit, and the memory requirements are proportional to the data size. It doesn't make sense to try to make hardware implementations of DFDL if that's not going to be the case. Turns out, using DFDL v1.0 using dfdl:occursCountKind='expression' and dfdl:inputValueCalc, can create an infoset from parsing that is super-exponentially larger than the input data. Yes, it is finite, and bounded to some function of the size of the input data. But we don't even have notations for this kind of numeric growth. The schema I created consumes a 3 bit integer as the data. The size of the infoset grows like this: - input 000 -> infoset size 1 - input 001 -> infoset size 1 - input 010 -> infoset size 256 - input 011 -> infoset size is over 10,000 elements - ... - input 111 -> infoset size is larger than the number of atoms in the universe. There are TDML tests for this schema that illustrate this (except that last one) But it is always finite! No recursion is required to do this. With infinite space and time available it WILL terminate. I believe the language remains theoretically decidable. The README in this schema project discusses this and other pathological cases, and suggests things we can do to restrict DFDL so that this kind of pathological behavior is not allowed. Really we want the largest infoset that can be created from - n bits of data - a schema of size s to be O(sn) in size. I have started an argument in the README about why O(sn) is the right target. Basically there are practical schemas that can meet O(sn) as a lower bound. I believe this can be achieved by adding more forward-progress requirements to DFDL - all arrays should have to make forward progress when parsing, not just unbounded arrays, and reused groups/types should have to make forward progress at points of reuse. By doing that, we ensure that to make a giant infoset, you have to be consuming a giant number of bits. Right now this stuff is just living in a personal github repo, but I'll give it an official better home either in DFDLSchemas or as part of Apache Daffodil, or something later. If this is of interest, please take a look at this github pull request below. ---------- Forwarded message --------- From: Mike Beckerle <notifications(a)github.com> Date: Wed, Jan 24, 2024 at 3:50 PM Subject: [mbeckerle/superexp] Example schema that illustrates pathological DFDL behaviors (PR #1) To: mbeckerle/superexp <superexp(a)noreply.github.com> Cc: Mike Beckerle <mbeckerle(a)apache.org>, Your activity < your_activity(a)noreply.github.com> Illustrates DFDL issues where parsing small data can lead to explosively larger parse output. Super exponentially larger. The expressive power of DFDL should be restricted so that this is not possible. See the README.md for discussion of the issue. ------------------------------ You can view, comment on, or merge this pull request online at: https://github.com/mbeckerle/superexp/pull/1 Commit Summary - 46dbae6 <https://github.com/mbeckerle/superexp/pull/1/commits/46dbae6b2dc5acd76d6954…> First version of this example schema File Changes (9 files <https://github.com/mbeckerle/superexp/pull/1/files>) - *A* .gitattributes <https://github.com/mbeckerle/superexp/pull/1/files#diff-618cd5b83d62060ba3d…> (5) - *M* .gitignore <https://github.com/mbeckerle/superexp/pull/1/files#diff-bc37d034bad56458379…> (7) - *M* README.md <https://github.com/mbeckerle/superexp/pull/1/files#diff-b335630551682c19a78…> (96) - *A* build.sbt <https://github.com/mbeckerle/superexp/pull/1/files#diff-5634c415cd8c8504fdb…> (30) - *A* project/build.properties <https://github.com/mbeckerle/superexp/pull/1/files#diff-cbc85b9df7818a480f2…> (1) - *A* src/exp.dfdl.xsd <https://github.com/mbeckerle/superexp/pull/1/files#diff-9751c75e50c64f92bee…> (84) - *A* src/superexp.dfdl.xsd <https://github.com/mbeckerle/superexp/pull/1/files#diff-2c057f270cab45f3f05…> (72) - *A* test/TestSuperexp.scala <https://github.com/mbeckerle/superexp/pull/1/files#diff-c6b466d6536f0f670f9…> (35) - *A* test/TestSuperexp.tdml <https://github.com/mbeckerle/superexp/pull/1/files#diff-f1b4e40b448ec3ab530…> (117) Patch Links: - https://github.com/mbeckerle/superexp/pull/1.patch - https://github.com/mbeckerle/superexp/pull/1.diff — Reply to this email directly, view it on GitHub <https://github.com/mbeckerle/superexp/pull/1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALUDAZF7ZM5XWD72HZ4Z5DYQ…> . You are receiving this because you are subscribed to this thread.Message ID: <mbeckerle/superexp/pull/1(a)github.com>

1 1

Action 336: ISO announcement and branding
by Mike Beckerle 07 Feb '24

07 Feb '24

Please add to the list of questions for short term. If this looks ok then we'll forward the questions to the ISO JTC1 secretary to get info. - When can we update OGF web sites, WIkipedia, blogs/news, X/Twitter, and other web content to announce and reflect our ISO status? - Are there branding guidelines for how one does this in conformance with ISO practices/policy? E.g., must use specific phrasing, must cite specific trademarks or such? Are there official logos we are supposed to use? (Or rules about when we can use those or are not to use them, etc.) - Can/should we update the cover of the OGF DFDL spec (GFD.240) on the OGF web site to reflect the ISO PAS status? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0