- dfdl-wg - lists.ogf.org

Topic: Creating ISO document from XML source
by Mike Beckerle 06 Jun '24

06 Jun '24

A goal/need is for our next revision of the DFDL spec to be in ISO format. Bill Ash (ISO JTC1 SC38) put me in contact with Jim Melton (ISO SQL standard) about their process which starts from an XML source to create their document. Jim has passed along a chapter of their XML, the XSLT, and other documentation about how they do it, which we can learn from. Below is the email thread with Jim Melton. I'm putting the zip-file attachment materials mentioned into our github repo. Now all we have to do is somehow convert the DFDL v1.0 MS-Word document into a marked up XML. Interestingly, practices for doing this kind of thing to large data format specifications at my employer are to first convert the MS-Word to text using PDFBox, then use DFDL to mine the text. The resulting DFDL infoset is converted to XML, then Scala or XSLT or both transformations can turn it into something maintainable. ---------- Forwarded message --------- From: Jim Melton <SheltieJim(a)xmission.com> Date: Thu, Jun 6, 2024 at 12:46 PM Subject: Re: Creating ISO document from XML source To: <mbeckerle(a)apache.org> Cc: <SheltieJim(a)xmission.com> Mike, Sorry I took several days to reply, but I'm leaving for a WG 3 meeting in China today and have been busy preparing for that meeting. I've attached a ZIP file to this message. In that ZIP archive, you will find some XML files and some XSL (XSLT, actually) files, as well as a DTD. These should have enough information to give you an idea of how we're using XML. The ZIP also contains some HTML files (generated from MarkDown) that serve as our documentation. The file named sql-psm.xml is the source file for one part of the SQL standard, ISO/IEC 9075-4:2023. I did not include the PDF that results from compiling sql-psm.xml, in large part because it's over 7MB. If you want that file, I can send it separately, but not for a couple of days (because I'm leaving shortly for the airport). If you want to talk about any of this, ask questions about how/why we did something, etc., I'll be back from China on June 17 or 18. Hope this helps, Jim On 2024-06-03 09:31, Mike Beckerle wrote: Jim, Sure. Please send the bundle of things you suggested would be of interest. We can start by looking at what you have done and seeing how our situation matches and differs. If it's too large for a single big zip attachment, you can split it up how you like, or we can come up with some alternative path. Thank you Mike Beckerle On Fri, May 31, 2024 at 4:11 PM Jim Melton <SheltieJim(a)xmission.com> wrote: > > Mike, > > Pleased to meet you 😉 > > In JTC 1/SC 32/WG 3, we never developed our standards in Word, although other WGs in SC 32 do so. ISO/IEC 9075 (Database language SQL) was originally developed in some primitive IBM markup language whose name I have forgotten. When I became editor of the standard, I converted it to an "SGML-like" product named DECdocument (I worked at Digital Equipment Corp at the time). Later, after I left Digital, I converted the DECdocument markup to XML, created an XML vocabulary (in the form of a DTD), generated XSL-FO from the XML, then found a rendering engine (XEP, by RenderX) to convert to final PDF form. > > At this point (mid-2024), we have a highly functional system that generated PDF documents with massive numbers of internal and inter-document hot links, smaller "pointer documents" that allow a given document to be referenced in multiple meetings without duplicating the storage, artifacts (such as the BNF of the language), etc. We also have numerous programs doing source code validity checking (i.e., following our coding standards and ISO's Directives). I realize that not everything we have done will be relevant to other standards, but it will provide assistance in figuring out how to do what you want and may serve as a starting point for you. > > I would be happy to send you a standard document in both PDF and XML versions, as well as the DTD and primary stylesheets, and a MarkDown document that explains our coding standards and more. Just let me know! And, of course, I'm happy to answer any questions you might have about all of this stuff. > > Hope this helps, > Jim > > On 2024-05-31 09:15, Mike Beckerle wrote: > > Jim, > > I have heard you have an ISO-document creation system that starts from an XML source. > > We have a monster MS-Word document that is unwieldy, and not useful for generating any other formal artifacts. > > I hope to convert it to XML and then generate the ISO document from that XML, as well as other artifacts like online-doc for GUIs, annotated versions of the spec, etc. > > Can you point me at how your team(s) work uses XML? > > Any help greatly appreciated. > > Mike Beckerle > OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl > Apache Daffodil PMC | daffodil.apache.org > Owl Cyber Defense | www.owlcyberdefense.com > > -- > Jim Melton SheltieJim at xmission dot com > ------------------------------------------- > Owner and Captain: > Passport 40 #018 Dream SeQueL > Co-Owner and Co-Captain > Passport 42 #040 Turtle Blues > ------------------------------------------- > Shelties since 1969; ASSA member since 1992 -- Jim Melton SheltieJim at xmission dot com ------------------------------------------- Owner and Captain: Passport 40 #018 Dream SeQueL Co-Owner and Co-Captain Passport 42 #040 Turtle Blues ------------------------------------------- Shelties since 1969; ASSA member since 1992

1 1

Action 322: StringAsXML Experience Document
by Mike Beckerle 06 Jun '24

06 Jun '24

See this Pull Request: https://github.com/OpenGridForum/DFDL/pull/57 to comment, or this to read the document: https://github.com/OpenGridForum/DFDL/blob/4b1f50220ee064f662e6fe1339f84916… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Action 337: ISO DFDL Workgroup Operation changes
by Mike Beckerle 20 May '24

20 May '24

This action item is about how we operate our workgroup going forward, and how that evolves now that we're an ISO/IEC standard as well as an OGF one. I believe we will need to create a workgroup document describing these new practices, particularly when they augment/vary some of the standard OGF practices. Please add to this list of issues about how we operate going forward. - Typography: I assume we have to revise the OGF Spec to match Standard ISO format (Beyond just A4 page size) so future versions come out in that conforming style. - Once we do this I don't want to be maintaining an OGF flavor and an ISO flavor of the formatting. - Does a preliminary working document of the DFDL workgroup have to be specifically marked as such (so that it is clearly preliminary)? What are these markings? - If that is done without revisions of content from current OGF content, would that replace the current ISO-provided document (which is just a new cover page on our existing OGF GFD.240 document) or does that wait for a content revision? - Who are contacts within JTC1 ? Does the OGF Workgroup contact JTC1 by way of OGF leadership or directly? (or... what is the chain of command here? Don't want to be jumping across protocol boundaries!) - Is there a TC within JTC1 that we should work more closely with? Right away? Eventually? Need contacts there. - In 6 years a PAS is supposed to become an International Standard. What does that mean? Does PAS status end? Does OGF participation end, or does the workgroup merge into a TC? Or is this just a status change assuming things are going well? - Day to day working group activity/meetings - is there guidance about notes/minutes-taking, etc. (templates)? - Such time as we have revisions to the document, how do we engage with ISO to get an update to the PAS document? Are these voted? - What about just typographical corrections? - Thoughts: Do we just use the OGF process for updates/corrections and then just submit the changes to JTC1 (some TC?) for consideration? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 1

DFDL implementation support for element refs
by Mike Beckerle 09 May '24

09 May '24

I'm interested in what DFDL implementations support element references? IBM ACE? IBM zTPF? DFDL4Space? Can you let me know whether these implementations support element refs? The reason I ask is below, which may be of interest or perhaps TL;DR. We support element references in Daffodil, but I'm coming around to the view that element refs are a bad idea in DFDL schemas. They're not needed for any specific data format expressive power. That suggests we should have left them out of DFDL, but for some reason we didn't. The problem is that most data languages have nothing like element references and the associated element namespace management complexity available. So as soon as you want to use a DFDL schema but not use it to interchange data as XML, element refs become a problem. I'm playing around with a best practice/subset/profile suggestion where: * The only global element declarations in the schema are for root elements. * Element references are disallowed * The root elements are declared in a root schema file that contains ONLY the root elements * Root elements should always be declared by one-liners like this: `<element name="rootElement" type="prefix:rootElementType"/>` * The root elements schema file has no target namespace. * All group, type, and DFDL format/escapeScheme/variable definitions must be declared in different schema files that may (and probably should) have a target namespace. The benefit of these restrictions is that the elements in the nest of a DFDL infoset never have any namespaces. This makes them compatible with non-namespaced data systems like JSON, Apache Drill, Apache NiFi, Generated C code, etc. This makes integration with those things *massively* simpler. Such schemas are still easily reused by reusing the type of the root element, so there is no need to ever use an element reference, and a nice composition property occurs - you don't need element references to assemble schemas from component schemas, and the assembled component has the same characteristic. There are a few other things this discipline also simplifies. Reusing test data becomes simpler if namespace URIs aren't getting embedded in every test infoset XML file, for example. All comments are welcome. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 3

Agenda for DFDL WG Call 2024-05-09
by Mike Beckerle 09 May '24

09 May '24

https://github.com/OpenGridForum/DFDL/blob/master/calls/2024/2024-05-09_DFD… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Reminder: Agenda topics for Thursday May 8 DFDL call
by Mike Beckerle 08 May '24

08 May '24

I believe the requested agenda topic for the May 8 call was discussion of DFDL 2.0 and/or DFDL 1.1? Please circulate emails about topics in that area you'd like to discuss on the call, or generally. There is a problem called the "second system effect" I think which is when version 2.0 never happens because the team wants to fix every single thing wrong with version 1.0, and perfectionism sets in. We want to avoid that. There is also a role for what I would call "standard profiles" which are knobs to turn to tell a DFDL processor that you want to limit your DFDL usage so that your schemas will have certain standard behaviours. An example is to disallow elements to be distinguished only by namespace, so that the DFDL schema can be used to convert data to/from JSON or other data frameworks where elements have only names, and there is no notion of namespaces for names. We've run into this in JSON and a major integration of Daffodil with Apache Drill. DFDL users would benefit from a way to say "keep me out of namespace troubles" like that. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Re: [DFDL-WG] [EXTERNAL] Fwd: proof text for ISO/IEC 23415
by Mike Beckerle 19 Apr '24

19 Apr '24

Lisa, We found some additional issues with the previously sent ISO DFDL PDF. Besides the problem with the footers being cut off (already reported) we found: - Links do not work: - Table-of-contents entries do not let you navigate to the sections. - Internal links like "[XML]" to the references section do not work. - Hyperlink text in the references sections do not work. - The page margins are inconsistent. Viewed with 2 pages on screen side by side, the left pages have narrow left margin in comparison to right pages. - Tables 46 and 47 in Sections 14.2.1 and 14.2.1 break across page boundaries. This makes them very hard to interpret, and while some large tables can split across pages normally, for these specific tables, the page boundary makes it very hard to understand them. Manual breaks should be inserted to avoid this. - Table 48 in Section 14.2.3 does not have a bad-break currently. But this table is also one that a page-break across it needs to be avoided. Please forward these to the right parties who are preparing the ISO/IEC version. Thank you Mike Beckerle OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Apache Daffodil PMC | daffodil.apache.org Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Fri, Mar 8, 2024 at 9:42 AM Lisa Rajchel <lrajchel(a)ansi.org> wrote: > Mike, > > I have sent your comment to ISO and now it is their hands. No one has > asked for anything more from you. I will keep you posted. > > Best > Lisa > > Sent with BlackBerry Work > (www.blackberry.com) > > > *From: *Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > *Date: *Friday, Mar 08, 2024 at 9:23 AM > *To: *Lisa Rajchel <lrajchel(a)ansi.org> > *Subject: *[EXTERNAL] Fwd: [DFDL-WG] proof text for ISO/IEC 23415 > > Lisa, > > I am following up on this need for another spin on the DFDL 23415 official > ISO version. > > As far as I know this is out of our hands, meaning the workgroup on DFDL. > However, we have a workgroup meeting this week, I am wondering if there are > any actions we need or could take to help get the 23415 through the process > and fully published. > > Please advise > > Thank you > > > Mike Beckerle > OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl > Apache Daffodil PMC | daffodil.apache.org > Owl Cyber Defense | www.owlcyberdefense.com > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the OGF Intellectual Property Policy > <http://www.ogf.org/About/abt_policies.php> > > > > ---------- Forwarded message --------- > From: Steve Hanson <smhdfdl(a)gmail.com> > Date: Sun, Feb 25, 2024 at 5:10 AM > Subject: Re: [DFDL-WG] proof text for ISO/IEC 23415 > To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > Cc: Lisa Rajchel <lrajchel(a)ansi.org>, DFDL-WG <dfdl-wg(a)ogf.org> > > > It looks like the footer is being cut off in order to accommodate the > three lines of licence text in every header. > > Regards > Steve > > On Sat, Feb 24, 2024 at 10:39 PM Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > wrote: > >> Lisa, >> >> The footer is cut off on every page of this PDF, making it impossible to >> see the page numbers. >> >> I have not so far found any problem with the content of the pages, but >> this problem with the page footers does require a re-spin of the PDF, and >> if the page sizes change the table-of-contents will need its page numbers >> updated. >> >> -mike >> >> Mike Beckerle >> OGF DFDL Workgroup Co-Chair | >> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl >> Apache Daffodil PMC | daffodil.apache.org >> Owl Cyber Defense | www.owlcyberdefense.com >> Please note: Contributions to the DFDL Workgroup's email discussions are >> subject to the OGF Intellectual Property Policy >> <http://www.ogf.org/About/abt_policies.php> >> >> >> >> On Tue, Feb 20, 2024 at 4:32 AM Lisa Rajchel <lrajchel(a)ansi.org> wrote: >> >>> Good morning, >>> >>> >>> >>> Attached please find the proof text for ISO/IEC 23415. Please confirm, >>> by March 18, 2024, that the content is ready for publication. >>> >>> >>> >>> Best >>> >>> Lisa >>> >> -- >> dfdl-wg mailing list >> dfdl-wg(a)lists.ogf.org >> https://lists.ogf.org/mailman/listinfo/dfdl-wg >> > > > This e-mail is intended exclusively for the individual or entity to which > it is addressed or otherwise directed. This e-mail (and its attachments) > may contain confidential, proprietary or privileged information, and > unauthorized disclosure or use of such information is prohibited absent > express authorization of the sender. If you receive this e-mail in error, > please notify the sender and delete this e-mail from your system. >

1 3

Call minutes for 2024-04-18
by Mike Beckerle 19 Apr '24

19 Apr '24

Call minutes are here: https://github.com/OpenGridForum/DFDL/blob/master/calls/2024/2024-04-18_DFD… A proof-copy of the ISO version of the DFDL spec is here: https://github.com/OpenGridForum/DFDL/blob/master/ISO-process/ISO_IEC%20234… The name reflects that the file is watermarked and for A.Sill/M.Beckerle, not for general distribution. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

3 7

Agenda for Call 2024-04-18 11:00-04:00
by Mike Beckerle 17 Apr '24

17 Apr '24

See https://github.com/OpenGridForum/DFDL/pull/55/files#diff-c18a5ebbb168df129b… Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

1 0

Fwd: [ANNOUNCE] Apache Daffodil 3.7.0 Released
by Mike Beckerle 15 Apr '24

15 Apr '24

---------- Forwarded message --------- From: Steve Lawrence <slawrence(a)apache.org> Date: Wed, Apr 10, 2024, 7:16 AM Subject: [ANNOUNCE] Apache Daffodil 3.7.0 Released To: <dev(a)daffodil.apache.org> The Apache Daffodil community is pleased to announce the release of version 3.7.0. This release includes a number of bug fixes and performance improvements. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/3.7.0/ Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team

1 0