- dfdl-wg - lists.ogf.org

possible issue - can't use pattern to validate DFDL content containing newlines
by Mike Beckerle 04 Mar '21

04 Mar '21

Since data contains any characters, the DFDL infoset allows any characters. However, XML does not allow any characters. Furthermore, XML Schema Pattern facets are expressed using this XML Schema fragment: <xs:pattern value="...some regex pattern here ..."/> But XML attributes are normalized by XML readers/parsers. Line endings in them are converted to single spaces. So <xs:pattern value="abc def"/> is equivalent to: <xs:pattern value="abc def"/> Furthermore <xs:pattern value="abc
def"/> is also normalized to <xs:pattern value="abc def"/> As far as I can tell there is no alternate notation to this. This means, if you want to use a pattern facet to specify that a DFDL infoset string can contain A-Za-z0-9 spaces and line endings, there is no way to express this. This pattern was the example I was dealing with. <xs:pattern value="[A-Za-z0-9 
]*"/> If you look at the string for the value attribute of this pattern element, that string already has the line ending characters converted into spaces. The attribute value is "[A-Za-z0-9 ]*" which has 3 spaces before the "]". I think there is no workaround for this in XML, XSD, or DFDL. I dug into the Daffodil implementation and in the code that accesses this attribute, you don't even get a NodeSeq containing a mixture of Text and Entity nodes. You just get a single Text node. So it is pretty well hopeless without reaching under the XML parser/reader's guts. Hence, in DFDL if you want to "validate" that a DFDL string contains content that includes line-endings with a regex, you have to use dfdl:assert with failureType="recoverableError" testKind="pattern" and testPattern with the regex of interest. This is then a DFDL regex, which is a Java regex, and you can be explicit about line endings allowed. You can't do it with a pattern facet. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 1

Contributing to github
by Bradd Kadlecik 03 Mar '21

03 Mar '21

Hi, I was unable to create a branch in github to upload the experience document for indirection. My userid there is tpfbraddk. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States

2 6

DFDL specification v1.0 - full recommendation status
by Mike Beckerle 23 Feb '21

23 Feb '21

Per thread below, the DFDL spec was reviewed by the OGF Standards Council, and with two minor changes (which are done) they are moving to publish this as a full recommendation. Congratulations to all involved. Final links will be forthcoming. I expect there to be some press announcement about this from somewhere. ---------- Forwarded message --------- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: Tue, Feb 23, 2021 at 10:44 AM Subject: Re: Final DFDL specification - Re: DFDL Public Comment Ended To: Greg Newby <gbnewby(a)petascale.org> Attached is PDF and docx of version 1.0.8. The only changes are the redraw of figure 2, and I sorted the items in 4.2.2. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Feb 23, 2021 at 9:33 AM Greg Newby <gbnewby(a)petascale.org> wrote: > Thanks, Mike. Appreciate it! > - Greg > > On Tue, Feb 23, 2021 at 7:02 AM Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > wrote: > >> I'll get these fixed and get you a new PDF and Docx today. >> >> Thank you very much. >> >> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | >> www.owlcyberdefense.com >> Please note: Contributions to the DFDL Workgroup's email discussions are >> subject to the OGF Intellectual Property Policy >> <http://www.ogf.org/About/abt_policies.php> >> >> >> >> On Mon, Feb 22, 2021 at 11:25 AM Greg Newby <gbnewby(a)petascale.org> >> wrote: >> >>> Hi, Mike. >>> >>> The standards council reviewed this, and was able to discuss during >>> today's meeting. >>> >>> There are only two very minor inquiries: >>> >>> Image 2 (p. 20) has some artifacts due to being a bitmap-type image. If >>> you still have the source for this, we wondered whether a higher resolution >>> version might be generated. In particular, there are some greyish >>> background artifacts behind 0..1, the shaded boxes, and perhaps some other >>> locations. It looks worse in the PDF, but this is also visible in the >>> .docx. If the source is no longer easily available, it's ok to leave this >>> as-is. >>> >>> 4.2.2 information items: It was noted that the items listed here are not >>> in alphabetical order. Is that intentional? Much of it is in alpha order, >>> but not the whole list. >>> >>> These are very minor comments. Overall, the document is excellent. Well >>> written and complete, and full due attention to the public comments. >>> >>> Let me know about the two items above. No further review is needed, and >>> once you confirm/provide any final document, I will get it published & >>> announced, and the obsolescence handled, in fairly short order. >>> >>> Best, >>> Greg >>> >>> >>> >>> On Wed, Feb 17, 2021 at 10:34 AM Greg Newby <gbnewby(a)petascale.org> >>> wrote: >>> >>>> Hi, Mike. In fact the first discussion happened just today. There will >>>> be a second discussion on Monday, February 22, because due to time >>>> constraints the discussion didn't conclude. >>>> >>>> There is a very good chance that final approval will occur on Monday. >>>> It will then take a few days for me to get the document + obsoleted >>>> document into their final PDF form and online at www.orf.org >>>> >>>> Best, >>>> Greg >>>> >>>> On Wed, Feb 17, 2021 at 10:32 AM Mike Beckerle < >>>> mbeckerle.dfdl(a)gmail.com> wrote: >>>> >>>>> Hi Greg, >>>>> >>>>> Any word on when the standards council will meet about this? >>>>> >>>>> I only nag because there are some other things happening related to >>>>> DFDL implementations where some press synergy is available depending on >>>>> time frames. >>>>> >>>>> Thanks for any info >>>>> >>>>> -mikeb >>>>> >>>>> >>>>> >>>>> >>>>> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | >>>>> www.owlcyberdefense.com >>>>> Please note: Contributions to the DFDL Workgroup's email discussions >>>>> are subject to the OGF Intellectual Property Policy >>>>> <http://www.ogf.org/About/abt_policies.php> >>>>> >>>>> >>>>> >>>>> On Mon, Feb 8, 2021 at 10:01 AM Greg Newby <gbnewby(a)petascale.org> >>>>> wrote: >>>>> >>>>>> This is perfect, Mike, and very much appreciated. I've pinged the >>>>>> standards council again to set up our next meeting, and otherwise they'll >>>>>> discuss by email. >>>>>> >>>>>> Hopefully we're just a couple of weeks away from putting this online >>>>>> as a new document, and obsoleting the old. I appreciate your stewardship >>>>>> for this document's journey! >>>>>> >>>>>> Best, >>>>>> Greg >>>>>> >>>>>> >>>>>> On Wed, Feb 3, 2021 at 3:08 PM Mike Beckerle < >>>>>> mbeckerle.dfdl(a)gmail.com> wrote: >>>>>> >>>>>>> Greg, >>>>>>> >>>>>>> Attached is a changes-tracked version containing the changes made >>>>>>> based on the public review comment period, and final corrections found >>>>>>> since. >>>>>>> That one is v1.0.6. >>>>>>> >>>>>>> The attached version 1.0.7 has all changes accepted and is the clean >>>>>>> version for final publication. docx and pdf attached. >>>>>>> >>>>>>> The public review comments included several statements of support, >>>>>>> but these public comments were substantive (links): >>>>>>> >>>>>>> https://github.com/OpenGridForum/Editor/issues/16 - a large and >>>>>>> extensive review >>>>>>> https://github.com/OpenGridForum/Editor/issues/15 >>>>>>> https://github.com/OpenGridForum/Editor/issues/14 >>>>>>> https://github.com/OpenGridForum/Editor/issues/9 >>>>>>> https://github.com/OpenGridForum/Editor/issues/8 >>>>>>> >>>>>>> The first of these was based on an extensive review of the >>>>>>> specification by Don Brutzman of the US Naval Research Laboratory. >>>>>>> There are change paragraphs in the attached having to do with the >>>>>>> other 4 public comments. But with respect to this one extensive review, >>>>>>> there are many smaller changes, as he pointed out a number of >>>>>>> English-language issues (use of 1st person and 2nd person, future vs. >>>>>>> present tense, wrong mood of verbs, etc.) as well as just wording >>>>>>> improvements, all of which were corrected. Those make up the bulk of the >>>>>>> smaller changes. >>>>>>> >>>>>>> >>>>>>> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | >>>>>>> www.owlcyberdefense.com >>>>>>> Please note: Contributions to the DFDL Workgroup's email discussions >>>>>>> are subject to the OGF Intellectual Property Policy >>>>>>> <http://www.ogf.org/About/abt_policies.php> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 27, 2021 at 3:04 PM Greg Newby <gbnewby(a)petascale.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, Mike. Thanks for this. I had a calendar reminder for when >>>>>>>> public comment ended, but it was right at new years and I missed or >>>>>>>> neglected it. >>>>>>>> >>>>>>>> Indeed, your actions are exactly right: make a final version of the >>>>>>>> document, based on any input received during the public comment period. >>>>>>>> >>>>>>>> The next step is a final review by the standards council. For this, >>>>>>>> it is helpful if you can provide either a "track changes" version of the >>>>>>>> document that was reviewed, or a written summary of the nature of changes. >>>>>>>> Whichever is easier - the idea is to make it easier for the standards >>>>>>>> council to know what's changed, based on public comments received. >>>>>>>> >>>>>>>> Once the standards council approves publication, I will proceed to >>>>>>>> put that online, and update #207 as obsolete (I have the editable version >>>>>>>> of that, I believe). >>>>>>>> >>>>>>>> The standards council's next meeting isn't scheduled yet. We are >>>>>>>> trying to identify a new fortnightly meeting time, and hopefully will be >>>>>>>> meeting by the 2nd week of February. If you can provide some sort of record >>>>>>>> of changes, as described above, the final review can proceed. >>>>>>>> >>>>>>>> Thanks again, and best regards, >>>>>>>> Greg >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jan 26, 2021 at 3:30 PM Mike Beckerle < >>>>>>>> mbeckerle.dfdl(a)gmail.com> wrote: >>>>>>>> >>>>>>>>> Greg et al., >>>>>>>>> >>>>>>>>> We have ended the public comment period on DFDL earlier this >>>>>>>>> month. >>>>>>>>> >>>>>>>>> We got some good support statements, and one person gave the spec >>>>>>>>> a quite thorough review even, which was great. >>>>>>>>> >>>>>>>>> I have edited in changes per the public comments that suggested >>>>>>>>> edits/improvements. >>>>>>>>> >>>>>>>>> I think we are ready to finalize DFDL v1.0 GFD.240 as an official >>>>>>>>> OGF Recommendation, and mark the prior GFD.207 as obsolete. >>>>>>>>> >>>>>>>>> What is next? Do I just send you the pdf and ms-word docx files? >>>>>>>>> We are just proofreading my final edits and this will be ready. >>>>>>>>> >>>>>>>>> Thank you >>>>>>>>> >>>>>>>>> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | >>>>>>>>> www.owlcyberdefense.com >>>>>>>>> Please note: Contributions to the DFDL Workgroup's email >>>>>>>>> discussions are subject to the OGF Intellectual Property Policy >>>>>>>>> <http://www.ogf.org/About/abt_policies.php> >>>>>>>>> >>>>>>>>>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 18 Feb '21

18 Feb '21

Please find minutes from the latest call at https://github.com/OpenGridForum/DFDL/wiki/DFDL-Workgroup-News Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 18 Feb '21

18 Feb '21

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 18 Feb '21

18 Feb '21

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 18 Feb '21

18 Feb '21

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 18 Feb '21

18 Feb '21

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 18 Feb '21

18 Feb '21

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 17 Feb '21

17 Feb '21

Please find agenda for next call on GitHub at https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-02-18_DFDL_WG_… Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0