- dfdl-wg - lists.ogf.org

Confirmed: OGF DFDL Working Group Call (8 Jul 16:00 GDT in https:/ibm.webex.com/meet/smh)
by Steve Hanson 08 Jul '21

08 Jul '21

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Clarification needed " .[1] " expression on an array element
by Mike Beckerle 30 Jun '21

30 Jun '21

Apache Daffodil has this bug report: https://issues.apache.org/jira/browse/DAFFODIL-2182 It calls for getting clarification from the DFDL WG. Suppose I have an array element named "a" and it contains an assert. The assert expression can contain the subexpression " .[1] " Section 18 does say predicates (i.e., the square brackets) are only used for indexing arrays. It says it is an SDE if the predicate value doesn't evaluate to an integer index. But it does not say that one cannot put the predicate brackets on scalars, context elements, relative path steps, etc., however, this can only be meaningful if the predicate value evaluates to 1. The grammar given for the expression language explicitly does allow " .[1] " to be syntactically allowed. The question is does the ".[1]" mean: (a) the whole array, so that ".[1]" refers to the first element of the array, so equivalent to "../a[1]" (b) the current element, so equivalent to "../a[dfdl:occursIndex()]" (c) SDE the expression is illegal as "." is never an array, and only arrays can be indexed with predicates. I believe (b) is consistent with XPath behavior for XML documents. An online XPath tool verifies this. That means the only way to refer to other array element items in the infoset is to reference the array itself via ../a[N]. This has the drawback of not being polymorphic - the array element name must be used as a path step. That may well be livable as a restriction. -mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Re: [DFDL-WG] DFDL4S Library - version 1.7.0 Released
by Steve Hanson 30 Jun '21

30 Jun '21

Hi Marcos Guidelines here https://github.com/OpenGridForum/DFDL/blob/master/docs/current/gwdi-experim… Best thing to do is follow the example of other experimental feature documents (but not the errata ones) for example: https://github.com/OpenGridForum/DFDL/blob/master/docs/current/gwde-dfdl-ex… Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: "Marcos Bento (external)" <Marcos.Bento(a)esa.int> To: "Steve Hanson" <smh(a)uk.ibm.com> Cc: "DFDL-WG" <dfdl-wg(a)ogf.org> Date: 29/06/2021 17:11 Subject: [EXTERNAL] Re: [DFDL-WG] DFDL4S Library - version 1.7.0 Released Hi Steve, This release indeed has developments related to action 287, but we only plan to put the new features into "action" with the upcoming version of S2G (which is the data visualiser that makes most use of DFDL4S). Up until the ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Steve, This release indeed has developments related to action 287, but we only plan to put the new features into "action" with the upcoming version of S2G (which is the data visualiser that makes most use of DFDL4S). Up until the next release of the S2G we should have some news regarding a preparing an 'experimental feature' document. I'm not sure exactly what is needed for such a proposal, so some guidance on that process would be welcome. --Marcos On Tue, Jun 29, 2021 at 11:58 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Hi Marcos Congratulations on your new release! Does it include the XPath wildcard support which the WG had been tracking under action 287? I think we agreed that it would appear as an 'experimental feature' at some point? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: "Marcos Bento (external)" <Marcos.Bento(a)esa.int> To: DFDL-WG <dfdl-wg(a)ogf.org> Date: 23/06/2021 17:28 Subject: [EXTERNAL] [DFDL-WG] DFDL4S Library - version 1.7.0 Released Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> As mentioned in one of the last DFDL calls, the latest DFDL4S release is available and enables support for dfdl:defineVariable and dfdl:newVariableInstance. Check out the complete list of improvements/bug fixes below. --Marcos This email contains graphics, so if you don't see them, view it in your browser New Release Announcement DFDL4S Library - version 1.7.0 Released Dear User, Please be informed that version 1.7.0 of the DFDL4S Library has been released and is available for download at the following URL: http://eop-cfi.esa.int/index.php/applications/dfdl4s/dfdl4s-download The Release Notes can be found here. The following is a summary of the changes this version. New Functionality and Improvements Added support for DFDL variables (dfdl:defineVariable and dfdl:newVariableInstance) Added support for Reporting capabilities (including CRC/Reed-Solomon) Added support to checking CRC and apply Reed-Solomon Error Correction Aligned implementation of occursCountKind=”fixed” with DFDL standard Added support for paths to array elements Bug Fixes Corrected faulty instantiation of DataSize objects (C++ only) Corrected trimming in StorageInFile detected during Transfer Frame to ISP transformation Corrected evaluation of non-integer values in length fields Best Regards The DFDL4S Support Team -- HE Space for ESA - European Space Agency Marcos Bento Mission Analysis & Software Engineer Mission Analysis Office System Support Division Earth Observation Projects Department Directorate of Earth Observation Programmes ESTEC Keplerlaan 1, PO Box 299 NL-2200 AG Noordwijk, The Netherlands marcos.bento(a)esa.int | www.esa.int T +31 71 565 3749 This message is intended only for the recipient(s) named above. It may contain proprietary information and/or protected content. Any unauthorised disclosure, use, retention or dissemination is prohibited. If you have received this e-mail in error, please notify the sender immediately. ESA applies appropriate organisational measures to protect personal data, in case of data privacy queries, please contact the ESA Data Protection Officer (dpo(a)esa.int) -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU This message is intended only for the recipient(s) named above. It may contain proprietary information and/or protected content. Any unauthorised disclosure, use, retention or dissemination is prohibited. If you have received this e-mail in error, please notify the sender immediately. ESA applies appropriate organisational measures to protect personal data, in case of data privacy queries, please contact the ESA Data Protection Officer (dpo(a)esa.int) Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL4S Library - version 1.7.0 Released
by Marcos Bento (external) 29 Jun '21

29 Jun '21

As mentioned in one of the last DFDL calls, the latest DFDL4S release is available and enables support for dfdl:defineVariable and dfdl:newVariableInstance. Check out the complete list of improvements/bug fixes below. --Marcos This email contains graphics, so if you don't see them, view it in your browser <http://eop-cfi.esa.int/index.php?option=com_acymailing&ctrl=archive&task=vi…> New Release Announcement DFDL4S Library - version 1.7.0 Released Dear User, Please be informed that version 1.7.0 of the DFDL4S Library has been released and is available for download at the following URL: http://eop-cfi.esa.int/index.php/applications/dfdl4s/dfdl4s-download The Release Notes can be found here <http://eop-cfi.esa.int/Repo/PUBLIC/DOCUMENTATION/APPLICATIONS/DFDL4S/dfdl4s…>. The following is a summary of the changes this version. *New Functionality and Improvements* - Added support for DFDL variables (dfdl:defineVariable and dfdl:newVariableInstance) - Added support for Reporting capabilities (including CRC/Reed-Solomon) - Added support to checking CRC and apply Reed-Solomon Error Correction - Aligned implementation of occursCountKind=”fixed” with DFDL standard - Added support for paths to array elements *Bug Fixes * - Corrected faulty instantiation of DataSize objects (C++ only) - Corrected trimming in StorageInFile detected during Transfer Frame to ISP transformation - Corrected evaluation of non-integer values in length fields Best Regards The DFDL4S Support Team [image: Powered by AcyMailing] <https://www.acyba.com/?utm_source=acymailing&utm_medium=e-mail&utm_content=…> -- *HE Space for ESA - European Space Agency* *Marcos Bento* Mission Analysis & Software Engineer Mission Analysis Office System Support Division Earth Observation Projects Department Directorate of Earth Observation Programmes *ESTEC* Keplerlaan 1, PO Box 299 NL-2200 AG Noordwijk, The Netherlands marcos.bento(a)esa.int | www.esa.int T +31 71 565 3749 This message is intended only for the recipient(s) named above. It may contain proprietary information and/or protected content. Any unauthorised disclosure, use, retention or dissemination is prohibited. If you have received this e-mail in error, please notify the sender immediately. ESA applies appropriate organisational measures to protect personal data, in case of data privacy queries, please contact the ESA Data Protection Officer (dpo(a)esa.int)

4 5

DFDL4S Library - version 1.7.0 Released
by michele.zundo＠esa.int 23 Jun '21

23 Jun '21

FYI > This email contains graphics, so if you don't see them, view it in your browser <http://eop-cfi.esa.int/index.php?option=com_acymailing&ctrl=archive&task=vi…> > > > New Release Announcement > DFDL4S Library - version 1.7.0 Released > Dear User, > > Please be informed that version 1.7.0 of the DFDL4S Library has been released and is available for download at the following URL: > > http://eop-cfi.esa.int/index.php/applications/dfdl4s/dfdl4s-download <http://eop-cfi.esa.int/index.php/applications/dfdl4s/dfdl4s-download> > The Release Notes can be found here <http://eop-cfi.esa.int/Repo/PUBLIC/DOCUMENTATION/APPLICATIONS/DFDL4S/dfdl4s…>. The following is a summary of the changes this version. > > > New Functionality and Improvements > > Added support for DFDL variables (dfdl:defineVariable and dfdl:newVariableInstance) > Added support for Reporting capabilities (including CRC/Reed-Solomon) > Added support to checking CRC and apply Reed-Solomon Error Correction > Aligned implementation of occursCountKind=”fixed” with DFDL standard > Added support for paths to array elements > Bug Fixes > > Corrected faulty instantiation of DataSize objects (C++ only) > Corrected trimming in StorageInFile detected during Transfer Frame to ISP transformation > Corrected evaluation of non-integer values in length fields > > Best Regards > > The DFDL4S Support Team > > > > <https://www.acyba.com/?utm_source=acymailing&utm_medium=e-mail&utm_content=…> ESA - European Space Agency Michele Zundo Head of Ground System Definition and Verification Office System Support Division Earth Observation Project Department Directorate of Earth Observation Programmes ESTEC Kepleerlaan 1, 2200AZ Noordwijk, Netherlands michele.zundo(a)esa.int <mailto:michele.zundo@esa.int> | www.esa.int <file:///x-msg/::11:www.esa.int> T. +31 71 565 3924 | F +31 71 565 3171 This message is intended only for the recipient(s) named above. It may contain proprietary information and/or protected content. Any unauthorised disclosure, use, retention or dissemination is prohibited. If you have received this e-mail in error, please notify the sender immediately. ESA applies appropriate organisational measures to protect personal data, in case of data privacy queries, please contact the ESA Data Protection Officer (dpo(a)esa.int)

1 0

Generic Schema-based transformation language: example based on DFDL capabilities
by Beckerle, Mike 17 Jun '21

17 Jun '21

DFDL isn't normally thought of as a data transformation language. Yet it has computed elements, hidden groups, expressions that can refer to elements in arrays using the index position within the current array (via the dfdl:occursIndex() function). These result in it having substantial data transformation capabilities. For a while I have said that I bet one can invert a matrix in DFDL. So I finally created an example that does so. It takes a representation of data as a pair of lists and creates a logical infoset that is a list of pairs. By keeping the infoset of the pair of lists in a DFDL hidden group, this DFDL schema effectively parses the file of data, and transforms one infoset into another that is entirely different in shape. https://github.com/OpenDFDL/examples/tree/master/pairsTransform The conclusion of this little experiment is that while this is possible for parsing, you can't invert the process perfectly in unparsing due to some DFDL v1.0 restrictions. Specifically, the fact that when dfdl:occursCountKind is 'expression' at unparse time the expression is NOT evaluated (DFDL Spec 16.1.4) prevents this example from allocating the representation infoset arrays inside a hidden group at unparse time. Previously, we had identified the lack of unparse-time evaluation of dfdl:choiceDispatchKey was a similar limitation on unparsing things in hidden groups, and this issue with arrays is directly analogous. I may do experiments in Daffodil to lift those restrictions just to play around with the idea. The concept here is not that DFDL should become a better transformation language. Rather, it is about "Schema-based Transformation" generally. That is to say, there is nothing in these transformation capabilities in DFDL that is about the format of data, per se, and no reason they cannot be lifted out of DFDL and used, for example, in ordinary XML schema to express XML transformations. This is quite interesting given that typical XML transformation languages such as XSLT and XQuery are both template/instance-document based, not schema based. Mike Beckerle | Principal Engineer [cid:eb186e2a-a515-4148-a5af-3d5f693737bd] mbeckerle(a)owlcyberdefense.com<mailto:bhummel@owlcyberdefense.com> P +1-781-330-0412

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 10 Jun '21

10 Jun '21

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 10 Jun '21

10 Jun '21

Please find agenda for next call on GitHub at https://github.com/OpenGridForum/DFDL/blob/master/calls/2021-06-10_DFDL-WG-… . Please note call is 1 hour earlier than normal. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

OGF DFDL Working Group Call
by Steve Hanson 08 Jun '21

08 Jun '21

1 0

example for extraEscapedCharacters
by Mike Beckerle 25 May '21

25 May '21

I've been searching our email history for the motivating example for extraEscapedCharacters property. I can't find it. Does anyone recall this? I vaguely recall a format where we had CSV-like data that already contained escaped characters like "foo\nbar\,baz,next,next" Where the \n is just part of the data, regardless of the fact that \ is also the escape character. Normal rules would remove the "\" since it is the escape character, and that would result in corrupting such data. But it makes no sense to have extraEscapedCharacter="n" since n is just a letter and could appear without the escape. It would be wrong to escape every letter n. So this example isn't sufficient. I see no way to cope with this particular case other than lengthKind='pattern' which is the usual way to cope with complexities like this. But I know there was a motivating example for extraEscapedCharacters property. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1