November 2020 - dfdl-wg - lists.ogf.org

Action 315: fn:count(.), fn:exists(.)
by Mike Beckerle 10 Dec '20

10 Dec '20

There are 4 functions which operate on the infoset and it is unclear their behavior depending on when they are evaluated during parse/unparse. The 4 functions are fn:count, fn:exists, fn:exactly-one, and fn:empty. The behavior when unparsing is less problematic, because one could simply require the infoset nodes being referenced to be fully-constructed before these functions are allowed to evaluate. However, when parsing the behavior is more subtle, and unparsing may want to be made consistent with decisions about behavior for parsing. Our call minutes about this action item suggest reviewing the known-to-exist and known-not-to-exist definitions to see whether these function definitions should be defined in terms of that. I have reviewed those sections, and so far I'm not sure they will contribute. The general problem is this, in terms of fn:count(path). The path is to an infoset node or an array of occurrences that is currently being parsed. It is possible that the status of known-to-exist or not is simply not well known at the point the expression is being evaluated. The answer to fn:count(path) wants to always be the same as if the infoset were fully constructed at the time the expression is evaluated. As evaluation may occur during parsing, it is just not defined if the evaluation of the expression itself determines whether the item itself is known to exist or not. Ex: <xs:element name="outerArray" maxOccurs="unbounded"> <xs:complexType> <xs:sequence> <xs:element name="innerArray" maxOccurs="unbounded"> <xs:complexType> <xs:sequence> <xs:element name="count" type="xs:int" dfdl:inputValueCalc='{ fn:count(../..) }'/> .... </xs:sequence> </x:complexType> </xs:element> </xs:sequence> </xs:complexTYpe> </xs:element> In the above, we see that fn:count has as argument a relative path to the array element named "outerArray". There are a few observations here. 1) If we define fn:count in this case to actually have anything to do with the current number of array elements in outerArray, then we will have tightly constrained implementations to a very sequential notion of parsing. The notion of "current" state of the array implies an algorithm where the number of current occurrences is changing. E.g., we would preclude an implementation that knows the length of all outerArray elements from parsing all the children simultaneously in parallel, or at minimum make this quite hard to achieve because each parallel computation would have to somehow simulate the right "current number" of occurrences. 2) The question arises of fn:count(../..) vs. fn:count(../../../outerArray), vs. fn:count(../../../outerArray[i]) where i is the index of the enclosing parent outerArray instance that contains this calculation. Arguably, fn:count(../..) could be considered equivalent to fn:count(../../../outerArray[i]), both of which seem like they should always return '1' since the count of number of instances of a single index point, single node, is 1. 3) Arguably, fn:count(path) could be illegal whenever the path is to an enclosing element. We could simply define this usage to be illegal. I cannot come up with any reason to actually need this functionality. When parsing we could require the path argument to be to pre-existing part of the infoset, and when unparsing it would have to be to either pre-existing or later parts of the infoset, but specifically not the current infoset elements. If we make this an SDE, then this would seem to be the conservative design point which preserves our ability to assign a future meaning to this usage, should a need arise. My recommendation: Expressions evaluated as part of an element parsing or unparsing cannot refer to the count or existence of the current element occurrence being parsed, nor any enclosing element occurrence, nor any enclosing array element. This would seem to rule out any use of absolute paths in arguments to fn:count, because the root element is not (necessarily) known-to-exist until the entire parse completes successfully. Yet clearly we want to be able to refer to the fn:count of a prior sibling array, and that reference should be able to use either a relative or absolute path. So it's not that the argument path "passes through" a node that may or may not exist, but that it ends on one that the existence or not of which doesn't depend on the existence or not of the current node. I'm a bit uncertain of good language to express this constraint on what the path argument is allowed to refer to, but the notion is one of a sort of circular definition; hence, it's a schema definition error. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 3

OGF DFDL WG Call 2020-10-26 cancelled
by Steve Hanson 25 Nov '20

25 Nov '20

Thanksgiving in USA Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group Call
by Steve Hanson 25 Nov '20

25 Nov '20

1 0

Trackers all moved to github
by Mike Beckerle 23 Nov '20

23 Nov '20

All open trackers on DFDL v1.0 have been converted to public comments (there were 2) All open trackers on DFDL v2.0 have been converted to issues here: https://github.com/OpenGridForum/DFDL/issues with DFDL 2.0 label. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

[ANNOUNCE] Apache Daffodil (incubating) 3.0.0 Released
by Mike Beckerle 23 Nov '20

23 Nov '20

The Apache Daffodil (incubating) community is pleased to announce the release of version 3.0.0. Notable changes in this release include SAX support, streaming, reduced memory usage, improved diagnostics, and bug fixes. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/3.0.0/ Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team ===== *Disclaimer* Apache Daffodil is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 12 Nov '20

12 Nov '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 12 Nov '20

12 Nov '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13628?download= Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday

1 0

Re: [DFDL-WG] DFDL pointer & offset proposal
by Bradd Kadlecik 11 Nov '20

11 Nov '20

I've updated the proposal for the following issues: 1) Undefined term "underlying elements" used in indirectionUnusedValue description 2) Undefined term "prior element" used in indirectionBase description 3) Grammar needs to have indirection elements follow document element (See attached file: DFDL_Indirection_v6.docx) Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 07/24/2020 01:08 PM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal "Actually, the target of the indirection is just like an element or model-group, except without alignment regions. Ugly to have to repeat the whole grammar for that. ". Correct, and I haven't. My changes are for the indirection value, not the target. We have to handle 'indirectionUnusedValue' and reflect that initiator & terminator do not apply. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 24/07/2020 13:59 Subject: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal Actually, the target of the indirection is just like an element or model-group, except without alignment regions. Ugly to have to repeat the whole grammar for that. Perhaps we can get away with just documenting that the alignment regions must be zero length when target of an indirection? I did not try to put this modified grammar into my commented copy of the proposal (attached). The "connection" to the indirect pointer or offset is not something that the grammar can express. Attached * my review (with new comments that start with "V4:"), * my first cut of a DFDL schema for TIFF v6.0, * a copy of the TIFF v6.0 spec. I wasn't actually able to pull off TIFF. Comments in the schema discuss why, which is due to TIFF being a linked list, and I'm not able to terminate the chain. There's maybe a trick using negative offsets, but I don't want to consider that yet, as those are entirely problematic. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Fri, Jul 24, 2020 at 7:03 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Bradd, Mike - as discussed on WG call please review my version of the grammar for next call. The key issue is how to link the target of the indirection into the main grammar, as it is dis-contiguous. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 23/07/2020 14:52 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I incorporated some of your grammar changes but had a few modifications of my own we can discuss further in case I'm not understanding some of the subtleties. I think there may be a case for a nil value with indirection however as well. I've updated the document to include the changes as version 4. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 07/10/2020 08:11 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I think 'indirect' in property names should be 'indirection', so I will assume that in what follows. I am not happy with 'indirectionEmptyValue' or 'indirectionMissingValue'. Both 'empty' and 'missing' are concepts with well-defined meanings in DFDL. How about 'indirectionInactiveValue' or better 'indirectionUnusedValue'? My stab at an updated grammar below (changes in blue). There is no concept of an empty indirection value or a nil indirection value, so we don't need those clauses. I also think we can simplify the EnclosedElement construct (changes in red) Bradd I know you were struggling with how to link the target of the indirection into the main grammar, as it is dis-contiguous. I think we can achieve that by incorporating the target(s) into the Document as a whole. If we think that the concept of an indirection can only ever happen inside a complex type, then we could move IndirectionRep from SimpleElement/ComplexElement to EnclosedElement. I don't think it is worth getting more specific with IndirectElement. Specifically, it has no LeadingAlignment or TrailingAlignment, but to reflect that requires duplicating most of the rest of the grammar. You have not disallowed lengthKind 'prefixed' for the 'indirectionType', so that is included in the grammar. Productions --------------------------------------------------------------------------------------- Document = DocumentElement IndirectElement* DocumentElement = SimpleElement | ComplexElement SimpleElement = SimpleLiteralNilElementRep | SimpleEmptyElementRep | SimpleNormalRep | IndirectionRep SimpleEnclosedElement = SimpleElement | AbsentElementRep ComplexElement = ComplexLiteralNilElementRep | ComplexNormalRep | ComplexEmptyElementRep | IndirectionRep ComplexEnclosedElement = ComplexElement | AbsentElementRep EnclosedElement = SimpleElement | ComplexElement | AbsentElementRep IndirectElement = SimpleElement | ComplexElement --------------------------------------------------------------------------------------- AbsentElementRep = Absent --------------------------------------------------------------------------------------- IndirectionRep = IndirectionLeftFraming PrefixLength IndirectionContent IndirectionRightFraming IndirectionLeftFraming = LeadingAlignment IndirectionRightFraming = TrailingAlignment IndirectionContent = LeftPadding [ IndirectionValue | IndirectionUnusedValue ] RightPadOrFill IndirectionValue = SimpleValue --------------------------------------------------------------------------------------- SimpleEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming ComplexEmptyElementRep = EmptyElementLeftFraming EmptyElementRightFraming EmptyElementLeftFraming = LeadingAlignment EmptyElementInitiator PrefixLength EmptyElementRightFraming = EmptyElementTerminator TrailingAlignment --------------------------------------------------------------------------------------- SimpleLiteralNilElementRep = NilElementLeftFraming [NilLiteralCharacters | NilElementLiteralContent] NilElementRightFraming ComplexLiteralNilElementRep = NilElementLeftFraming NilLiteralValue NilElementRightFraming NilElementLeftFraming = LeadingAlignment NilElementInitiator PrefixLength NilElementRightFraming = NilElementTerminator TrailingAlignment NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill --------------------------------------------------------------------------------------- SimpleNormalRep = LeftFraming PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming PrefixLength ComplexContent ElementUnused RightFraming LeftFraming = LeadingAlignment Initiator RightFraming = Terminator TrailingAlignment PrefixLength = SimpleContent | PrefixPrefixLength SimpleContent PrefixPrefixLength = SimpleContent SimpleContent = LeftPadding [ NilLogicalValue | SimpleValue ] RightPadOrFill ComplexContent = Sequence | Choice --------------------------------------------------------------------------------------- Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedContent [ Separator EnclosedContent ]* PostfixSeparator ] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedContent ] ChoiceUnused EnclosedContent = [ EnclosedElement | Array | Sequence | Choice ] Array = [ EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] StopValue = SimpleElement --------------------------------------------------------------------------------------- LeadingAlignment = LeadingSkip AlignmentFill TrailingAlignment = TrailingSkip RightPadOrFill = RightPadding | RightFill | RightPadding RightFill Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 09/07/2020 15:09 Subject: Re: [EXTERNAL] Re: [DFDL-WG] DFDL pointer & offset proposal I have updated the document proposal with proposed DFDL grammar changes: [attachment "DFDL_Indirection_v3.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU [attachment "TIFF-v6.pdf" deleted by Bradd Kadlecik/Poughkeepsie/IBM] [attachment "DFDL_Indirection_v4_mjb_comments.docx" deleted by Bradd Kadlecik/Poughkeepsie/IBM] [attachment "tiff.dfdl.xsd" deleted by Bradd Kadlecik/Poughkeepsie/IBM]

1 0

OGF DFDL Working Group Call
by Steve Hanson 11 Nov '20

11 Nov '20

1 0

Action 320: Migration to Github from redmine trackers
by Mike Beckerle 05 Nov '20

05 Nov '20

We have 3 trackers left on redmine for DFDL v1.0 version. I propose these be re-created as github issues as public comments, and they be closed on redmine. The 16 redmine issues targeted at DFDL v2.0 version should become issue trackers on the DFDL project repository. Note that this is a separate pool of trackers from the public comments, which are trackers on the editor github repo, not the DFDL repo. That would then end use of redmine trackers. Going forward, if a public comment is resolved by being tracked for consideration in DFDL v2.0, then the public comment would be closed and a DFDL project issue opened reflecting the content of the public comment. As for the documents, I am not yet proposing a migration strategy for that, nor am I clear on what the replacement on github is for redmine "news". The trackers, documents, and news appear to be the only redmine features we were using. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0