- dfdl-wg - lists.ogf.org

Confirmed: OGF DFDL Working Group call (17 Oct 16:00 GDT in https:/ibm.webex.com/meet/smh)
by Steve Hanson 16 Oct '19

16 Oct '19

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 16 Oct '19

16 Oct '19

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13597?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Tracker 333 - new bitOrder byteOrder combimation - dever to DFDL 2.0 ?
by Mike Beckerle 15 Oct '19

15 Oct '19

See https://redmine.ogf.org/issues/333 This tracker was to add the leastSignificantBitFirst bigEndian combination to the legal values. for bitOrder byteOrder I propose this is deferred to DFDL 2.0. While we have seen one case where this does appear to be used, it is not significant enough to warrant this change in DFDL v1.0. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

terminology corrections on initiatedContent descriptions
by Mike Beckerle 14 Oct '19

14 Oct '19

Section 14 For property initiatedContent The sentence "If the child is optional then it is deemed to have been found when its initiator has been found." should use consistent terminology of "known to exist" as follows "If the child is optional then it is known to exist when its initiator is found." Section 15 as well, also for initiatedContent "The branch is deemed to have been found when its initiator has been found." replace "The branch is known to exist when its initiator is found." I would categorize these corrections as "minor/typographical". Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Tracker 340 (Action 301) Experimental Features Conventions
by Mike Beckerle 11 Oct '19

11 Oct '19

I have uploaded a GWD-I document describing these. https://redmine.ogf.org/dmsf_files/13594?download= Please review. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

recent puzzlers on fn:count and fn:exists
by Mike Beckerle 09 Oct '19

09 Oct '19

These questions came up recently from daffodil users. I know what Daffodil will return here, but the spec is not clear to me as to whether that's the right answer or is even defined. Can an array count itself? Like this: <xs:element name="name" type="xs:string" maxOccurs="unbounded"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator test="{ fn:count(.) le 5 }" /> </xs:appinfo> </xs:annotation> </xs:element> The above can be reformulated by calling instead of fn:count, dfdl:occursIndex(), and that's definitely well defined in this case. But what does fn:count return in this case? Daffodil will return the same value as the occursIndex, because it will have speculatively attached the array element to the infoset at the time the fn:count function is being evaluated. But I don't know if that is the right answer or if we should simply say that fn:count is not defined in this case. Similarly, does an array element exist when it is being parsed? <xs:element name="name" type="xs:string" maxOccurs="unbounded"> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/"> <dfdl:discriminator test="{ fn:exists(.) }" /> </xs:appinfo> </xs:annotation> </xs:element> Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 2

Problem: simple format that is impossible to model
by Mike Beckerle 03 Oct '19

03 Oct '19

I have a dead-simple little format: data/data/data/data data/data/data/data it is lines of "/" separated strings. All elements are optional. I simply want this: data//data to round trip. For that to happen I need it to parse into <field>data</field><field></field><field>data</field> That is, I require that empty field element in the middle to be created and put into the infoset. I can find no way to do this. The strings have no initiator/terminator, so dfdl:emptyValueDelimiterPolicy is not relevant. All the elements are optional, so default values aren't relevant. The spec states: 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none'[12] <https://daffodil.apache.org/docs/dfdl/#_ftn12> then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, *otherwise nothing is added to the Infoset*. There are errata/actions to clarify wording here around dfdl:emptyValueDelimiterPolicy being in effect or not (because there is no initiator/terminator for it to use as opposed to the property in isolation just being 'none'). But that doesn't change anything about this issue. If this very simple format is not possible, then we need a property or new property enum value that makes it possible. Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 8

RFC 2119 word usage in DFDL Spec
by Mike Beckerle 30 Sep '19

30 Sep '19

In studying Tracker 304 https://redmine.ogf.org/issues/304 I have found that there are many uses of words like must and should, which are not specifically in accord with the notions of RFC 2119, because they are not involved in statements about requirements. For example, in section 2.3.1.1 Usually, the behavior of the unparser is symmetric to the behavior of the parser; however, there are cases where the DFDL schema will accept several equivalent representations for the same logical data. In this case it would be ambiguous which of these equivalent representations should be produced by the unparser. The DFDL standard contains representation properties which are used to eliminate this ambiguity. It is a schema definition error if a DFDL schema is being used to unparse data and there is any ambiguity about the representation. We can either ignore such issues, because the context doesn't require us to consider this a requirement statement, or reword so as to avoid RFC2119 terms. I am not sure it is worth changing this prose in all the places where this sort of thing happens. The word 'should' is less problematic than the word 'must' of which there are hundreds of occurrences. Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Re: [DFDL-WG] DFDL pointer & offset proposal
by Bradd Kadlecik 25 Sep '19

25 Sep '19

Properties for indirection: The following properties allow for data to be accessed in a non-sequential or non-contiguous format by using indirection values to locate the data contents. When parsing, the indirection value is used to access the contents in the data stream to populate the infoset. When unparsing, an indirection value is created to store the contents in the data stream when creating the data stream from the infoset. The indirection value does not exist in the infoset, but only in the data stream. The LeadingAlignment and TrailingAlignment regions apply to the indirection value in the data stream. The initiator and terminator as well as all regions between the LeftFraming and RightFraming regions are applied to the data content accessed through the indirection value. Property Name | Description ----------------------+-------------------------------------------------------- indirectKind | Enum | | Specifies the type of indirection value used to | access the the contents in the data stream. | | Valid values ‘pointer’, ‘offset’, ‘none’. | | ‘pointer’ means a numeric value gives an | implementation-dependent starting position of the | content. | | ‘offset’ means a numeric value that when added to | indirectBase gives the starting position of the | content. | | ‘none’ means no indirection is used and is the | default value. | | Annotation: dfdl:choice, dfdl:element, dfdl:group, | dfdl:simpleType, dfdl:sequence ----------------------+-------------------------------------------------------- indirectType | QName | | Name of a simple type derived from xs:integer or any | subtype of it. | | This type, with its DFDL annotations specifies the | representation of the indirection value. | | It is a schema definition error if the xs:simpleType | specifies any of: | · dfdl:lengthKind 'delimited', 'endOfParent', or | 'pattern' | · dfdl:lengthKind 'explicit' where length is an | expression | · dfdl:outputValueCalc | · dfdl:initiator or dfdl:terminator other than | empty string | · dfdl:alignment other than '1' | · dfdl:leadingSkip or dfdl:trailingSkip other | than '0'. | | Annotation: dfdl:choice, dfdl:element, dfdl:group, | dfdl:simpleType, dfdl:sequence ----------------------+-------------------------------------------------------- indirectBase | String | | Provides the relative path to a prior element upon | which the offset is based. | | Required only when dfdl:indirectKind is ‘offset’. | | Annotation: dfdl:choice, dfdl:element, dfdl:group, | dfdl:simpleType, dfdl:sequence ----------------------+-------------------------------------------------------- indirectEmptyValue | Integer | | Specifies the indirection value that indicates when | the data content is empty. If this property is | specified, all underlying elements must contain a | default value specification or it is a schema | definition error. | | Annotation: dfdl:choice, dfdl:element, dfdl:group, | dfdl:simpleType, dfdl:sequence The following example illustrates how to describe a pointer to a null-terminated string (common in languages like C). <xs:element name="myString" type="xs:string" dfdl:lengthKind="delimited" dfdl:encoding="UTF-8" dfdl:terminator="%NUL;" dfdl:indirectKind="pointer" dfdl:indirectType="ptr32_t" dfdl:indirectEmptyValue="0" default=""/> <xs:simpleType name="ptr32_t" dfdl:representation="binary" dfdl:lengthKind="explicit" dfdl:length="4" dfdl:byteOrder="bigEndian"> <xs:restriction base="integer"/> </xs:simpleType> The data stream may look like the following for a string value of “test”. Location Hex values 00000000 0012A000 0012A000 7465737400 t e s t The following example defines an array of 3 pointers to complex elements defined by ns0:myStruct. <xs:element name="myArray" type="ns0:myStruct" dfdl:lengthKind="implicit" dfdl:indirectKind="pointer" dfdl:indirectType="ptr32_t" dfdl:indirectEmptyValue="0" minOccurs="3" maxOccurs="3" dfdl:occursCountKind="fixed"/> <xs:simpleType name="ptr32_t" dfdl:representation="binary" dfdl:lengthKind="explicit" dfdl:length="4" dfdl:byteOrder="bigEndian"> <xs:restriction base="integer"/> </xs:simpleType> The data stream may look like the following with the contents of each occurrence in a different location and the contents of the 2nd occurrence being empty. Location Hex values 00000000 00147000 00000000 00146000 00146000 ... 00147000 ... The following example defines an offset to a 6 byte hexBinary value with the offset being calculated from the start of the current element. <xs:element name="myData" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="6" dfdl:indirectKind="offset" dfdl:indirectType="uint" dfdl:indirectBase="."/> <xs:simpleType name="uint" dfdl:representation="binary" dfdl:lengthKind="explicit" dfdl:length="4" dfdl:byteOrder="bigEndian"> <xs:restriction base="unsignedInt"/> </xs:simpleType> The data stream may look like the following for a hexBinary value of “123456789ABC”. Location Hex values 00000100 00000108 00000208 123456789ABC Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States

1 0

OGF DFDL WG Call on 2018-09-26 - cancelled
by Steve Hanson 25 Sep '19

25 Sep '19

Apologies but I have a deadline coming up and not able to do anything DFDL WG related right now. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0