- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] Byte-order-marks - was: Re: Part 1 - Re: Action 307 - Demonstrate implementation interoperability
by Steve Hanson 05 Apr '19

05 Apr '19

Agreed that BOM support would be dropped from DFDL 1.0 via erratum. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/04/2019 10:44 Subject: Re: Byte-order-marks - was: Re: Part 1 - Re: [DFDL-WG] Action 307 - Demonstrate implementation interoperability Hi Mike Found the thread on BOMs, so ignore my earlier email. I am leaning towards deprecation on the following grounds: - Only one customer that I know of ever requested BOM processing for non-XML data (in 2010, for MRM, before IBM DFDL available) - BOM processing only applies to the message as a whole, not to any embedded Unicode fragments, so support is selective anyway - It is possible to model an optional BOM and use it to set a user-defined encoding variable which is then used by the rest of the schema I have a schema that models UTF16 BOM and it successfully parses and unparses the 3 variants fine (no BOM present, BOM for BE present, BOM for LE present). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 16/10/2018 14:57 Subject: Byte-order-marks - was: Re: Part 1 - Re: [DFDL-WG] Action 307 - Demonstrate implementation interoperability To avoid changing behavior then, we will probably need a property to turn on/off the BOM behavior which strips/generates BOM. All implementations of DFDL that implement text (IBM, and Daffodil) currently do not treat BOMs specially currently, neither stripping them nor generating them. I suggest we need byteOrderMarkPolicy="use/ignore", with "ignore" meaning that the BOM is just treated as a character. Implementation of byteOrderMarkPolicy="use" would be an optional feature of DFDL. That way both IBM DFDL and Daffodil can be compliant without implementing this. (I'd like everything we've collectively been able to live without thus far, that isn't needed for interoperability testing, to ultimately get onto the optional features list.) Or we can simply deprecate the functionality in the spec and say BOMs must be modeled, and just strike the stuff from the next draft, and post an example on how to model BOMs. It sounds heavy handed, but nobody has asked for this feature (on the Daffodil project), and it was put into the DFDL spec way back in the early days when we expected BOMs to be popular, but they never caught on. However, I'll acknowledge that IBM would be in a better position to decide whether this feature is needed, given more users in Asia and other places where UTF-16 may be more popular. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Oct 10, 2018 at 5:24 AM Steve Hanson <smh(a)uk.ibm.com> wrote: I think the main thing with BOMs is that when support is added by an implementation, it should not break existing behaviour for a document that starts with a BOM. So if a user had a schema that explicitly modelled the BOM, or was treating BOM as a character so it appeared in the infoset, then a BOM aware implementation should not suddenly change that. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/10/2018 14:35 Subject: Re: Part 1 - Re: [DFDL-WG] Action 307 - Demonstrate implementation interoperability Very helpful Steve H., , thanks. re: UTF-8 and BOM, for UTF-8, the BOM can be viewed as "just a character", same as it is in UTF-16BE and UTF-16LE. Only utf-16 unadorned has to actually look at, and in theory strip the BOM if found. Nobody is implementing this, and it's not clear it matters much. Today I know that Daffodil just treats UTF-16 as meaning UTF-16BE. Hence, I suggest we consider just making BOM processing optional in DFDL and also make utf-16 (unadorned) optional - takes one small issue off of being "standard compliant". This leaves the question of what does "utf-16" unadorned do, and the answer I think is supposed to be guided by BOM, but if that is unimplemented then the behavior is "implementation defined" i.e., non-portable. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Oct 9, 2018 at 6:25 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Mike, responses in-line below. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 03/10/2018 23:00 Subject: Part 1 - Re: [DFDL-WG] Action 307 - Demonstrate implementation interoperability I'm going to reply to this in a few parts. With respect to: - dfdl:binaryBooleanTrueRep with value empty string - dfdl:assert on global element and simple type - dfdl:discriminator on global element and simple type - Multiple xs:appinfo elements within each xs:annotation element I think these are minor non-compliances with the DFDL spec, and for interoperability testing we can just revise schemas under test to not use these constructs. SMH: Agree. With respect to: - When parsing, the distinction between an element being 'missing', having an 'empty representation' and having an 'absent representation', is not in accordance with the specification. I think time will tell here, that is, there's nothing we can anticipate having to do because of this as yet. If this non-compliance does not cause interoperability problems for realistic and published DFDL schemas then I wouldn't worry about it. Like IBM DFDL, Daffodil does not implement default values during parsing, and that's a likely area where this issue of missing/empty/absent has effect on behavior. It is quite possible that despite this lack of conformance to the DFDL spec., interoperability testing would be successful. SMH: IBM DFDL gives a runtime SDE when parsing if it a zero-length representation is found for an occurrence AND the element has a default value That prevents a behaviour change when support for default values when parsing is implemented. Suggest Daffodil does same if it does not do so already. With that in place, I think we are ok. With respect to: - When encoding is 'UTF-8' or 'UTF-16', byte order marks are not processed Daffodil also does not implement byte-order-mark processing. We can dodge this issue entirely if we make the UTF-16 charset (specifically UTF-16 without the BE or LE suffix) encoding an optional DFDL feature. That effectively makes byte-order-mark processing also an optional feature, and then both IBM DFDL and Daffodil would be compliant and interoperable. SMH: UTF8 can also have a BOM so that does not solve the problem entirely. Needs some more thought. With respect to: - dfdl:encodingErrorPolicy "replace" This one is harder. Daffodil doesn't implement encodingErrorPolicy='error' so we have no common ground here for interoperability testing. Making the entire encodingErrorPolicy property optional - meaning behavior in the presence of encoding errors is implementation specified - that's super undesirable to me. I suspect that implementing encodingErrorPolicy 'error' will be necessary for Daffodil. If we do that then IBM DFDL can continue to document the lack of this missing required feature of DFDL, or we can make 'replace' optional in the spec., or IBM could implement 'replace'. SMH: This is top of the list of missing features for IBM DFDL. I have asked in the past if this could be added as it's technically a regression when compared to IIB's older text/binary parser (MRM). I will ask again. Additional Non-portable/Problematic Required Features I did an analysis of all DFDL properties, and those that must be implemented to meet the minimum functionality that is not optional for a DFDL implementation per Section 21 of the spec. Starting from a list of all DFDL properties, I eliminated any specific to unparsing, and then any that aren't relevant given something optional in Section 21. Here are the remaining properties I found. Restrictions on what values of these properties are mentioned where their full functionality is considered optional: length - integer values only lengthKind - explicit, implicit only lengthUnits - bytes or characters only representation - binary only byteOrder alignment - number or 'implicit' alignmentUnits - bytes only fillByte leadingSkip trailingSkip encoding - 'UTF-8'', 'UTF-16', 'UTF-16BE', 'UTF-16LE', 'ASCII', and 'ISO-8859-1' encodingErrorPolicy - (Already discussed above, so not further discussed in this section) utf16Width - because UTF-16 is allowed for encoding, 'variable' is problematic. textPadKind textTrimKind textStringJustification textStringPadCharacter binaryNumberRep - binary only binaryFloatRep - ieee only binaryBooleanTrueRep binaryBooleanFalseRep - IBM DFDL doesn't allow empty string for this. (Minor.) binaryCalendarRep - binarySeconds, binaryMillseconds only binaryCalendarEpoch occursCountKind - fixed only occursCount - integer only Looking at this list, there is only 1 additional issue to portability/interoperability this raises today given what I know about the Daffodil implementation and the IBM implementation. Issue: utf16Width='variable' This issue can be addressed with a minor change to the DFDL specification. When the type is xs:string, lengthUnits is 'characters', then the length in characters should take surrogate-pairs found in the UTF-16 data, and count those as occupying 1 character. This utf16Width='variable' feature of DFDL should be optional, as Java JVM-based implementations will find this extremely difficult to support, since JVM standard string representations cannot represent individual characters with code points greater than 0xFFFF occupying 1 location in a string. Daffodil does not implement this 'variable' behavior, and we have no good pathway to do so. Hence, prefer to change the DFDL spec to make this 'variable' optional. Only 'fixed' would be required. I could support deprecating the whole property even. SMH: This is already captured by action 290, which is waiting for me to do some tests with IBM DFDL which claims to have implemented this. Issue: lengthUnits='characters' and variable-width charset encodings I believe this is required behavior. I also believe the lack of support for this is missing from IBM's list of non-compliances. I recall discussion that IBM DFDL requires a fixed width encoding in this situation where lengthUnits is 'characters'. (Please correct me if I am wrong.) I suggest making this combination an optional feature of the DFDL spec., would resolve the issue. This complex feature was added to support naive data format conversions where data originally had ascii encoding and lengthUnits 'bytes' is changed to 'utf-8' with lengthUnits 'characters'. This is a rational way to modernize a data format adding internationalization capability. It however requires a significant change in runtime behavior because utf-8 characters occupy between 1 and 4 bytes per character. SMH: IBM DFDL certainly supports lengthUnits="characters" and encoding="UTF-8", which is an example of this. Optional Features that are Partially Implemented The bigger set of concerns for interoperability is the behavior of a DFDL processor for features that are optional by strict interpretation of Section 21, but are implemented by a specific DFDL implementation, but the implementation is partial. This is the subject of other email messages however. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Sep 11, 2018 at 11:33 AM Steve Hanson <smh(a)uk.ibm.com> wrote: Action 307 was raised recently and first task is for implementations to identify which core spec behaviour is not implemented. IBM DFDL The following is the list of DFDL 1.0 spec core features that IBM DFDL does not yet implement. - dfdl:encodingErrorPolicy "replace" - dfdl:binaryBooleanTrueRep with value empty string - dfdl:assert on global element and simple type - dfdl:discriminator on global element and simple type - Multiple xs:appinfo elements within each xs:annotation element - When parsing, the distinction between an element being 'missing', having an 'empty representation' and having an 'absent representation', is not in accordance with the specification. - When encoding is 'UTF-8' or 'UTF-16', byte order marks are not processed The above lists are derived from information at https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… and are those that apply to core spec features. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL Working Group call
by Steve Hanson 05 Apr '19

05 Apr '19

1 0

Agenda for tomorrow's OGF DFDL WG Call
by Steve Hanson 04 Apr '19

04 Apr '19

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13586?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Re: [DFDL-WG] Latest OGF DFDL WG Call Minutes
by Steve Hanson 03 Apr '19

03 Apr '19

Hi Bradd I have a few questions please ... inline below. I'm still going to need a real worked example, starting with some actual data and its schema, what it appears like after paring in the infoset, and how unparsing lays it back out again. Are you able to make the rescheduled call this Friday? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 27/02/2019 22:22 Subject: Re: Latest OGF DFDL WG Call Minutes Regarding proposal for offsets and pointers: The following are the properties to be defined: SMH: I was expecting to see these properties only on dfdl:element, especially as you say '...the element contents... ? indirectKind Enum Valid values 'pointer', 'offset' (there is also a thought of objectId or refId for handling BSON but not at this time) Specifies the type of indirection used to access the element contents in the data stream. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I am missing the distinction between offset and pointer. Is one relative to current position and the other relative to start of bitstream? SMH: In earlier DFDL proposals for offset support, we had used the term to refer to a property to be used to establish position of the current element instead of assuming the current element followed straight after the previous one. It would allow sparse modelling of fixed structures. The offset could be relative to start of bitstream or some other point. I don't think that's what you mean when you say 'offset' so I will refer to your new concept as 'pointer'. SMH: Assuming that indirectKind is a normal DFDL property, it can be in scope. It would therefore need to have an enum 'None' which would be the default used in most schemas. indirectLength Non-negative Integer or DFDL expression Specifies the length of the indirection in units according to the indirectUnits property. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group indirectUnits Enum Valid values 'bytes','bits' Specifies the units to be used for reading or writing the indirection according to indirectLength. The default value is 'bytes'. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I think a better approach is to provide a property dfdl:indirectType, instead of indirectLength/indirectUnits, which refers to a simple type (not element) that carries its own lengthKind, length & lengthUnits properties. Similar idea to dfdl:prefixLengthType. That allows a lot of flexibility on how the pointer can appear. offsetBase non-empty string containing an absolute or relative XPath expression for the base element. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group The proposal would be to have the contents of the indirection be after the LeadingAlignment and before the TrailingAlignment. This would mean the aligment and skip factors apply to the indirection values in the data stream instead of the the contents of the indirection. SMH: Agree. This also then means in an array element, each element has its own indirection value (pointer or offset) and the alignment,skip factors then apply to each of these indirection values. SMH: Do you mean '...each occurrence...' ? It would be thought that the indirection values apply only to the data stream and not the infoset. During parse when the infoset is populated from the data stream, the indirection values are replaced by the contents. During unparse, the indirection values don't exist in the infoset and are created during the writing to/creation of the data stream. SMH: I agree that the indirection should be a purely physical thing, but I am not clear how the value is filled in when unparsing. Where does the value come from? outputValueCalc? Or maybe it's not needed when unparsing, and the data is always contiguous? For pointers, a null pointer creates the scenario of either nil representation or empty representation depending on whether or not nillable is defined as true. Unless default values (or 0 occurrence) are defined for all underlying content, then this is a processing error. During unparse, the only scenario in which a null pointer would be created is for a nil representation. SMH: This needs more thought. The nil & default properties apply to the contents of the indirection, not to the pointer. If you want to give a nil semantic to the pointer value itself, then that would require a new enum for dfdl:nilKind. I don't see why a pointer value 0 can't be treated like any other indirection value. A missing pointer is an error - it must be present - there is no way to control optionality because minOccurs/maxOccurs apply to the contents. (Alternatively, if you want the concepts of nil, default, occurs to apply to the indirect value, then dfdl:indirectType could point at an element instead of a simple type - but that seems way too over engineered). Examples: The following is the definition for the address of a null-terminated string in which the string address may be NULL as indicated by a nillable value of true: <xs:element name="myString" type="xs:string" dfdl:lengthKind="delimited" dfdl:encoding="UTF-8" dfdl:terminator="%NUL;" dfdl:indirectKind="pointer" dfdl:indirectLength="8" dfdl:indirectUnits="bytes" nillable="true" /> The following is the definition for an array of three 4 byte addresses of a complex element defined by ns0:myStruct: <xs:element name="myArray" type="ns0:myStruct" dfdl:lengthKind="implicit" dfdl:indirectKind="pointer" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" minOccurs="3" maxOccurs="3" dfdl:occursCountKind="fixed" /> The following is the definition for a 4 byte offset to a 100 byte hexBinary value from the start of the parent element definition: <xs:element name="myData" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="100" dfdl:lengthUnits="bytes" dfdl:indirectKind="offset" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" dfdl:offsetBase=".." /> SMH: I don't see how unparsing works. What provides the value? The proposal would also allow for the following optional item but I don't currently see a need for this: dfdl:offsetKind with values "startToStart" or "endToStart" - indicates if the offset is from the start of the base element or the end of the base element. I tried getting this out before my vacation so it might take a little bit to respond for issues. Thank you for your time. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 02/07/2019 12:32 PM Subject: Latest OGF DFDL WG Call Minutes Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Rescheduled: OGF DFDL Working Group call (5 Apr 16:00 GDT)
by Steve Hanson 03 Apr '19

03 Apr '19

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 306 - IBM DFDL behaviour when parsing empty strings
by Steve Hanson 03 Apr '19

03 Apr '19

306 Confirm IBM DFDL behaviour when parsing empty strings (Steve) 7/8: IBM DFDL has not fully implemented the behaviour changes arising from action 140 with respect to empty string elements. Daffodil is about to do so. IBM DFDL users have complained about lack of defaults when parsing but other than that appear happy. Are the rules in the spec for empty strings over complicated? Steve to document the behaviour for IBM DFDL to inform the discussion. ... 1/11: In progress - there are a lot of subtle scenarios 15/11: Not discussed ... 7/2/19: No further progress Some progress :) 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the Infoset. IBM DFDL behaviour: Required. IBM DFDL does not implement default values when parsing, so an empty occurrence with a default value gives an SDE (to prevent backtracking). An empty occurrence with no default gives a Processing Error. If you need to add an empty string to the infoset, you can add default=""(when default values implemented, of course). Optional. IBM DFDL adds nothing to the infoset regardless of presence of initiator and/or terminator. No way to get empty string into the infoset. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then an empty string (type xs:string), empty hexBinary (type xs:hexBinary), or default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) IBM DFDL behaviour: Required. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). Optional. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). So ... The spec today is consistent in one way, in that for both complex & string elements a) a required empty occurrence always adds to the infoset; & b) an optional empty occurrence adds to the infoset if initiator/terminator present; & c) an optional empty occurrence does not add to the infoset if no initiator/terminator present. If the simple string behaviour was to change to match IBM DFDL then that consistency is lost, but the string behaviour then matches that for other simple types. Section 9.4.2.2 disappears as the behaviour is same as 9.4.2.1. Section 9.4.2.3 becomes as below. We lose the ability to get an empty string into the infoset for an optional string with initiator/terminator. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then a default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) We also need to be sure that any other implementations have not yet implemented the current spec behaviour. Need to check with DFDL4S and IBM TPF. To be discussed on next WG call ... Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 307 - Demonstrate implementation interoperability
by Steve Hanson 03 Apr '19

03 Apr '19

Action 307 was raised recently and first task is for implementations to identify which core spec behaviour is not implemented. IBM DFDL The following is the list of DFDL 1.0 spec core features that IBM DFDL does not yet implement. - dfdl:encodingErrorPolicy "replace" - dfdl:binaryBooleanTrueRep with value empty string - dfdl:assert on global element and simple type - dfdl:discriminator on global element and simple type - Multiple xs:appinfo elements within each xs:annotation element - When parsing, the distinction between an element being 'missing', having an 'empty representation' and having an 'absent representation', is not in accordance with the specification. - When encoding is 'UTF-8' or 'UTF-16', byte order marks are not processed The above lists are derived from information at https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… and are those that apply to core spec features. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 6

Re: [DFDL-WG] Part 2 - Re: Action 307 - Demonstrate implementation interoperability - BOM
by Steve Hanson 03 Apr '19

03 Apr '19

Ignore this, see other thread Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 27/03/2019 16:15 Subject: Re: Part 2 - Re: Action 307 - Demonstrate implementation interoperability - BOM Hi Mike The outstanding item to resolve is what to about BOMs. 307 Demonstrate implementation interoperability (Steve, Mike) 4/9: Need to make sure that DFDL spec section 21 lists a correct set of optional features, the implication being that Daffodil and IBM DFDL (and any other minimally conforming implementation) correctly implement the remaining required features. First step - see if there are any obvious omissions. 16/10: Steve sent email stating IBM DFDL's missing core features and non-compliant behaviour, and Mike responded. Discussion continuing via two separate email threads. Part 1 for core features. Part 2 for optional features. For the core features, agreed that the following needs to happen: 1) IBM adds encodingErrorPolicy='replace' 2) Daffodil adds encodingErrorPolicy='error' 3) Daffodil ensures that, if not implementing default/fixed when parsing, it gives an SDE if a required occurrence has empty rep and element has default/fixed set. 4) A position is agreed on BOM handling - ongoing via email. 1/11: Just BOM to conclude on from the above list 15/11: Not discussed 29/11: No further progress. 10/1/19: 1) IBM have started the work to add encodingErrorPolicy='replace'. 2) Daffodil have a temp setting to tolerate encodingErrorPolicy='error' with a warning. 3) Daffodil to investigate whether this is feasible. 4) More discussion needed on BOM 7/2: Updates: 1) In progress 2) As above. 3) In progress 4) No progress I can't find the email thread the action mentions, but my thoughts are as follows: There are 3 options - a) keep the spec as it is which implies BOM processing is core - a problem as neither Daffodil nor IBM DFDL implement it b) make BOM processing optional - which means there would need to be a property to switch it on or off in case an implementation started to support it later c) remove BOM processing altogether from 1.0 and add to the 2.0 list I am leaning towards c) on the following grounds: - Only one customer that I know of ever requested BOM processing for non-XML data (in 2010, for MRM, before IBM DFDL available) - BOM processing only applies to the message as a whole, not to any embedded Unicode fragments, so support is selective anyway - It is possible to model an optional BOM and use it to set a user-defined encoding variable which is then used by the rest of the schema I have a schema that models BOM and it successfully parses and unparses the 3 variants fine (no BOM present, BOM for BE present, BOM for LE present). If you have the BOM email thread please can you forward it, so I can see if I have missed any part of the thought process? I have found the original DFDL WG thread from 2011 when we added BOM support to the spec via erratum 3.7, which discusses the original motivation and design, I'll send you it. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/10/2018 12:16 Subject: Re: Part 2 - Re: Action 307 - Demonstrate implementation interoperability Mike, responses in-line below. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 04/10/2018 00:50 Subject: Part 2 - Re: Action 307 - Demonstrate implementation interoperability Based on Daffodil JIRA ticket backlog, and documentation at https://daffodil.apache.org/unsupported/, below are DFDL non-core features that are not supported by Daffodil, but that seem to be supported by IBM DFDL (based on my not finding anything that says they aren't implemented), and so are possibly in use in DFDL schemas we will need to use for interoperability testing. Please advise if IBM DFDL does *not* implement any of these. * default, fixed - for defaulting values at parse time - Daffodil support for this is partial at parse time, unsupported at unparse time. The fixed attribute isn't supported at all. * unordered sequences. IBM DFDL does not support default/fixed when parsing (see other thread). * byte-value entities - in contexts other than fillByte * ICU symbols 'u' and 'I' in calendarPattern * binaryFloatRep 'ibm390Hex' * documentFinalTerminatorCanBeMissing * textStandardBase - with value not equal to 10 * lengthKind 'prefixed', and prefixIncludesPrefixLength, prefixLengthType - Note IBM restricts prefixLengthType to a type that itself cannot be prefixed. Correct. * assert with failure type 'recoverableError' * calendarObserveDST * calendarCenturyStart * textNumberPattern 'V' and 'P' symbols * CCSID for specifying dfdl:encoding * nilKind 'literalValue' for binary data * choiceLengthKind 'explicit' and choiceLength * separatorSuppressionPolicy - behaviors for these in Daffodil are known to be both non-standard currently and also different from IBM DFDL. This needs correcting. IBM DFDL does not support 'trailingEmptyStrict' . The above list (after review/correction) needs to be crossed with the published DFDL schemas on github that were published by IBM. The features required to run those DFDL schemas are required for Daffodil to implement before the interoperability demonstration. Below are features of DFDL I believe neither IBM DFDL nor Daffodil implement, and as they are non-core, they need not be implemented by either for interoperability testing: * lengthKind 'endOfParent' * nilKind 'logicalValue' IBM DFDL implements this. * occursCountKind 'stopValue' (and occursStopValue) * textBiDi - and other related biDi properties * useNilForDefault IBM DFDL implements this * floating * fn:exactlyOne function * fn:namespace-URI() function * dfdl:escapeCharacterPolicy 'delimiters' - daffodil doesn't implement this property at all. Below are features of Daffodil that are not implemented by IBM DFDL and so cannot be used in schemas created using Daffodil that intend to be interoperable. These are either easy to work around, or impossible to work around, so are not a big deal, they just have to be kept in mind if considering a DFDL schema for use in interoperability testing. This includes schemas published on github, for image formats, CSV, etc., and a number of the FOUO schema published on DI2E.net/forge.mil - some of those quite possibly can work with IBM DFDL, and if they can do so, they should be modified so that they can be included in the interoperability testing. The list below mostly comes from https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… * calendarTimeZone specified as "" (empty string) - This is the most problematic one, so I've put it first. The predefined DFDL named format that is supplied with Daffodil and used as a starting point by most schemas has calendarTimeZone="". This is because customers didn't like that their datetimes were all being appended with "+00:00" for UTC time zone (in the infoset) when the data simply didn't specify a time zone. Schemas intended for interoperability testing should specify 'UTC' for this property. * calendarTimeZone specified as an Olson format time zone * inputValueCalc, outputValueCalc * hiddenGroupRef * Asserts and discriminators on simple type definitions or global element definitions * fn:concat with more than 4 arguments * non-8-bit charset encodings * bitOrder not mostSignifcantBitFirst * '@' in textNumberPattern (TBD: unsure if Daffodil has this) * "_" in calendarLanguage * calendarLanguage an expression * assert & discriminator messages an expression * binaryBooleanTrueRep as "" (empty string) * checks for binary packed numbers with length units 'bits' and not a multiple of 4 length, and similarly for alignmentUnits bits and alignment not multiple of 4. (relevant to negative tests only) * lengthKind 'implicit' complex elements inside lengthKind 'delimited' complex elements. Additionally IBM DFDL contains these bugs in its expression processing: 1) Path locations are not correctly validated. Specifically, array elements without predicates and references into other choice branches are not flagged as errors. 2) In DFDL expression functions, the namespace prefixes for http://www.w3.org/2001/XMLSchema and http://www.w3.org/2005/xpath-functions" must be 'xs' and 'fn' respectively, even if not declared. 3) In DFDL expressions, namespace prefixes in paths are ignored and matching is against element name only. For interoperability testing therefore: - For 1) avoid the use of either example - For 2) always declare xmlns:xs and xmlns:fn and always use those prefixes in expressions - For 3) avoid sibling elements that have same name but different namespace; use elementFormDefault="unqualified" to avoid namespaces for local elements altogether Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Part 2 - Re: Action 307 - Demonstrate implementation interoperability - BOM
by Steve Hanson 27 Mar '19

27 Mar '19

Hi Mike The outstanding item to resolve is what to about BOMs. 307 Demonstrate implementation interoperability (Steve, Mike) 4/9: Need to make sure that DFDL spec section 21 lists a correct set of optional features, the implication being that Daffodil and IBM DFDL (and any other minimally conforming implementation) correctly implement the remaining required features. First step - see if there are any obvious omissions. 16/10: Steve sent email stating IBM DFDL's missing core features and non-compliant behaviour, and Mike responded. Discussion continuing via two separate email threads. Part 1 for core features. Part 2 for optional features. For the core features, agreed that the following needs to happen: 1) IBM adds encodingErrorPolicy='replace' 2) Daffodil adds encodingErrorPolicy='error' 3) Daffodil ensures that, if not implementing default/fixed when parsing, it gives an SDE if a required occurrence has empty rep and element has default/fixed set. 4) A position is agreed on BOM handling - ongoing via email. 1/11: Just BOM to conclude on from the above list 15/11: Not discussed 29/11: No further progress. 10/1/19: 1) IBM have started the work to add encodingErrorPolicy='replace'. 2) Daffodil have a temp setting to tolerate encodingErrorPolicy='error' with a warning. 3) Daffodil to investigate whether this is feasible. 4) More discussion needed on BOM 7/2: Updates: 1) In progress 2) As above. 3) In progress 4) No progress I can't find the email thread the action mentions, but my thoughts are as follows: There are 3 options - a) keep the spec as it is which implies BOM processing is core - a problem as neither Daffodil nor IBM DFDL implement it b) make BOM processing optional - which means there would need to be a property to switch it on or off in case an implementation started to support it later c) remove BOM processing altogether from 1.0 and add to the 2.0 list I am leaning towards c) on the following grounds: - Only one customer that I know of ever requested BOM processing for non-XML data (in 2010, for MRM, before IBM DFDL available) - BOM processing only applies to the message as a whole, not to any embedded Unicode fragments, so support is selective anyway - It is possible to model an optional BOM and use it to set a user-defined encoding variable which is then used by the rest of the schema I have a schema that models BOM and it successfully parses and unparses the 3 variants fine (no BOM present, BOM for BE present, BOM for LE present). If you have the BOM email thread please can you forward it, so I can see if I have missed any part of the thought process? I have found the original DFDL WG thread from 2011 when we added BOM support to the spec via erratum 3.7, which discusses the original motivation and design, I'll send you it. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/10/2018 12:16 Subject: Re: Part 2 - Re: Action 307 - Demonstrate implementation interoperability Mike, responses in-line below. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 04/10/2018 00:50 Subject: Part 2 - Re: Action 307 - Demonstrate implementation interoperability Based on Daffodil JIRA ticket backlog, and documentation at https://daffodil.apache.org/unsupported/, below are DFDL non-core features that are not supported by Daffodil, but that seem to be supported by IBM DFDL (based on my not finding anything that says they aren't implemented), and so are possibly in use in DFDL schemas we will need to use for interoperability testing. Please advise if IBM DFDL does *not* implement any of these. * default, fixed - for defaulting values at parse time - Daffodil support for this is partial at parse time, unsupported at unparse time. The fixed attribute isn't supported at all. * unordered sequences. IBM DFDL does not support default/fixed when parsing (see other thread). * byte-value entities - in contexts other than fillByte * ICU symbols 'u' and 'I' in calendarPattern * binaryFloatRep 'ibm390Hex' * documentFinalTerminatorCanBeMissing * textStandardBase - with value not equal to 10 * lengthKind 'prefixed', and prefixIncludesPrefixLength, prefixLengthType - Note IBM restricts prefixLengthType to a type that itself cannot be prefixed. Correct. * assert with failure type 'recoverableError' * calendarObserveDST * calendarCenturyStart * textNumberPattern 'V' and 'P' symbols * CCSID for specifying dfdl:encoding * nilKind 'literalValue' for binary data * choiceLengthKind 'explicit' and choiceLength * separatorSuppressionPolicy - behaviors for these in Daffodil are known to be both non-standard currently and also different from IBM DFDL. This needs correcting. IBM DFDL does not support 'trailingEmptyStrict' . The above list (after review/correction) needs to be crossed with the published DFDL schemas on github that were published by IBM. The features required to run those DFDL schemas are required for Daffodil to implement before the interoperability demonstration. Below are features of DFDL I believe neither IBM DFDL nor Daffodil implement, and as they are non-core, they need not be implemented by either for interoperability testing: * lengthKind 'endOfParent' * nilKind 'logicalValue' IBM DFDL implements this. * occursCountKind 'stopValue' (and occursStopValue) * textBiDi - and other related biDi properties * useNilForDefault IBM DFDL implements this * floating * fn:exactlyOne function * fn:namespace-URI() function * dfdl:escapeCharacterPolicy 'delimiters' - daffodil doesn't implement this property at all. Below are features of Daffodil that are not implemented by IBM DFDL and so cannot be used in schemas created using Daffodil that intend to be interoperable. These are either easy to work around, or impossible to work around, so are not a big deal, they just have to be kept in mind if considering a DFDL schema for use in interoperability testing. This includes schemas published on github, for image formats, CSV, etc., and a number of the FOUO schema published on DI2E.net/forge.mil - some of those quite possibly can work with IBM DFDL, and if they can do so, they should be modified so that they can be included in the interoperability testing. The list below mostly comes from https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… * calendarTimeZone specified as "" (empty string) - This is the most problematic one, so I've put it first. The predefined DFDL named format that is supplied with Daffodil and used as a starting point by most schemas has calendarTimeZone="". This is because customers didn't like that their datetimes were all being appended with "+00:00" for UTC time zone (in the infoset) when the data simply didn't specify a time zone. Schemas intended for interoperability testing should specify 'UTC' for this property. * calendarTimeZone specified as an Olson format time zone * inputValueCalc, outputValueCalc * hiddenGroupRef * Asserts and discriminators on simple type definitions or global element definitions * fn:concat with more than 4 arguments * non-8-bit charset encodings * bitOrder not mostSignifcantBitFirst * '@' in textNumberPattern (TBD: unsure if Daffodil has this) * "_" in calendarLanguage * calendarLanguage an expression * assert & discriminator messages an expression * binaryBooleanTrueRep as "" (empty string) * checks for binary packed numbers with length units 'bits' and not a multiple of 4 length, and similarly for alignmentUnits bits and alignment not multiple of 4. (relevant to negative tests only) * lengthKind 'implicit' complex elements inside lengthKind 'delimited' complex elements. Additionally IBM DFDL contains these bugs in its expression processing: 1) Path locations are not correctly validated. Specifically, array elements without predicates and references into other choice branches are not flagged as errors. 2) In DFDL expression functions, the namespace prefixes for http://www.w3.org/2001/XMLSchema and http://www.w3.org/2005/xpath-functions" must be 'xs' and 'fn' respectively, even if not declared. 3) In DFDL expressions, namespace prefixes in paths are ignored and matching is against element name only. For interoperability testing therefore: - For 1) avoid the use of either example - For 2) always declare xmlns:xs and xmlns:fn and always use those prefixes in expressions - For 3) avoid sibling elements that have same name but different namespace; use elementFormDefault="unqualified" to avoid namespaces for local elements altogether Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call on 2018-03-21 - cancelled
by Steve Hanson 21 Mar '19

21 Mar '19

Apologies but Steve has a deadline for this Friday. Should be ok for next call in 2 weeks time. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0