April 2019 - dfdl-wg - lists.ogf.org

Agenda for tomorrow's OGF DFDL WG Call
by Steve Hanson 04 Apr '19

04 Apr '19

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13586?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Re: [DFDL-WG] Latest OGF DFDL WG Call Minutes
by Steve Hanson 03 Apr '19

03 Apr '19

Hi Bradd I have a few questions please ... inline below. I'm still going to need a real worked example, starting with some actual data and its schema, what it appears like after paring in the infoset, and how unparsing lays it back out again. Are you able to make the rescheduled call this Friday? Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org, "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int> Date: 27/02/2019 22:22 Subject: Re: Latest OGF DFDL WG Call Minutes Regarding proposal for offsets and pointers: The following are the properties to be defined: SMH: I was expecting to see these properties only on dfdl:element, especially as you say '...the element contents... ? indirectKind Enum Valid values 'pointer', 'offset' (there is also a thought of objectId or refId for handling BSON but not at this time) Specifies the type of indirection used to access the element contents in the data stream. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I am missing the distinction between offset and pointer. Is one relative to current position and the other relative to start of bitstream? SMH: In earlier DFDL proposals for offset support, we had used the term to refer to a property to be used to establish position of the current element instead of assuming the current element followed straight after the previous one. It would allow sparse modelling of fixed structures. The offset could be relative to start of bitstream or some other point. I don't think that's what you mean when you say 'offset' so I will refer to your new concept as 'pointer'. SMH: Assuming that indirectKind is a normal DFDL property, it can be in scope. It would therefore need to have an enum 'None' which would be the default used in most schemas. indirectLength Non-negative Integer or DFDL expression Specifies the length of the indirection in units according to the indirectUnits property. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group indirectUnits Enum Valid values 'bytes','bits' Specifies the units to be used for reading or writing the indirection according to indirectLength. The default value is 'bytes'. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group SMH: I think a better approach is to provide a property dfdl:indirectType, instead of indirectLength/indirectUnits, which refers to a simple type (not element) that carries its own lengthKind, length & lengthUnits properties. Similar idea to dfdl:prefixLengthType. That allows a lot of flexibility on how the pointer can appear. offsetBase non-empty string containing an absolute or relative XPath expression for the base element. Annotation: dfdl:element, dfdl:simpleType, dfdl:choice, dfdl:sequence, dfdl:group The proposal would be to have the contents of the indirection be after the LeadingAlignment and before the TrailingAlignment. This would mean the aligment and skip factors apply to the indirection values in the data stream instead of the the contents of the indirection. SMH: Agree. This also then means in an array element, each element has its own indirection value (pointer or offset) and the alignment,skip factors then apply to each of these indirection values. SMH: Do you mean '...each occurrence...' ? It would be thought that the indirection values apply only to the data stream and not the infoset. During parse when the infoset is populated from the data stream, the indirection values are replaced by the contents. During unparse, the indirection values don't exist in the infoset and are created during the writing to/creation of the data stream. SMH: I agree that the indirection should be a purely physical thing, but I am not clear how the value is filled in when unparsing. Where does the value come from? outputValueCalc? Or maybe it's not needed when unparsing, and the data is always contiguous? For pointers, a null pointer creates the scenario of either nil representation or empty representation depending on whether or not nillable is defined as true. Unless default values (or 0 occurrence) are defined for all underlying content, then this is a processing error. During unparse, the only scenario in which a null pointer would be created is for a nil representation. SMH: This needs more thought. The nil & default properties apply to the contents of the indirection, not to the pointer. If you want to give a nil semantic to the pointer value itself, then that would require a new enum for dfdl:nilKind. I don't see why a pointer value 0 can't be treated like any other indirection value. A missing pointer is an error - it must be present - there is no way to control optionality because minOccurs/maxOccurs apply to the contents. (Alternatively, if you want the concepts of nil, default, occurs to apply to the indirect value, then dfdl:indirectType could point at an element instead of a simple type - but that seems way too over engineered). Examples: The following is the definition for the address of a null-terminated string in which the string address may be NULL as indicated by a nillable value of true: <xs:element name="myString" type="xs:string" dfdl:lengthKind="delimited" dfdl:encoding="UTF-8" dfdl:terminator="%NUL;" dfdl:indirectKind="pointer" dfdl:indirectLength="8" dfdl:indirectUnits="bytes" nillable="true" /> The following is the definition for an array of three 4 byte addresses of a complex element defined by ns0:myStruct: <xs:element name="myArray" type="ns0:myStruct" dfdl:lengthKind="implicit" dfdl:indirectKind="pointer" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" minOccurs="3" maxOccurs="3" dfdl:occursCountKind="fixed" /> The following is the definition for a 4 byte offset to a 100 byte hexBinary value from the start of the parent element definition: <xs:element name="myData" type="xs:hexBinary" dfdl:lengthKind="explicit" dfdl:length="100" dfdl:lengthUnits="bytes" dfdl:indirectKind="offset" dfdl:indirectLength="4" dfdl:indirectUnits="bytes" dfdl:offsetBase=".." /> SMH: I don't see how unparsing works. What provides the value? The proposal would also allow for the following optional item but I don't currently see a need for this: dfdl:offsetKind with values "startToStart" or "endToStart" - indicates if the offset is from the start of the base element or the end of the base element. I tried getting this out before my vacation so it might take a little bit to respond for issues. Thank you for your time. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Cc: "Mike Beckerle" <mbeckerle(a)tresys.com>, "Michele Zundo" <michele.zundo(a)esa.int>, Bradd Kadlecik/Poughkeepsie/IBM@IBMUS Date: 02/07/2019 12:32 PM Subject: Latest OGF DFDL WG Call Minutes Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Rescheduled: OGF DFDL Working Group call (5 Apr 16:00 GDT)
by Steve Hanson 03 Apr '19

03 Apr '19

Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 306 - IBM DFDL behaviour when parsing empty strings
by Steve Hanson 03 Apr '19

03 Apr '19

306 Confirm IBM DFDL behaviour when parsing empty strings (Steve) 7/8: IBM DFDL has not fully implemented the behaviour changes arising from action 140 with respect to empty string elements. Daffodil is about to do so. IBM DFDL users have complained about lack of defaults when parsing but other than that appear happy. Are the rules in the spec for empty strings over complicated? Steve to document the behaviour for IBM DFDL to inform the discussion. ... 1/11: In progress - there are a lot of subtle scenarios 15/11: Not discussed ... 7/2/19: No further progress Some progress :) 9.4.2.2 Simple element (xs:string or xs:hexBinary) Required occurrence: If the element has a default value then an item is added to the infoset using the default value, otherwise an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset using empty string (type xs:string) or empty hexBinary (type xs:hexBinary) as the value, otherwise nothing is added to the Infoset. IBM DFDL behaviour: Required. IBM DFDL does not implement default values when parsing, so an empty occurrence with a default value gives an SDE (to prevent backtracking). An empty occurrence with no default gives a Processing Error. If you need to add an empty string to the infoset, you can add default=""(when default values implemented, of course). Optional. IBM DFDL adds nothing to the infoset regardless of presence of initiator and/or terminator. No way to get empty string into the infoset. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then an empty string (type xs:string), empty hexBinary (type xs:hexBinary), or default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) IBM DFDL behaviour: Required. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). Optional. IBM DFDL follows the spec (modulo 1 when an error would have been thrown, as per its 9.4.2.2 behaviour). So ... The spec today is consistent in one way, in that for both complex & string elements a) a required empty occurrence always adds to the infoset; & b) an optional empty occurrence adds to the infoset if initiator/terminator present; & c) an optional empty occurrence does not add to the infoset if no initiator/terminator present. If the simple string behaviour was to change to match IBM DFDL then that consistency is lost, but the string behaviour then matches that for other simple types. Section 9.4.2.2 disappears as the behaviour is same as 9.4.2.1. Section 9.4.2.3 becomes as below. We lose the ability to get an empty string into the infoset for an optional string with initiator/terminator. 9.4.2.3 Complex element Required occurrence: An item is added to the Infoset. Optional occurrence: If dfdl:emptyValueDelimiterPolicy is not 'none' then an item is added to the Infoset, otherwise nothing is added to the Infoset. For both required and optional occurrences, the Infoset item may also have a child item. 1. If the first child element of the complex type is a required simple element, then a default value will also be added to the Infoset. 2. If the first child element of the complex type is a required complex element, then an item is added to the Infoset (which may itself have a child via (1)) We also need to be sure that any other implementations have not yet implemented the current spec behaviour. Need to check with DFDL4S and IBM TPF. To be discussed on next WG call ... Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Action 307 - Demonstrate implementation interoperability
by Steve Hanson 03 Apr '19

03 Apr '19

Action 307 was raised recently and first task is for implementations to identify which core spec behaviour is not implemented. IBM DFDL The following is the list of DFDL 1.0 spec core features that IBM DFDL does not yet implement. - dfdl:encodingErrorPolicy "replace" - dfdl:binaryBooleanTrueRep with value empty string - dfdl:assert on global element and simple type - dfdl:discriminator on global element and simple type - Multiple xs:appinfo elements within each xs:annotation element - When parsing, the distinction between an element being 'missing', having an 'empty representation' and having an 'absent representation', is not in accordance with the specification. - When encoding is 'UTF-8' or 'UTF-16', byte order marks are not processed The above lists are derived from information at https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… and are those that apply to core spec features. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 6

Re: [DFDL-WG] Part 2 - Re: Action 307 - Demonstrate implementation interoperability - BOM
by Steve Hanson 03 Apr '19

03 Apr '19

Ignore this, see other thread Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 27/03/2019 16:15 Subject: Re: Part 2 - Re: Action 307 - Demonstrate implementation interoperability - BOM Hi Mike The outstanding item to resolve is what to about BOMs. 307 Demonstrate implementation interoperability (Steve, Mike) 4/9: Need to make sure that DFDL spec section 21 lists a correct set of optional features, the implication being that Daffodil and IBM DFDL (and any other minimally conforming implementation) correctly implement the remaining required features. First step - see if there are any obvious omissions. 16/10: Steve sent email stating IBM DFDL's missing core features and non-compliant behaviour, and Mike responded. Discussion continuing via two separate email threads. Part 1 for core features. Part 2 for optional features. For the core features, agreed that the following needs to happen: 1) IBM adds encodingErrorPolicy='replace' 2) Daffodil adds encodingErrorPolicy='error' 3) Daffodil ensures that, if not implementing default/fixed when parsing, it gives an SDE if a required occurrence has empty rep and element has default/fixed set. 4) A position is agreed on BOM handling - ongoing via email. 1/11: Just BOM to conclude on from the above list 15/11: Not discussed 29/11: No further progress. 10/1/19: 1) IBM have started the work to add encodingErrorPolicy='replace'. 2) Daffodil have a temp setting to tolerate encodingErrorPolicy='error' with a warning. 3) Daffodil to investigate whether this is feasible. 4) More discussion needed on BOM 7/2: Updates: 1) In progress 2) As above. 3) In progress 4) No progress I can't find the email thread the action mentions, but my thoughts are as follows: There are 3 options - a) keep the spec as it is which implies BOM processing is core - a problem as neither Daffodil nor IBM DFDL implement it b) make BOM processing optional - which means there would need to be a property to switch it on or off in case an implementation started to support it later c) remove BOM processing altogether from 1.0 and add to the 2.0 list I am leaning towards c) on the following grounds: - Only one customer that I know of ever requested BOM processing for non-XML data (in 2010, for MRM, before IBM DFDL available) - BOM processing only applies to the message as a whole, not to any embedded Unicode fragments, so support is selective anyway - It is possible to model an optional BOM and use it to set a user-defined encoding variable which is then used by the rest of the schema I have a schema that models BOM and it successfully parses and unparses the 3 variants fine (no BOM present, BOM for BE present, BOM for LE present). If you have the BOM email thread please can you forward it, so I can see if I have missed any part of the thought process? I have found the original DFDL WG thread from 2011 when we added BOM support to the spec via erratum 3.7, which discusses the original motivation and design, I'll send you it. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 09/10/2018 12:16 Subject: Re: Part 2 - Re: Action 307 - Demonstrate implementation interoperability Mike, responses in-line below. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 04/10/2018 00:50 Subject: Part 2 - Re: Action 307 - Demonstrate implementation interoperability Based on Daffodil JIRA ticket backlog, and documentation at https://daffodil.apache.org/unsupported/, below are DFDL non-core features that are not supported by Daffodil, but that seem to be supported by IBM DFDL (based on my not finding anything that says they aren't implemented), and so are possibly in use in DFDL schemas we will need to use for interoperability testing. Please advise if IBM DFDL does *not* implement any of these. * default, fixed - for defaulting values at parse time - Daffodil support for this is partial at parse time, unsupported at unparse time. The fixed attribute isn't supported at all. * unordered sequences. IBM DFDL does not support default/fixed when parsing (see other thread). * byte-value entities - in contexts other than fillByte * ICU symbols 'u' and 'I' in calendarPattern * binaryFloatRep 'ibm390Hex' * documentFinalTerminatorCanBeMissing * textStandardBase - with value not equal to 10 * lengthKind 'prefixed', and prefixIncludesPrefixLength, prefixLengthType - Note IBM restricts prefixLengthType to a type that itself cannot be prefixed. Correct. * assert with failure type 'recoverableError' * calendarObserveDST * calendarCenturyStart * textNumberPattern 'V' and 'P' symbols * CCSID for specifying dfdl:encoding * nilKind 'literalValue' for binary data * choiceLengthKind 'explicit' and choiceLength * separatorSuppressionPolicy - behaviors for these in Daffodil are known to be both non-standard currently and also different from IBM DFDL. This needs correcting. IBM DFDL does not support 'trailingEmptyStrict' . The above list (after review/correction) needs to be crossed with the published DFDL schemas on github that were published by IBM. The features required to run those DFDL schemas are required for Daffodil to implement before the interoperability demonstration. Below are features of DFDL I believe neither IBM DFDL nor Daffodil implement, and as they are non-core, they need not be implemented by either for interoperability testing: * lengthKind 'endOfParent' * nilKind 'logicalValue' IBM DFDL implements this. * occursCountKind 'stopValue' (and occursStopValue) * textBiDi - and other related biDi properties * useNilForDefault IBM DFDL implements this * floating * fn:exactlyOne function * fn:namespace-URI() function * dfdl:escapeCharacterPolicy 'delimiters' - daffodil doesn't implement this property at all. Below are features of Daffodil that are not implemented by IBM DFDL and so cannot be used in schemas created using Daffodil that intend to be interoperable. These are either easy to work around, or impossible to work around, so are not a big deal, they just have to be kept in mind if considering a DFDL schema for use in interoperability testing. This includes schemas published on github, for image formats, CSV, etc., and a number of the FOUO schema published on DI2E.net/forge.mil - some of those quite possibly can work with IBM DFDL, and if they can do so, they should be modified so that they can be included in the interoperability testing. The list below mostly comes from https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools… * calendarTimeZone specified as "" (empty string) - This is the most problematic one, so I've put it first. The predefined DFDL named format that is supplied with Daffodil and used as a starting point by most schemas has calendarTimeZone="". This is because customers didn't like that their datetimes were all being appended with "+00:00" for UTC time zone (in the infoset) when the data simply didn't specify a time zone. Schemas intended for interoperability testing should specify 'UTC' for this property. * calendarTimeZone specified as an Olson format time zone * inputValueCalc, outputValueCalc * hiddenGroupRef * Asserts and discriminators on simple type definitions or global element definitions * fn:concat with more than 4 arguments * non-8-bit charset encodings * bitOrder not mostSignifcantBitFirst * '@' in textNumberPattern (TBD: unsure if Daffodil has this) * "_" in calendarLanguage * calendarLanguage an expression * assert & discriminator messages an expression * binaryBooleanTrueRep as "" (empty string) * checks for binary packed numbers with length units 'bits' and not a multiple of 4 length, and similarly for alignmentUnits bits and alignment not multiple of 4. (relevant to negative tests only) * lengthKind 'implicit' complex elements inside lengthKind 'delimited' complex elements. Additionally IBM DFDL contains these bugs in its expression processing: 1) Path locations are not correctly validated. Specifically, array elements without predicates and references into other choice branches are not flagged as errors. 2) In DFDL expression functions, the namespace prefixes for http://www.w3.org/2001/XMLSchema and http://www.w3.org/2005/xpath-functions" must be 'xs' and 'fn' respectively, even if not declared. 3) In DFDL expressions, namespace prefixes in paths are ignored and matching is against element name only. For interoperability testing therefore: - For 1) avoid the use of either example - For 2) always declare xmlns:xs and xmlns:fn and always use those prefixes in expressions - For 3) avoid sibling elements that have same name but different namespace; use elementFormDefault="unqualified" to avoid namespaces for local elements altogether Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0