Re: [DFDL-WG] Action 174: Making DFDL implementations easier
Tim I tend to agree with endOfParent as optional. Expression language was kept in the core to handle occursCountKind 'expression' which is also in the core. The examples of binary data we have seen from NSA and ESA both have occurs counts in the data, in the same way as COBOL does. When you have untagged binary data, it's typically the way to provide an array size. It was felt that dropping this from the core did left us with too little capability. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg@ogf.org, dfdl-wg-bounces@ogf.org Date: 25/05/2012 10:07 Subject: Re: [DFDL-WG] Action 174: Making DFDL implementations easier 1) I would make endOfParent an optional feature - there are not many formats that require it. 3) There are many formats that do not require the expression language - it is only required when a property value or an assert/discriminator needs to query already-parsed data. On that basis, I think the entire expression language feature should be optional. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg@ogf.org Date: 25/05/2012 00:08 Subject: [DFDL-WG] Action 174: Making DFDL implementations easier Sent by: dfdl-wg-bounces@ogf.org Agreed on list, just need to answer questions 1) and 3) below. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/05/2012 18:21 ----- From: Steve Hanson/UK/IBM To: dfdl-wg@ogf.org Date: 15/05/2012 09:55 Subject: Making DFDL implementations easier Please see below for a proposal to make an additional set of DFDL features optional. The goal is to make it considerably easier to create a minimal conforming DFDL processor for binary data. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/05/2012 12:50 ----- Feature Detection Text representation for types other than String dfdl:representation="text" for Number, Calendar or Boolean types Delimiters dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or dfdl:lengthKind="delimited" BCD calendars dfdl:binaryCalendarRep="bcd" Multiple schemas xs:include or xs:import in xsd Named Formats dfdl:defineFormat or dfdl:ref Choices xs:choice in xsd ** Arrays where size not known in advance dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' ** Advanced expressions Advanced features of the DFDL expression language (tbd) ** Including one of these features mean that speculative parsing is needed. Remaining questions: 1) What about lengthKind 'endOfParent' ? 2) Is leaving out choices too restrictive? 3) Expression language subset The result is that a minimal conformant DFDL implementation just needs to support the following annotations and properties, and does not need speculative parsing. dfdl:element dfdl:sequence dfdl:format byteOrder encoding utf16width alignment alignmentUnits (bytes) fillByte leadingSkip trailingSkip lengthKind (explicit, implicit) length lengthUnits (bytes, characters) representation (binary) textPadKind textTrimKind textStringJustification textStringPadCharacter truncateSpecifiedLengthString decimalSigned binaryNumberRep binaryVirtualDecimalPoint binaryFloatRep (ieee) binaryBooleanTrueRep binaryBooleanFalseRep binaryCalendarRep (binarySeconds, binaryMilliseconds) binaryCalendarEpoch sequenceKind (ordered) occursCountKind (fixed, expression) occursCount Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
But the expression language is highly restricted in the subset. On Fri, May 25, 2012 at 7:59 AM, Steve Hanson <smh@uk.ibm.com> wrote:
Tim
I tend to agree with endOfParent as optional.
Expression language was kept in the core to handle occursCountKind 'expression' which is also in the core. The examples of binary data we have seen from NSA and ESA both have occurs counts in the data, in the same way as COBOL does. When you have untagged binary data, it's typically the way to provide an array size. It was felt that dropping this from the core did left us with too little capability.
Regards
Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848
From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg@ogf.org, dfdl-wg-bounces@ogf.org Date: 25/05/2012 10:07 Subject: Re: [DFDL-WG] Action 174: Making DFDL implementations easier ------------------------------
1) I would make endOfParent an optional feature - there are not many formats that require it.
3) There are many formats that do not require the expression language - it is only required when a property value or an assert/discriminator needs to query already-parsed data. On that basis, I think the entire expression language feature should be optional.
regards,
Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 246742
From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg@ogf.org Date: 25/05/2012 00:08 Subject: [DFDL-WG] Action 174: Making DFDL implementations easier Sent by: dfdl-wg-bounces@ogf.org ------------------------------
Agreed on list, just need to answer questions 1) and 3) below.
Regards
Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/05/2012 18:21 -----
From: Steve Hanson/UK/IBM To: dfdl-wg@ogf.org Date: 15/05/2012 09:55 Subject: Making DFDL implementations easier ------------------------------
Please see below for a proposal to make an additional set of DFDL features optional. The goal is to make it considerably easier to create a minimal conforming DFDL processor for binary data.
Regards
Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh@uk.ibm.com* <smh@uk.ibm.com> tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/05/2012 12:50 ----- *Feature* *Detection* Text representation for types other than String dfdl:representation="text" for Number, Calendar or Boolean types Delimiters dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or dfdl:lengthKind="delimited" BCD calendars dfdl:binaryCalendarRep="bcd" Multiple schemas xs:include or xs:import in xsd Named Formats dfdl:defineFormat or dfdl:ref Choices xs:choice in xsd ** Arrays where size not known in advance dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' ** Advanced expressions Advanced features of the DFDL expression language (tbd) * ** Including one of these features mean that speculative parsing is needed. *
Remaining questions:
1) What about lengthKind 'endOfParent' ? 2) Is leaving out choices too restrictive? 3) Expression language subset
The result is that a minimal conformant DFDL implementation just needs to support the following annotations and properties, and does not need speculative parsing.
dfdl:element dfdl:sequence dfdl:format
byteOrder encoding utf16width alignment alignmentUnits (bytes) fillByte leadingSkip trailingSkip lengthKind (explicit, implicit) length lengthUnits (bytes, characters) representation (binary) textPadKind textTrimKind textStringJustification textStringPadCharacter truncateSpecifiedLengthString decimalSigned binaryNumberRep binaryVirtualDecimalPoint binaryFloatRep (ieee) binaryBooleanTrueRep binaryBooleanFalseRep binaryCalendarRep (binarySeconds, binaryMilliseconds) binaryCalendarEpoch sequenceKind (ordered) occursCountKind (fixed, expression) occursCount
Regards
Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> IBM SWG, Hursley, UK* **smh@uk.ibm.com* <smh@uk.ibm.com> tel:*+44-1962-815848* <%2B44-1962-815848>
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg
-- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412
My current thinking is: - Minimum expression language is a simple variable reference. e.g. {$myLength} or {$myParentStructureLength}. - That would require the ability to declare DFDL variables, set them, read them and put them into and out of scope. And restore their value after backtracking if backtracking is supported in the implementation. This would make all the XPath functions optional, and would also make it unnecessary to use an XPath query engine. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Mike Beckerle <mbeckerle.dfdl@gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: Tim Kimber/UK/IBM@IBMGB, dfdl-wg@ogf.org, dfdl-wg-bounces@ogf.org Date: 25/05/2012 13:58 Subject: Re: [DFDL-WG] Action 174: Making DFDL implementations easier But the expression language is highly restricted in the subset. On Fri, May 25, 2012 at 7:59 AM, Steve Hanson <smh@uk.ibm.com> wrote: Tim I tend to agree with endOfParent as optional. Expression language was kept in the core to handle occursCountKind 'expression' which is also in the core. The examples of binary data we have seen from NSA and ESA both have occurs counts in the data, in the same way as COBOL does. When you have untagged binary data, it's typically the way to provide an array size. It was felt that dropping this from the core did left us with too little capability. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg@ogf.org, dfdl-wg-bounces@ogf.org Date: 25/05/2012 10:07 Subject: Re: [DFDL-WG] Action 174: Making DFDL implementations easier 1) I would make endOfParent an optional feature - there are not many formats that require it. 3) There are many formats that do not require the expression language - it is only required when a property value or an assert/discriminator needs to query already-parsed data. On that basis, I think the entire expression language feature should be optional. regards, Tim Kimber, Common Transformation Team, Hursley, UK Internet: kimbert@uk.ibm.com Tel. 01962-816742 Internal tel. 246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg@ogf.org Date: 25/05/2012 00:08 Subject: [DFDL-WG] Action 174: Making DFDL implementations easier Sent by: dfdl-wg-bounces@ogf.org Agreed on list, just need to answer questions 1) and 3) below. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/05/2012 18:21 ----- From: Steve Hanson/UK/IBM To: dfdl-wg@ogf.org Date: 15/05/2012 09:55 Subject: Making DFDL implementations easier Please see below for a proposal to make an additional set of DFDL features optional. The goal is to make it considerably easier to create a minimal conforming DFDL processor for binary data. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/05/2012 12:50 ----- Feature Detection Text representation for types other than String dfdl:representation="text" for Number, Calendar or Boolean types Delimiters dfdl:separator <> "" or dfdl:initiator <> "" or dfdl:terminator <> "" or dfdl:lengthKind="delimited" BCD calendars dfdl:binaryCalendarRep="bcd" Multiple schemas xs:include or xs:import in xsd Named Formats dfdl:defineFormat or dfdl:ref Choices xs:choice in xsd ** Arrays where size not known in advance dfdl:occursCountKind 'implicit', 'parsed', 'stopValue' ** Advanced expressions Advanced features of the DFDL expression language (tbd) ** Including one of these features mean that speculative parsing is needed. Remaining questions: 1) What about lengthKind 'endOfParent' ? 2) Is leaving out choices too restrictive? 3) Expression language subset The result is that a minimal conformant DFDL implementation just needs to support the following annotations and properties, and does not need speculative parsing. dfdl:element dfdl:sequence dfdl:format byteOrder encoding utf16width alignment alignmentUnits (bytes) fillByte leadingSkip trailingSkip lengthKind (explicit, implicit) length lengthUnits (bytes, characters) representation (binary) textPadKind textTrimKind textStringJustification textStringPadCharacter truncateSpecifiedLengthString decimalSigned binaryNumberRep binaryVirtualDecimalPoint binaryFloatRep (ieee) binaryBooleanTrueRep binaryBooleanFalseRep binaryCalendarRep (binarySeconds, binaryMilliseconds) binaryCalendarEpoch sequenceKind (ordered) occursCountKind (fixed, expression) occursCount Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg@ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
participants (3)
-
Mike Beckerle -
Steve Hanson -
Tim Kimber