June 2014 - dfdl-wg - lists.ogf.org

OGF DFDL WG Call Agenda 2014-06-10
by Steve Hanson 09 Jun '14

09 Jun '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13263?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Spec question: Parsing sequence groups with separators
by Tim Kimber 09 Jun '14

09 Jun '14

Yes - I wanted to keep the scope of the initial question as small as possible, but Steve is right to raise these related issues. I think the question about occursCountKind='expression' is a related but separate issue ( the problems that I raised in relation to SSP are not dependent on OCK='expression' ). Worth a separate action, I think. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 ----- Forwarded by Tim Kimber/UK/IBM on 09/06/2014 11:05 ----- From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 09/06/2014 10:36 Subject: Re: [DFDL-WG] Spec question: Parsing sequence groups with separators Sent by: dfdl-wg-bounces(a)ogf.org To put Tim's concerns in another way, the spec defines 'positional sequence' and 'non-positional sequence' in terms of the value of separatorSuppressionPolicy (section 14.2). But separatorSuppressionPolicy only applies when occursCountKind is 'implicit', for other occursCountKinds there is an implied separatorSuppressionPolicy value (section 14.2.2) . We did this partly so that separatorSuppressionPolicy can be put in scope and not cause errors. However when you create a sequence that contains elements with different occursCountKinds, you can end up with a hybrid which is positional in places and non-positional in others. We need to decide whether these kind of sequences are allowed. You can always wrap a group of elements in a sequence in order to change separatorSuppressionPolicy. occursCountKind 'expression'. This is stated as having implied separatorSuppressionPolicy 'never' on the grounds that is very like 'fixed'. That implies positional behaviour. But you need to parse the data in order to know the number of occurrences, so doesn't that make it non-positional? Also. section 16 states that when unparsing, 'expression' behaves like 'parsed' - and 'parsed' has implied separatorSuppressionPolicy 'empty'. Something not quite straight here. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 07/06/2014 21:31 Subject: [DFDL-WG] Spec question: Parsing sequence groups with separators Sent by: dfdl-wg-bounces(a)ogf.org The rules outlined in section 14.2.2 'Parsing Sequence Groups with Separators' are not properly specified, and probably cannot be consistently implemented. The last paragraph of Section 14.2.1 says this: "In the sections that follow, it is important to remember that the dfdl:separatorSuppressionPolicy property is carried on the sequence, while the XSDL minOccurs, XSDL maxOccurs and dfdl:occursCountKind properties are is carried on an element in that sequence." This is true, and this 'local overriding' of separatorSuppressionPolicy ( by arrays within the group ) is the cause of most of the problems. Problem #1: Complexity Consider a sequence group that has SSP='never' and the separator is a comma. Its members ( A,B,C ) must always be represented as follows: "a,b,c" or ",b,c" or ",,c" but never "b,c" because that would imply that the separator for an empty A had been suppressed. Now suppose that B is an array with minOccurs=0 and maxOccurs=3 and occursCountKind='implicit'. Acceptable representations are now: "a,b1,b2,b3,c" or "a,b1,,,c" or even "a,,,c" But if occursCountKind is changed to 'parsed' then the acceptable representations suddenly alter, and empty occurrences of B can be completely omitted. "a,b1,b2,b3,c" or "a,b1,c" or even "a,c" [ or should that be "a,,c" ] This seems wrong. The logic that implements suppression policy is hard enough to implement already. Bringing in an extra layer of complexity around arrays will make it so hard that most implementations would contain defects, leading to interoperability issues. Problem #2 Ambiguity See the brackets in the preceding paragraph. [ or should that be "a,,c" ] It is far from obvious whether the group should insist on having a delimiter for the array ( because its SSP is 'never' ) or whether the array should take liberty to suppress the separators for all of its members ( as I assumed when I wrote this email). The text of the specification is either silent or unclear on this point. Possible resolution: Rather than attempting to specify implied behaviours for the various occursCountKind settings, I believe the specification should a) prohibit the use of certain occursCountKinds within positional sequences b) require array occurrences to use the same SSP as other sequence members. After some discussion with the IBM team, I believe a) will not generate too many prohibited combinations, and the rationale for those prohibitions will be consistent with already-existing schema definition errors. b) will simplify the implementation of separation suppression, thus addressing the complexity problem. I expect we will need an action to be opened so that this can be discussed in the working group meetings. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Spec question: Parsing sequence groups with separators
by Tim Kimber 09 Jun '14

09 Jun '14

The rules outlined in section 14.2.2 'Parsing Sequence Groups with Separators' are not properly specified, and probably cannot be consistently implemented. The last paragraph of Section 14.2.1 says this: "In the sections that follow, it is important to remember that the dfdl:separatorSuppressionPolicy property is carried on the sequence, while the XSDL minOccurs, XSDL maxOccurs and dfdl:occursCountKind properties are is carried on an element in that sequence." This is true, and this 'local overriding' of separatorSuppressionPolicy ( by arrays within the group ) is the cause of most of the problems. Problem #1: Complexity Consider a sequence group that has SSP='never' and the separator is a comma. Its members ( A,B,C ) must always be represented as follows: "a,b,c" or ",b,c" or ",,c" but never "b,c" because that would imply that the separator for an empty A had been suppressed. Now suppose that B is an array with minOccurs=0 and maxOccurs=3 and occursCountKind='implicit'. Acceptable representations are now: "a,b1,b2,b3,c" or "a,b1,,,c" or even "a,,,c" But if occursCountKind is changed to 'parsed' then the acceptable representations suddenly alter, and empty occurrences of B can be completely omitted. "a,b1,b2,b3,c" or "a,b1,c" or even "a,c" [ or should that be "a,,c" ] This seems wrong. The logic that implements suppression policy is hard enough to implement already. Bringing in an extra layer of complexity around arrays will make it so hard that most implementations would contain defects, leading to interoperability issues. Problem #2 Ambiguity See the brackets in the preceding paragraph. [ or should that be "a,,c" ] It is far from obvious whether the group should insist on having a delimiter for the array ( because its SSP is 'never' ) or whether the array should take liberty to suppress the separators for all of its members ( as I assumed when I wrote this email). The text of the specification is either silent or unclear on this point. Possible resolution: Rather than attempting to specify implied behaviours for the various occursCountKind settings, I believe the specification should a) prohibit the use of certain occursCountKinds within positional sequences b) require array occurrences to use the same SSP as other sequence members. After some discussion with the IBM team, I believe a) will not generate too many prohibited combinations, and the rationale for those prohibitions will be consistent with already-existing schema definition errors. b) will simplify the implementation of separation suppression, thus addressing the complexity problem. I expect we will need an action to be opened so that this can be discussed in the working group meetings. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1

Re: [DFDL-WG] Action 259 - Consider allowing more flexible escapeBlock schemes
by Steve Hanson 04 Jun '14

04 Jun '14

WG call 3rd June: DFDL spec will change so that an escape block end does not have to be the last thing in the data (after trimming). It must always be present. A new erratum will be raised. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 20/05/2014 17:50 Subject: Re: [DFDL-WG] Action 259 - Consider allowing more flexible escapeBlock schemes As discussed on the call, there is an import case that is not covered in the table, namely where quotes surround a delimiter but the opening quote is not at the start of the data. I imported the following text string into Excel: This is "," two separate fields And indeed two columns were created, meaning the comma was treated as a delimiter and not escaped. This matches DFDL so good. Interestingly, the first column was as expected... This is " ...but the second was not: two separate fields Notice the leading quote was removed without error, meaning that the absence of the closing quote is permitted! Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 13/05/2014 15:37 Subject: Re: [DFDL-WG] Action 259 - Consider allowing more flexible escapeBlock schemes Sent by: dfdl-wg-bounces(a)ogf.org That looks fairly conclusive to me. DFDL should fall into line with established practice. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 13/05/2014 11:50 Subject: [DFDL-WG] Action 259 - Consider allowing more flexible escapeBlock schemes Sent by: dfdl-wg-bounces(a)ogf.org Action 259 was raised last call to decide what to do about the following, as minuted: Steve has an example of an escape block where the escape block end is not at the end of the un-trimmed data. This gives a processing error. Another IBM product accepts this usage. Should DFDL allow this? Or should there be a new escapeKind that allows escapeBlockStart/End anywhere? Tried importing these values from a CSV file into an Excel spreadsheet, a Symphony spreadsheet (ie, successor to 123), and also accessing them via ODBC using a Microsoft driver, to compare with IBM DFDL and IBM Cast Iron behaviour. Test Data IBM DFDL IBM Cast Iron MS Excel Lotus Symphony ODBC 1 This is normal This is normal This is normal This is normal This is normal This is normal 2 "This is OK" This is OK This is OK This is OK This is OK This is OK 3 "This| is expected" This| is expected This| is expected This| is expected This| is expected This| is expected 4 This too "is OK" This too "is OK" This too "is OK" This too "is OK" This too "is OK" This too 5 Even "this" is OK Even "this" is OK Even "this" is OK Even "this" is OK Even "this" is OK Even 6 "This" is NOT OK PARSE FAILED This is NOT OK This is NOT OK This is NOT OK This 7 "This"" is still OK" This" is still OK This" is still OK This" is still OK This" is still OK This" is still OK The data under discussion is 6. It looks like DFDL is out of step with the behaviour of Excel / Symphony spreadsheets, and Cast Iron has adopted that behaviour too. Out of interest I also checked the output behaviour from Excel. That escaped all instances of embedded quotes in the same way as DFDL, so no issues there. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

data where inactive escape character is to be retained
by Mike Beckerle 04 Jun '14

04 Jun '14

We have data that has ; as separator and has fields that look like this: abcd \; efgh That's a single field. The backslash escapes the ; so that the data is abcd ; efgh. This same data set also has abcd \n efgh Here the backslash precedes an ordinary non-delimiter. The data is supposed to be abcd \n efgh. That is, this data set requires the backslash to be retained in the data when it is not preceding the start of a delimiter. Am I missing something or is it impossible to model this? It would seem there needs to be a flag to indicate whether the escape characters that don't actually escape a delimiter are to be retained or not. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

3 2

OGF DFDL WG Call Minutes 2014-06-03
by Steve Hanson 03 Jun '14

03 Jun '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13256?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2014-05-20
by Steve Hanson 02 Jun '14

02 Jun '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13256?download= Might start a few minutes late Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: DFDL mini-tutorials
by Steve Hanson 02 Jun '14

02 Jun '14

I've also reviewed the tutorials with following comments in addition to Mark's. Daffodil_Fixed_Values_AveryBibeau.docx 'fixed' checking only takes place if validation is enabled during DFDL parse or unparse. That aspect is better covered by a tutorial on validation. Because 'fixed' also implies a defaulting behaviour, the checking aspect can not be dealt with in isolation, and needs to be part of a tutorial on default values. Daffodil_Trimming_AveryBibeau_3.docx Mentions textStringJustification but not textNumberJustification or textBooleanJustification or textCalendarJustification Doesn't mention 'center' justification The 'time' example should use logical integers as the element type, and therefore textNumberJustification and textNumberPadCharacter. Should also cover padding as the properties are related and often used together. As Mark says, the email header example is a poor one to use, as reading the schema indicates that the spaces are being consumed by the %WSP*; used in the initiators, and not by the trimming properties at all. Escape_Block_AveryBibeau.docx In the DFDL language, defineEscapeScheme is unique in that provides a way of grouping a specific set of related properties, and referring to them, instead of carrying them directly on an object. This is not properly covered. The dfdl:escapeSchemeRef is key, but is not mentioned in the text and only appears in the example. The name DefaultPropertiesEscapeScheme could be better. extraEscapeCharacter should be extraEscapedCharacters and is not described. Escape_Character_AveryBibeau.docx Same comments as above. Better to have a single tutorial that covers and contrasts both kinds of escape scheme. Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/06/2014 10:25 ----- From: Mark Frost/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Cc: jgarriss(a)mitre.org Date: 13/05/2014 16:32 Subject: Re: [DFDL-WG] DFDL mini-tutorials Sent by: dfdl-wg-bounces(a)ogf.org Hi All, I've had a read over these mini-tutorials, and think they're a great intro to specific features. In particular the style is clear and brief, and formatting is readable and attractive. It'd be great to weave these into the existing 'lessons' available at http://www.ogf.org/dfdl/. That could be an opportunity to polish the presentation of the lessons : existing lessons are not very high-profile in web searches (perhaps due to being pdfs linked only in the sidebar of the ogf site). A separate section with an introduction (and html content?) could help some cross-links between the lessons and tutorials would be handy - eg. lesson 4 covers fixed-length data, where trimming is often useful a little colour and formatting as used in these tutorials aids readability these tutorials use an XML format for presenting logical infosets, where as the lessons use a more generic style without xml tags. I wonder if using an XML style presentation might confuse some readers that an infoset "is" XML or the purpose of DFDL is to create XML? w3schools is an example of a tutorial style I find very effective : http://www.w3schools.com/schema/schema_intro.asp Below are a few specific points that came to mind in each of the mini-tutorials - Daffodil_Fixed_Values_AveryBibeau.docx discussion of schema vs. dfdl properties would be useful (explain why 'fixed' is not in the dfdl namespace) could also cover the defaulting behaviour of xs:fixed (it works like xs:default), which makes it more powerful than an equivalent enumeration facet with a single value. Daffodil_Trimming_AveryBibeau_3.docx email header example is a little confusing as it uses %WSP*; in dfdl:initiator, to consume unwanted whitespace, when the tutorial is about padding/trimming Escape_Block_AveryBibeau.docx cross ref to escape character tutorial would be useful when published "Escaping is a useful capability for using alternative data representation that includes the separator character" - more generally, all in-scope markup including separators - being more explicit might help some readers : that escaping is necessary to represent characters in logical values that would otherwise be interpreted as markup CSV is a good real-world example that could be included. Escape_Character_AveryBibeau.docx cross ref to escape block tutorial would be useful when published "Escape characters can be used to essentially ignore the following separator character in an element." - more generally, all in-scope markup including separators - being more explicit might help some readers : that escaping is necessary to represent characters in logical values that would otherwise be interpreted as markup "defineEscapeScheme is required to create properties related to escaping. This definition exists outside of the default properties schema." - I'm not clear what "outside default properties schema" means here Regarding possible future topics, I'd suggest leaving inputValueCalc/outputValueCalc for later as these aren't yet available in the IBM implementation. Regards, Mark Mark Frost IBM United Kingdom Software Engineer Hursley Park IBM DFDL, IBM Integration Bus Winchester SO21 2JN Phone: +44 (0)1962 817009 England e-mail: frostmar(a)uk.ibm.com From: "Cranford, Jonathan W." <jcranford(a)mitre.org> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Cc: "Garriss Jr., James P." <jgarriss(a)mitre.org> Date: 15/04/2014 16:30 Subject: [DFDL-WG] DFDL mini-tutorials Sent by: dfdl-wg-bounces(a)ogf.org DFDL Working Group, Attached are four mini-tutorials written by Avery Bibeau, a high school student. James Garriss of MITRE is mentoring him on a school project which includes writing tutorials for DFDL. The idea is that these mini-tutorials could be folded into tutorials in the future more in line with the existing DFDL tutorials at www.ogf.org/dfdl. Avery is contributing these mini-tutorials to the Working Group with the hope that he would be credited in the tutorials. Any feedback would be appreciated. Also, below is a list of topics he's planning to work on. If you see topics that aren't implemented in either of the implementations, please let us know, as he is only trying to write tutorials on features that he can test in the existing implementations. * asserts * discriminators * input value calculations He had planned to do a mini-tutorial on default values, but had problems testing his examples in both implementations because apparently support for default values is not quite implemented yet. Thanks in advance, -- Jonathan W. Cranford Senior Information Systems Engineer The MITRE Corporation (http://www.mitre.org) [attachment "Daffodil_Fixed_Values_AveryBibeau.docx" deleted by Mark Frost/UK/IBM] [attachment "Escape_Block_AveryBibeau.docx" deleted by Mark Frost/UK/IBM] [attachment "Escape_Character_AveryBibeau.docx" deleted by Mark Frost/UK/IBM] [attachment "Daffodil_Trimming_AveryBibeau_3.docx" deleted by Mark Frost/UK/IBM] -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0