October 2012 - dfdl-wg - lists.ogf.org

7-bit ascii packed together
by Mike Beckerle 31 Oct '12

31 Oct '12

I have a data format in front of me that has 64 7-bit ASCII characters, but the format has them bit-packed, i.e., 448 = 7 * 64 bits, so ....the character codes aren't octet/byte aligned. Furthermore, the 'string' either uses up the entire 64 character maximum length OR it has a terminating character which is a 0x7F character code. I believe I was the advocate for a position that character codes should always be 8-bit aligned. That would be because I had never seen anything like this. I am told there are also 6-bit ascii-variations, similarly packed together to save space. BTW: This occurs in a specific US MIL STD message header format, so it's not like it's some obscure unused corner case. Right now, the best I think I can do is to model this data not as a string at all, but as an array of integers, each one having 7-bit length, and not aligned (that is, aligned to 1-bit). Doing that I can use occursCountKind='parsed', and an assertion to deal with the optional termination by 0x7F value. To handle this as a string, we'd need to be able to specify that the character codes are not aligned, and the width of the bit-fields making up each character code. Or I suppose we could just say this is a special kind of character set encoding "ASCII-7-bit-packed" or something. Having that, I could deal with the termination via a choice of either the terminated flavor, or the fixed length flavor (which excludes the terminator) by way of a choice of two strings each having a lengthKind="pattern". Comments? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 2

Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update
by Steve Hanson 31 Oct '12

31 Oct '12

Thanks Suman. Specifically, please can you confirm that the P symbol and the E symbol may not be used together in a COBOL PIC clause. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Suman Kalia/Toronto/IBM@IBMCA To: Steve Hanson/UK/IBM@IBMGB, Cc: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org Date: 31/10/2012 15:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update
by Suman Kalia 31 Oct '12

31 Oct '12

We tried to model the representation of the picture clause in memory.. The best way to capture was through the string. For the following example (EBCDIC ) , this is how the memory is laid out as per IBM compiler documentation.. External floating point PIC +9(2).9(2)E+99 DISPLAY + 1234 4E F1 F2 4B F3 F4 C5 4E F0 F2 SKK --> +12.34E+02 - 1234 60 F1 F2 4B F3 F4 C5 4E F0 F2 SKK-> -12.34E+02 One can make an argument, the logical type for external float is could xsd:float/xsd:double with textPattern associated with it but this is not how COBOL compiler lays out this type in memory. P symbol Because the scaling position character P implies an assumed decimal point (to the left of the Ps if the Ps are leftmost PICTURE characters; to the right of the Ps if the Ps are rightmost PICTURE characters), the assumed decimal point symbol V is redundant as either the leftmost or rightmost character within such a PICTURE description. Based on my testing, The P symbol is allowed only numeric types and comp-3 i.e. on zoned decimal and packed decimal.. 01 a. 05 b pic 9(5). 05 Compute-Result Pic -9v9(9)E-99. 05 c pic PPP999. 05 d pic 99PP comp-3. Suman Kalia IBM Canada Lab WMB Toolkit Architect and Development Lead Tel: 905-413-3923 T/L 313-3923 Email: kalia(a)ca.ibm.com For info on Message broker http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: Andrew Edwards/UK/IBM@IBMGB, dfdl-wg(a)ogf.org, Suman Kalia/Toronto/IBM@IBMCA Date: 10/23/2012 07:04 AM Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update I've updated my comments based on yesterday's call and Mike's mail. Suman - please can you help with item 4 below ? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, Andrew Edwards/UK/IBM@IBMGB Date: 22/10/2012 17:29 Subject: Re: [DFDL-WG] One email or a flock or... - re: 10.03 draft - open review items - update Suggest wording for issue 1 below. Change in italics: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. (Speculative execution may try the terminator subsequently.) On Wed, Oct 17, 2012 at 6:56 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I have added some comments in-line to reflect the WG call on Tuesday, we will continue on Friday. (Andy - please see 5 below) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 02/10/2012 18:41 Subject: One email or a flock or... - re: 10.03 draft - open review items Steve, I've got the issues below left after your review pass on 10.03, minus 2 I send emails to you about separately. Should I issue this email to the WG, or do you want me to decompose this into separate emails, or do you just want to list these as agenda topics for next call? I think it is good if people get to look at them in advance of a call. ...mikeb -------------------------------------------------------- This is a list of items left open after a review pass by SMH(on draft in preparation r010.03). These items need specific WG discussion on a call. They may be small enough to resolve there, or may be escalated into action items. (A couple issues already clearly action-item related are not listed here.) Note: please Ignore the identifiers like SMH107 or m236 I'm tagging these with. Those are just for me editing the text. (Those change ...grrr... if someone inserts a comment into the document, so they're not good issue identifiers). 1. SMH107 Spec says: When the separator and terminator on a group have the same value, then at a point where either separator or terminator could be found, the separator is tried first. Issue is that this language still feels ambiguous. E.g., So it tries the separator first, let’s say it finds it. Will a subsequent processing error cause it to backtrack and revisit this and try the terminator? Or does finding the separator confirm that it IS a separator, resolve forever that point of uncertainty? I believe the latter is what was intended (delimiter decisions drive parsing and are not revisited), but we need to state this (or do we somewhere else already?) SMH: Mike will add the following after the sentence in question " (Speculative execution may try the terminator subsequently.)". Agreed that encountering a separator does not resolve a point of uncertainty. 2. SMH169 - Some numeric types are signed, others unsigned. Some representations are sign-capable, some are not (BCD specifically). Right now spec draft says you can't have bcd as rep for signed integer types long, int, short, byte. But you CAN have bcd for rep of decimal, integer. We could allow bcd only for nonNegativeInteger type, but there is no nonNegativeDecimal type, so....how to resolve? I would suggest that we simply allow bcd as rep for both signed and unsigned types, and it's a processing error to unparse a negative value into bcd rep. SMH: Noted that for a decimal, property decimalSigned is used to indicate whether the logical value is signed or not. So we could disallow BCD for integer and for decimal when decimalSigned is 'yes'. Interestingly, section 3.7.1 states "Signed numbers with dfdl:binaryNumberRep 'bcd' are always positive. On unparsing it is a processing error if the data is negative." which is admitting that BCD can be used with signed types. IBM DFDL currently implements the table in the description of binaryNumberRep, and so allows BCD for integer and decimal regardless of decimalSigned, but does not allow long, short, int, byte. Agreed to leave rules as they are today. 3. m229 - textStandardZeroRep - should this allow %ES; as one of the list of possibles? SMH: Decided not to allow %ES; because it adds some complexity to the 'empty representation' processing rules, in the same way that xs:string and xs:hexBinary do. Can always make an element required and use default of 0. 4. m236 - is V (virtual decimal point position) and also P allowed in the textNumberPattern for double and float types? SMH: Post-call investigation: Errata 2.80 says they are allowed, but not in conjunction with E, @ and * symbols. This is reflected in BNF as subpattern := prefix? ((number exponent?) | vpinteger) suffix?. This may be too restrictive. COBOL supports 'external floating point' numbers which contain +/-, E and . characters, and so are DFDL text standard floats. See http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2F… and its child topic for some examples of these floats. Note that one example shows use of V symbol. Also experimentation with the IBM DFDL COBOL importer shows that both V & P symbols in the PIC clause are allowed. This suggest that the BNF should be revised. It be should be noted though that IBM DFDL COBOL imports these floats as xs:string and makes no attempt to convert to a number. Need to ask Suman why this is. 5. m237 - Do we check that the various symbols used for infinity, digits, grouping separators, decimal separators are properly distinct to allow parsing? E.g., that the decimal separator and grouping separator aren't the same, and that the positive and negative pattern variants are distinguishable? ICU library supposedly doesn't do this checking. Do we state this is an SDE in DFDL. If so then is this checking required? Can we make it possible for implementations to not check somehow? Other grammar ambiguity situations like separator and terminator being ambiguous are specifically NOT checked for, because determining if a grammar is ambiguous is hard or undecidable, and would have to be done at runtime because delimiters can be run-time computed. Buf for the syntax components of text numbers do we require checking or not? SMH: Post-call investigation: IBM DFDL gives an error if the decimal & grouping separators are the same, but does not check any of the other characters for uniqueness. (Andy - please can you check what ICU does if you set various of the text number characters to be the same value, eg, decimal sep, grouping sep, exponent and (for floats) Nan and Inf reps ?, in both strict & lax modes) ICU does not appear to check when setting the values for symbols if they overlap. It has precedence rules when matching (eg, match NaN before Inf, match decimal separator before grouping separator). It gives parse errors if the data does not work with its rules. To avoid having to understand these rules, it was agreed that decimal separator, group separator, exponent rep, Inf, NaN and zero rep must all be distinct, schema definition error otherwise. Noted that if decimal separator, group separator and exponent rep are expressions, this checking must be deferred until parsing/unparsing. 6. m370 - multiple PoU resolutions: If you have initiatedContent, AND a choiceBranchRef, AND a discriminator all on the same element, and there are 3 enclosing nested PoU, which one controls which? Precedence is the issue. Or..... do we really need to allow this? Why don’t we just disallow this kind of piling-on of complexity and make the user choose which PoU resolution technique they want? SMH: a) initiatedContent 'yes' on a choice/sequence, and a discriminator on a child of the choice/sequence. Allowed at the moment, and it is quite possible that users of IBM DFDL have this combination. Agreed not to make this an SDE (it's a difficult check to get right anyway due to the nature of discriminator placement), nor to issue a warning either. b) initiatedContent 'yes' and choiceBranchRef together on a choice. Yes one is redundant. Agreed to make the combination an SDE. Noted that might force users to explictly set initiatedContent to 'no' if initiatedContent 'yes' in scope. c) choiceBranchRef on choice and a discriminator on a child of the choice. Agreed not to make this an SDE (same reasoning as a) above), nor to issue a warning either. 7. m396 - is BCD representation a mandatory feature, or optional? SMH: BCD calendars and BCD numbers are independent optional features. Will update errata document to make this clear. 8. m398 - portability at risk if subset processors ignore properties they don't implement. We relaxed this from a more rigid policy, and now allow subsets to not validate properties they don't implement. However, is there a better compromise, e.g., require a warning about all unimplemented/unrecognized properties? E.g., dfdl:textBiDi='no' yields SDE "unrecognized property 'textBiDi' with value 'no'. SMH: Agreed that this had been relaxed too much and a warning MUST be issued by implementations. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

question relevant to evaluation timing - short form
by Mike Beckerle 31 Oct '12

31 Oct '12

Can I do this? <sequence> <annotation><appinfo...> <dfdl:setVariable ref="foo" .../> <dfdl:sequence initiator="{...expression involving foo...}"/> <dfdl:newVariableInstance ref="foo" value="{...some expression...}"/> <dfdl:sequence terminator="{...expression involving foo...}"/> </appinfo></annotation> </element> See how I am spreading out the evaluation of the runtime-valued properties with the setVariable and newVariable stuff. This works so long as we evaluate everything in schema definition order. Now let me throw one more wrench (spanner for the UK folks) in the works. I'm going to add a short form annotation: <sequence dfdl:separator="{... expression...}"> <annotation><appinfo...> <dfdl:setVariable ref="foo" .../> <dfdl:sequence initiator="{...expression involving foo...}"/> <dfdl:newVariableInstance ref="foo" value="{...some expression...}"/> <dfdl:sequence terminator="{...expression involving foo...}"/> </appinfo></annotation> </element> Now, a short form annotation is equivalent to a long form conversion thereof, but in this case is it this one??? <sequence> <annotation><appinfo...>  <dfdl:sequence separator="{...expression ...}"/>  <dfdl:setVariable ref="foo" .../> <dfdl:sequence initiator="{...expression involving foo...}"/> <dfdl:newVariableInstance ref="foo" value="{...some expression...}"/> <dfdl:sequence terminator="{...expression involving foo...}"/> </appinfo></annotation> </element> Or... is it this one: <sequence> <annotation><appinfo...> <dfdl:setVariable ref="foo" .../> <dfdl:sequence initiator="{...expression involving foo...}"/> <dfdl:newVariableInstance ref="foo" value="{...some expression...}"/> <dfdl:sequence terminator="{...expression involving foo...}"/>  <dfdl:sequence separator="{...expression ...}"/>  </appinfo></annotation> </element> I would argue I prefer the first one, because makes the lexical order of appearance consistent with evaluation order. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 4

Document Migration GridForge -> Redmine
by Steve Hanson 31 Oct '12

31 Oct '12

To all DFDL WG mailing list members. As per the email below, the DFDL WG is moving its operations to the new OGF Redmine repository. The OGF GridForge repository will not be deleted but should no longer be used. All documents including WG call minutes have been migrated to Redmine. The URL is http://redmine.ogf.org/projects/dfdl-wg. Please note that if you had a user account with GridForge then you must create a new account for Redmine. The URL for this is http://redmine.ogf.org/account/register. Please don't bookmark any DFDL WG documents in Redmine just yet. The migration has placed our existing folder structure into a 'gridforge' folder, and I will be restructuring over the coming week. Please let me know if you have any problems. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 31/10/2012 12:08 ----- From: Andre Merzky <andre(a)merzky.net> To: wg-all(a)ogf.org, Date: 31/10/2012 11:29 Subject: [wg-all] Document Migration GridForge -> Redmine Sent by: wg-all-bounces(a)ogf.org Dear OGF Groups, Most groups have by now migrated their operations from GridForge to Redmine. Over the last days, we have been migrating documents from the various GridForge projects to the respective Redmine projects. We plan a similar migration of tracker items (sorry, we can't give a timeline for that, yet). A couple of notes about the document migration: - The documents (incl. the folder structure) have been moved into a separate folder on your redmine project's document module. That folder is named 'gridforge'. That way, the migrations would not collide with existing redmine documents. Note that you can move folders and documents to other folders, as needed. - For some GridForge projects, we do not have equivalent redmine projects. The respective documents have been moved to a separate archival project: http://redmine.ogf.org/dmsf/migration-archive Please let us know if you want to have access to that project, (it is publicly readable though), and if you want to have parts of that archive migrated to specific redmine projects. - We are aware of some problems, but decided not to delay the migration as several groups are waiting for the documents. Please let us know if you encounter faulty migrations. In particular, you may encounter folders which cannot be opened. Please pop me a note, and I'll make sure that this is getting fixed timely. Those problems should be rare though (so far 2 documents made trouble, out of O(100) checked). It is very likely that for many redmine accounts, permissions are not yet set correctly. In particular, chair and secretary roles may not yet be assigned to your account. Please let me know when you have trouble with permissions, and I'll flip the preciousss bitsesss... Best, Andre. -- Nothing is really difficult... -- The wg-all mailing list is an umbrella list for all OGF mailing lists. If you are subscribed to any OGF mailing list, you will receive mails to wg-all. You can unsubscribe from this list on http://www.ogf.org/cgi-bin/perl/unsubscribe-wg-all.pl wg-all mailing list wg-all(a)ogf.org https://www.ogf.org/mailman/listinfo/wg-all Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance)
by Steve Hanson 30 Oct '12

30 Oct '12

Regarding the self-axis, the rule is simple. If the expression contains any self-axis references it is evaluated after. Otherwise it is evaluated before - even if an absolute or relative path turns out to be to the element in question. This means it is easy to decide when to evaluate it. (I meant to say that originally). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 30/10/2012 13:34 Subject: Re: [DFDL-WG] DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance) I agree with most of that. I certainly would want newVariableInstance to be evaluated before a complex element is parsed. The complication with the setVariable rule is that an absolute path may be equivalent to a path which uses the self-axis but it is not possible, in general, to determine this by static analysis of the xsd. Furthermore, one XPath expression can contain zero to many path refs, any or all of which might ( or might not ) start with the self-axis. I can only see two possible solutions to this. - Asserts/discriminators/setVariable are always executed after the component on which they are positioned has been fully parsed. With the exception of asserts/discriminators with test=pattern. - Asserts/discriminators/setVariable have a 'timing' flag that defaults to 'after'. If it is set to 'before' and the expression does not evaluate successfully then it is a schema definition error. This would allow earlier rejection of the wrong branch in the model ( and therefore more efficient parsing ) in cases where the expression only refers to items that have already been parsed. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 30/10/2012 13:12 Subject: Re: [DFDL-WG] DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance) Sent by: dfdl-wg-bounces(a)ogf.org Thanks for writing this up, I have a couple of comments from thinking about this in parallel, and reading the write-up. - In 2), replace element with object as applies to sequence/choice as well - newVariableInstance should be evaluated before the object is parsed (note: self-axis is not allowed) - setVariable should be evaluated before the object is parsed unless it uses self-axis - setVariable should be evaluated after the object is parsed if it uses self-axis - consider only allowing self-axis in setVariable for simple elements/types - add statement about early evaluation of non-pattern asserts/discriminators if that can be done (as per IBM implementation) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 29/10/2012 22:54 Subject: [DFDL-WG] DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance) Sent by: dfdl-wg-bounces(a)ogf.org I'll write this up like an errata, but this is for discussion of whether we believe this is clear and complete. ------------------------------------- Glossary: DFDL Statements are the annotation elements dfdl:assert, dfdl:discriminator, dfdl:setVariable, and dfdl:newVariableInstance. Errata: Locations where DFDL Statements are allowed to appear are extended to also include Global Element Declarations, and on Simple Type Definitions. Errata: Clarification about discriminators: Discriminators exclude Assertions even when combining across references. Beyond the stipulation that there can be only one dfdl:discriminator at any annotation point of the DFDL schema, there are further constraints. A single dfdl:discriminator annotation may appear on an element reference, or on the global element declaration it refers to, or on the simple type appearing immediately within or referenced from the global element declaration. But only one of those places. In addition, if a discriminator occupies one of those places, then no dfdl:assert annotations may appear in any of those locations. A dfdl:discriminator annotation may appear on a group reference or on the model group within the global group definition it refers to. But only one of those places, and similarly, if a discriminator appears in any of those places, then no dfdl:assert annotations may appear in any of those locations. Errata: Clarification about the execution order of DFDL Statements when they appear on an element reference or element declaration. DFDL Statement annotations for a given schema component are executed as follows: 1) all relevant DFDL statement annotations are gathered to form a single list which preserves schema-definition order. For a simpleType definition, the DFDL statement annotations found immediately on it are kept in schema-definition order, and are appended to the end of a list of those from any base simpleType definition. For an element declaration having simple type, the DFDL statement annotations found immediately on the declaration are appended to the end of the list of those from its simple type. For an element reference, DFDL statement annotations found immediately on the element reference are appended to the end of the list of those from the global element declaration it references. 2) given the combined list, the annotations are executed as follows: 1. before any parsing of the element, a dfdl:discriminator with testKind="pattern" is executed. 2. if there is no discriminator, then before any parsing of the element, all dfdl:asserts (there could be several) with testKind="pattern" are executed in the order they appear in the list of DFDL statements. 3. The element itself is parsed, or its inputValueCalc property is evaluated to create its value. 4. all newVariableInstance annotations are executed and new variables are placed into scope for the duration of these remaining steps. The statements are executed in the order they appear in the list of DFDL statements. 5. all setVariable annotations are executed. The statements are executed in the order they appear in the list of DFDL statements. 6. if a discriminator is present it is executed 7. if no discriminator is present, then assert annotations can be present, and they are executed. If there are multiple assert annotations the statements are executed in the order they appear in the list of DFDL statements. If the element reference or local element declaration is an array, then this evaluation is repeated for each occurrence of the array. Discussion: The above allows the default expressions associated with any statement to refer to the value of the element itself as "." However, there's this anomaly of syntax where things don't seem right: <dfdl:newVariableInstance ref="foo:bar" default="{....some expression ...}"/> creates a new variable for the scope of the entity it annotates. It's clear what this means if this annotation is placed on a sequence or choice. For the children of that sequence/choice the new variable instance is in effect. On a simpleType or element having simple type, similarly it is clear (in that case it's a very local variable, just for the expressions in other newVariableInstance statemsnts, setVariableStatements, and discriminators and assertions). The rub: on an element declaration (or reference) when there is a complex type, its not clear. <element name="foo"> <annotation><appinfo ....> <dfdl:newVariableInstance......>  </appinfo></annotation> <complexType> <sequence> <annotation><appinfo ....> <dfdl:assert>... according to rules above this cannot refer to the newVariableInstance...</dfdl:assert> </appinfo></annotation> .... </sequence> </complexType> </element> The above rules say the newVariableInstance isn't evaluated until AFTER the element is parsed, so the assert down inside on the sequence will NOT see this new variable even tho textually, the newVariableInstance annotation looks like it would be in scope over the sequence. Possible ways to avoid this oddity without messing up evaluation order: allow newVariableInstance only on simpleType, element declarations and element references having simpleType, group references, and sequence/choice. Disallow them on element declarations of complex type or on element references to those. The schema author can always introduce an extra tier of sequence to provide the exact behavior they need, and this otherwise error-prone issue can be avoided. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-10-30
by Steve Hanson 29 Oct '12

29 Oct '12

Please find agenda for call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

2 1

new action item needed: DFDL URN specification
by Mike Beckerle 29 Oct '12

29 Oct '12

Introduction: Using URLs as identifiers has caused no end of problems. E.g., in DFDL we have http://www.ogf.org/dfdl/dfdl-1.0/ as an identifier. W3C has http://www.w3.org/2001/XMLSchema and others. W3C has badly regretted establishing this convention, as they have farms of servers that do nothing but quickly return 404 errors to save network-aware applications the network-timeout delay that would otherwise occur. There are parties interested in exploiting DFDL who want DFDL schemas to NOT contain network URLs because it simply creates a concern about network access whenever a DFDL schema is inspected/used. Pre-Proposal: The new way to do this is with URNs which would look roughly like this: urn:ogf:dfdl:standard:dfdl-1.0. The whole point is that some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. The point is that it is a name in a managed namespace which cannot be confused with a network protocol URL. OGF is already establishing urn:ogf, and an ogf subgroup has already proposed urn:ogf:network for network resources. DFDL schemas aren't network resources so we don't want to be a substructure underneath network. Some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. Summary: An action item should be to specify DFDL urn, submit to OGF as a proposed namespace, and then produce errata/spec changes to specify its use. This requires a small design activity to specify a scheme for the sub-structure of the DFDL URNs (i.e., scheme for the stuff after urn:ogf:dfdl:...) where we want standard identifiers for versions of the standard, but we probably also want a few other things (e.g., I would like a space for implementations to identify themselves, i.e., an implementation-specific sub-area within our URNs.) Our existing URLs can be compatible (deprecated) practice vs the preferred URNs. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 3

Re: [DFDL-WG] simpleType cannot contain assert?
by Mike Beckerle 29 Oct '12

29 Oct '12

The example is pretty simple. I have strings that need asserts to verify proper format. Nothing like working on a real format to put the small but critical issues into focus. I overlooked that you cannot put an assert on a global element. I read that too quickly, but there it is. I'm not sure what we were thinking here. Perhaps it was as simplistic as avoiding global constructs with ".." in the paths of expressions. But this just flies in the face of creating tidy schemas that avoid repetition of asserts or big complex expressions all over the place. And asserts are one of the very untidy things because they contain regular expressions, which are very unweildy and difficult to maintain and badly badly need to be centralized in any reasonable schema. So I'll amend my proposal: allow asserts on global element decls, and on simple type defs. That is, they are orthogonal in placement to whether the site is local or global. Here's the example: If I can put these as asserts on a simpleType, then I can abstract over them in a schema, using the same regex assert for many fields with different names. If I cannot, then I have no choice but to nest them inside a complex type, and all these simple string fields become complex types, which is adding a whole tier of elements to the schema. E.g. <simpleType name="dField" dfdl:ref="ex:dFieldDefaults"> <restriction base="xs:string"> </simpleType> then all over the schema..... <sequence dfdl:ref="dFieldListFormat"> <element name="Foo" type="ex:dField> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert testKind="pattern" testPattern="((\p{L}|\d|\.|,|$|$|\?|!|@|#|\$|%|\^|&|\*|=|_|\+|\[|\]|\{|\}|\\|"|'|;|<|>|~|\|)|\-)(((\p{L}|\d|\.|,|$|$|\?|!|@|#|\$|%|\^|&|\*|=|_|\+|\[|\]|\{|\}|\\|"|'|;|<|>|~|\|)|\-)| )*((\p{L}|\d|\.|,|$|$|\?|!|@|#|\$|%|\^|&|\*|=|_|\+|\[|\]|\{|\}|\\|"|'|;|<|>|~|\|)|\-)|(\p{L}|\d|\.|,|$|$|\?|!|@|#|\$|%|\^|&|\*|=|_|\+|\[|\]|\{|\}|\\|"|'|;|<|>|~|\|)" message="Assertion failed for data_code" /> </xs:appinfo> </xs:annotation> </element> .... repeat ad nauseum for all other elements of type dField, which is EVERYTHING practically. This absolutely flies in the face of any good principles of abstraction. I want that regex, and the characteristics of data that have to obey it, captured in one place. I also really do not want to have to turn every element like Foo, which is logically just a string, into a complex type. What I need to be able to do is rotate that assertion over into the definition of dField. I can't do it with a pattern facet, because that doesn't affect parsing, and this assertion needs to guide speculative parsing. (Note: XML Schema allows patterns on simpleType defs and so allows proper abstraction of simple types that use pattern validation. DFDL is currently not consistent with this way of abstracting.) Now, currently the spec says I can't put an assert on a global element either, so I can't fix the above issue by doing this: <element name="dValue" type="xs:string"> ....big mombo assert with regex ... NOT ALLOWED HERE EITHER </element> <complexType name="dField"> <sequence> <element ref="ex:dValue"/> </sequence> </complexType> <sequence dfdl:ref="dFieldListDefaults"> <element name="Foo" type="dField"/> .... But I could push the assert down inside the complex type definition, where it would be on a local element decl, and if everywhere I share use of that complex type, then I can centralize the regex. What is this restriction possibly achieving? The assertion is still "trying to be on a global definition", we've just prevented it from being in a convenient place for the modeler. Also, as soon as you force me to use a complex type, now I'm stuck with the complex type restrictions on nillability, which are insufficient for my needs, so now I can't take advantage of mapping data containing non-empty nil indicators to xsi:nil nillability. I have to create a <element name="noValue" ....> and get into using choices, etc. I should note that precedent in the spec already allows asserts on a sequence/choice that is a global group definition and global group references. I believe this is just a happy accident of the XML Schema object model i.e., the fact that the sequence/choice object isn't the same object as the global definition object, it is contained inside the global group def object. The point: there is already a precedent for having to combine the lists of assertions from both defines and references to them. I think the rule is simple, asserts on references go after the asserts on the defines, but otherwise it is just merging the lists onto the reference schema component to be executed at the reference. The concern about global objeccts, that is about things having expressions/properties that can't be resolved isn't one that keeping assertions off global defs/decls will help. A global def/decl can contain a subset of DFDL properties such that it is useless in DFDL outside of some complementary referencing context. This is inherent in DFDL, and any system that allows separation of concerns when describing something complex including XML Schema itself, which has global types, groups, etc. all of which are useless without their referencing contexts. I would close by noting that this change, (allowing these additional locations for asserts), is backward compatible with our current spec, because it only adds new places these assertions are allowed. The right fix here: DFDL 'statements' (setVariable, assert, discriminator, even newVariableInstance) should be allowed on simpleType and on global element decls. Even newVariableInstance is useful there. A new variable is created, used in expressions in asserts/discriminators/setVariable/ other newVariableInstance, and then it goes out of scope immediately at the end of the element. The newVariableInstance is in fact our only way of really controlling the complexity of an expression. It lets you create a big complicated expression out of several variables each of which is bound to a sub-calculation, and so it is useful even when the scope is just the asserts/discriminators/setVariables/newVariableInstance statements of a single simple-typed element. newVariableInstance basically lets us create local variables for use in our expression language. Normal XPath doesn't have this. ...mike On Fri, Oct 26, 2012 at 4:11 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike, it's not an oversight, I'm sure it is for the same reason that you > can't put an assert on a global element. I think the rationale is that > asserts (and discriminators) are only allowed at annotation points where > all properties can be resolved. > > Please can you show an example of what you want to achieve? > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org, > Date: 25/10/2012 17:47 > Subject: [DFDL-WG] simpleType cannot contain assert? > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > Is this just an oversight? > > I find it very tedious to model if I can't put the asserts on the > simpleTypes so that I don't have to repeat them over and over in the model. > > The composition rule here is simple. If an element has asserts, and its > simple type has asserts, both are executed, with the element's asserts run > after the simpleType's asserts. > > ...mike > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 3

OGF DFDL WG Call Minutes 2012-10-23
by Steve Hanson 29 Oct '12

29 Oct '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is still at v010 here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0