- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Agenda 2014-04-15
by Steve Hanson 14 Apr '14

14 Apr '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13240?download= Back to usual day and time. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Clarification on escape schemes - lookahead distance past an escape character
by Mike Beckerle 14 Apr '14

14 Apr '14

Consider this infix separator situation: <sequence dfdl:separator="/%WSP*;/"> <element name="x" type="xs:string"/> <element name="y" type="xs:string" minOccurs="0"/> </sequence> Length kind is delimited Suppose the escape character is "/" Suppose the data is "foo//bar" Should the above be (a) <x>foo/bar</x> or (b) <x>foobar</x> The problem is this. In order to produce <x>foobar</x> you have to recognize that the second / isn't in fact the start of a delimiter, and that requires lookahead for the entire possible length of the delimiter, and that's unbounded because of the %WSP*; in it. I believe the semantics of escape characters should not require looking at more than the next character after the escape character, but this will result in the escape character behaving as if it escapes any single character that follows it, not only the first character of a delimiter. Is the right behavior here clear? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

2 1

draft r08 of spec.
by Mike Beckerle 14 Apr '14

14 Apr '14

I have posted draft spec r08 http://redmine.ogf.org/dmsf_files/13239?download= I updated the references so they are hyperlinks now. Unfortunately, getting MS Word to do this easily required changing them from symbolic refs e.g., [XMLSch] to numeric ones e.g., [7]. I also updated affiliations for contributors where I could find their current affiliations. 3 are still unknown to me: Dave Glick, USA, Tara Talbot, PNNL, and Steve Marting, Progeny, USA. I am unable to find these people by web search or LinkedIn. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

1 0

Internal WG DFDL Spec r07 - incorporates all errata/pub-comment-changes through experience doc 1 - v1.2
by Mike Beckerle 11 Apr '14

11 Apr '14

This draft is current with all errata and pub-comment changes through errata 1 doc v1.2 I didn't attach it. You can get it from redmine http://redmine.ogf.org/dmsf_files/13236?download= DFDL Spec 1.0.4-r07 - GFD-P-R.207<http://redmine.ogf.org/dmsf_files/13236?download=>actual filename is draft-gwdrp-dfdl-v1.0.4-r07.docx If you edit, please roll the r07 to r08 etc. Known Remaining work: * One public comment has not yet been addressed which is changing all the [REFERENCES] into navigable xrefs. I think this is likely best done by making an actual MS-Word Bibliography and then using it. * Need to update the contributor affiliations at the back, as many of these people have changed affiliations by now. * Action items still open * Current discussion threads. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy<http://www.ogf.org/About/abt_policies.php>

1 0

OGF DFDL WG Call Minutes 2014-04-11
by Steve Hanson 11 Apr '14

11 Apr '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13233?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Action 247: Updated experience (errata) documents
by Steve Hanson 11 Apr '14

11 Apr '14

Updated after WG call on 11th April to include errata from this and other calls. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/04/2014 17:24 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 12/03/2014 10:38 Subject: Fw: Action 247: Updated experience (errata) documents Updated after WG call on 11th March, where it was noted that action 252 had not been fully covered. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 12/03/2014 10:37 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 07/03/2014 14:53 Subject: Action 247: Updated experience (errata) documents Please find attached the updated DFDL experience documents, which reflect all resolved public comments to date, and any other closed errata to date. Placeholder errata have been added for the 4 outstanding public comments. Public comments have been updated to reference their errata, or to say that no errata were needed, or in some cases marked 'Closed' as there was no action on the spec. The accompanying attachment is for Mike's benefit when updating the spec for public comment 25 (erratum 2.100). Please review for next DFDL WG call. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU #### gwde-dfdl-experience-2-v1.1.docx moved to MyAttachments Repository V3.8 () on 26 March 2014 by Steve Hanson. #### Elements of Specified Length-002.docx removed & duplicate added to MyAttachments Repository V3.8 () on 26 March 2014 by Steve Hanson. Original attachment record is here (). #### gwde-dfdl-experience-1-v1.1.docx (ATTPOZKQ.docx) moved to MyAttachments Repository V3.8 () on 26 March 2014 by Steve Hanson. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] lengthUnits bits not allowed for complex type - Unnecessary restriction on length units?
by Steve Hanson 11 Apr '14

11 Apr '14

Agreed that for complex elements use lengthKind 'implicit' and for simple BLOBs model as unsigned integer of specified bit length. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 26/03/2014 10:22 Subject: Re: [DFDL-WG] lengthUnits bits not allowed for complex type - Unnecessary restriction on length units? Mike, I think the expectation was that such fields would be modelled as unsigned integers. The issue then becomes one of max size. From your MIL spec quote below, it looks like this type is dfdl:lengthKind 'prefixed', with dfdl:prefixLengthIncludesPrefix 'yes' ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 25/03/2014 18:08 Subject: Re: [DFDL-WG] lengthUnits bits not allowed for complex type - Unnecessary restriction on length units? Sent by: dfdl-wg-bounces(a)ogf.org I know on the call I agreed that complex types with length bits wouldn't be needed, but I rechecked the format I am trying to implement, which is MIL-STD-2045-47001D (which is public) and I found this (emphasis mine): 5.6.42 Group Size field. This field shall be a 12-bit binary number indicating the size, in bits, of the Future Use Group in which this field is contained. A value of “0” should not be used for this field. If the parent group is specified present then this child field is mandatory. Turns out that these things are all just binary blobs in this spec document. There is no sub-structure provided as these are "Future Use", but it's always possible they will get specified or already are and I am just unaware of the document which gives the format of some of these. There are a flock of related standards to this one. Most are, unfortunately, not publicly available. I am at a bit of a loss how to model these bit-length fields. HexBinary type allows length only in bytes to be compatible with XSD hexBinary which has a string-of-hex representation. I suppose I could use an xs:nonNegativeInteger with up to 4096 bits as the blob. Is there any other viable option? (array of bit is not viable) My concrete suggestion: this data format is the same one that motivates the proposed dfdl:bitOrder property. I suggest that dfdl:bitOrder, along with any other issues needed to implement this standard, all be addressed at once when I've had time to complete an initial implementation in the daffodil code base. Lifting the restriction on these lengthUnits bits for complex types may well be required. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Mar 25, 2014 at 9:01 AM, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> wrote: In section 12.3.7.3 we have this sentence: "The dfdl:lengthUnits may be 'bytes' or 'characters' and it is a schema definition error otherwise. " Does anyone recall why we have this? I have data formats which are bit oriented and there are complex types which would naturally not be a multiple of 8 bytes long, e.g., 1 bit field, 3 bit field, 10 bit field, 6 bit field = 20 bits. I can't think of any reason for this restriction other than to explain how fillByte is used to fill in unused bits. But I think we can say that any unused bits are filled in with bits from the fillByte, and we don't have to be specific about which bits from the fillByte. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] alignment questions
by Steve Hanson 11 Apr '14

11 Apr '14

Erratum taken for the updated alignment fill paragraph. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, dfdl-wg-bounces(a)ogf.org Date: 26/03/2014 16:12 Subject: Re: [DFDL-WG] alignment questions Mike Looking at leadingSkip, spec says "A non-negative number of bytes or bits, depending on dfdl:alignmentUnits, to skip before alignment is applied" (my emphasis). And this is matched by the grammar which says: " LeadingAlignment = LeadingSkip AlignmentFill" Together these imply that any application of leadingSkip takes place before the calculation of the AlignmentFill region, and so is not itself aware of alignment. Same deal for trailingSkip. For it to be otherwise, I think the grammar would have to include 'leadingSkipFill' and 'trailingSkipFill' regions. ------------------------------- While researching this, I noted that the spec says: "The length of the AlignmentFill region is measured in bits. If alignmentUnits is 'bytes' then we multiply the alignment value by 8 to get the bit alignment, B. If the current position (first position after the end of the previous element) value is bit position N, then the length of the AlignmentFill region is the smallest non-negative integer L such that (L + N) mod B = 1. The position of the first bit of the aligned element is P = L + N." I read that as saying byte alignment moves you on to a byte boundary. Fine, but it doesn't say anything about the effect of leadingSkip. It also uses 'element' instead of 'component'. And the start of a component is not always straight after the end of the previous component - think about the first component in a parent where parent has leading framing. I think this should be: "The length of the AlignmentFill region is measured in bits. If alignmentUnits is 'bytes' then we multiply the alignment value by 8 to get the bit alignment, B. If the position in the data stream of the start of the AlignmentFill region is bit position N, then the length of the AlignmentFill region is the smallest non-negative integer L such that (L + N) mod B = 1. The position of the first bit of the aligned component is P = L + N." Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, Date: 26/03/2014 15:01 Subject: [DFDL-WG] alignment questions Sent by: dfdl-wg-bounces(a)ogf.org Are leadingSkip always byte aligned, or are they N * 8 bits added on. Similarly trailingSkip E.g., <element name="x" type="xs:int" dfdl:lengthKind="explicit" dfdl:length="1" dfdl:lengthUnits="bits" dfdl:trailingSkip="10" dfdl:alignmentUnits="bytes" dfdl:alignment="1"/> So, after this element am I at bit 81 or bit 88? In other words, does the trailingSkip region get aligned per size of alignment unit, or does alignment really only happen at the beginning of an element. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Action 254: Updated words for zoned numbers
by Steve Hanson 11 Apr '14

11 Apr '14

Agreed to take an errata and adopt the words below. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 10/04/2014 14:25 Subject: Action 254: Updated words for zoned numbers Updated description for textZonedSignStyle to reflect that not all EBCDIC encodings use the same characters for the code points. Text in red is from public comment 116. ---------------------- Specifies the code points that are used to overpunch the sign nibble when the encoding is an ASCII-derived character set.encoding. The location of this sign nibble is indicated in the dfdl:textNumberPattern. This property is applicable when dfdl:textNumberRep is 'zoned' Used only when dfdl:encoding is an ASCII-derived character set encoding. The encoding must provide the character to single byte code point mapping used by the specified value of dfdl:textZonedSignStyle, as stated below. Valid values 'asciiStandard', 'asciiTranslatedEBCDIC', 'asciiCARealiaModified', and 'asciiTandemModified' Which characters are used to represent 'overpunched' (included) positive and negative signs, varies by encoding, Cobol compiler and system. The code points are fixed for EBCDIC systems but not for ASCII. In EBCDIC-based encodings, code points 0xC0 to 0xC9 or 0xF0 to 0xF9 represent a positive sign and digits 0 to 9 (typically characters '{ABCDEFGHI' or '0123456789'), and code points 0xD0 to 0xD9 or 0xB0 to 0xB9 represent a negative sign and digits 0 to 9 (typically characters '}JKLMNOPQR' or '^£¥·©§¶¼½¾ ' ). On parsing both ranges will be accepted. On unparsing the range 0xC0 to 0xC9 will be produced for positive signs and the range 0xD0 to 0xD9 will be produced for negative signs. asciiStandard: ASCII characters '0123456789' represent a positive sign and the corresponding digit. (Sign nibble for '+' is 0x3, which is the high nibble of these code points unmodified.) ASCII characters 'pqrstuvwxy' represent negative sign and digits 0 to 9. (Code points 0x70 to 0x79) asciiTranslatedEBCDIC: The overpunched character is the ASCII equivalent of the typical EBCDIC above. So the characters '{ABCDEFGHI' still represent a positive sign and digits 0 to 9. (These are code points 0x7B, 0x41 through 0x49). The characters '}JKLMNOPQR' still represent negative sign and digits 0 to 9. (These are code points 0x7D, 0x4A through 0x52). This case comes up if EBCDIC zoned decimal data is translated to ASCII as if it were textual data.) asciiCARealiaModified: In this style, the ASCII characters '0123456789' represent positive sign and digits 0 to 9 as in standard. However, ASCII characters from code points 0x20 to 0x29 are used for negative sign and the corresponding decimal digit. This doesn't translate well into printing characters. These characters include the space (' ') for zero, characters '!"#$%&' for 1 through 6, the single quote character "'" for 7, and the parenthesis '()' for 8 and 9. asciiTandemModified: In this style the ASCII characters '0123456789' represent positive sign and digits 0 to 9, but code points 0x80 to 0x89 are used to represent negative sign and a digit. There are no corresponding code points in the standard ASCII encoding since these values are all above 128 (decimal). This means the resultant bytes are not code points in standard ASCII, so the modeller must specify an encoding like ISO-8859-1 in order for such zoned decimals to parse without an encoding error. (Note that neither ISO-8859-1 encoding nor Unicode have assigned glyphs for these code points. They are considered control characters.) --------------------- Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: DL11A #3: (Question)[1](Open)(English)need clarification
by Steve Hanson 11 Apr '14

11 Apr '14

Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 11/04/2014 15:36 ----- From: Steve Hanson/UK/IBM To: Kevin Safford/UK/IBM@IBMGB, Cc: Alex Wood1/UK/IBM@IBMGB, "Mike Beckerle" <mbeckerle(a)tresys.com> Date: 06/03/2014 16:32 Subject: Re: Fw: DL11A #3: (Question)[1](Open)(English)need clarification Kevin 1) When the dfdl:initiator property is a list of values, and the data matches more than one value, then the longest value is the one selected. Example: data is aaa12345, dfdl:initiator is "aa aaa", then aaa is the match as it is longest. However if the data matches more than one value and the values have the same length, then the one that is declared earlier in the list is selected. I have to say that I am struggling to come up with an example where it actually makes a material difference, because if it matches multiple of the same length, then surely they must be equivalent anyway? Maybe white space has an effect here. Mike Beckerle crafted these words, perhaps he can give an example. (Note that same words occur dfdl:terminator and dfdl:separator). 2) Agree, and have added to the Minor Editorial public comment on Redmine. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Kevin Safford/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Alex Wood1/UK/IBM@IBMGB, Date: 06/03/2014 14:40 Subject: Fw: DL11A #3: (Question)[1](Open)(English)need clarification Hi I need some input from you chaps on this translation query. Could you suggest a rephrase, or explanation of point 1? Point 2 I assume is a typo for precede. It's present in the spec. Can you confirm? Regards Kevin Safford Information Developer I work Mondays to Thursdays only. Phone: 44-1962-819217 E-mail: SAFFORDS(a)uk.ibm.com My web site: sites.google.com/site/kevinsafford Find me on: Hursley Park Hursley, SO21 2JN United Kingdom ----- Forwarded by Kevin Safford/UK/IBM on 06/03/2014 14:37 ----- From: tctadm(a)cn.ibm.com To: undisclosed-recipients:;, Date: 06/03/2014 10:01 Subject: DL11A #3: (Question)[1](Open)(English)need clarification *** This mail is sent by TCT system automatically. Please do NOT reply to this ID. *** Click here to read the article and response: https://www.ibm.com/software/globalization/translationcommunications/BaseSe… Project code DL11A Project admin terrychu Article subject need clarification Article type Question Category name English/Source files - PII,DL11AAAP002_Tooling Testcase/Filename/IU plugins/com.ibm.dfdl.help/f1/dfdlPorperties.xml Current status Open Priority level 1 Stop-Ship English:n/a Originator English Opened by oya.argun(a)superonline.com / Trans. Vendor, Turkey / English (oyargun) 2014-03-06 10:00:25 Hello, 1)The initiator with the longest match is the one that is selected as having been 'found', with length-ties being resolved so that the matching initiator is selected that is first in the order written in the schema. I am afraid this string is very complex, please re-phrase it if possible or provide a comprehensive explanation. 2) In the following string I could not quite understand this phrase: 'or the dfdl:escapeEscapeCharacter does not proceed the dfdl:escapeCharacter'. What is meant by 'proceed' here?" On parsing the dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed from the data, unless the dfdl:escapeCharacter is preceded or the dfdl:escapeEscapeCharacter does not proceed the dfdl:escapeCharacter. Thanks in advance Oya ------------------------ Welcome to the TCT website: ------------------------ https://www.ibm.com/software/globalization/translationcommunications/index.… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0