- dfdl-wg - lists.ogf.org

DFDL based application released
by Michele Zundo 06 Apr '12

06 Apr '12

Dear all, you might be interested to know that yesterday we have freely released an application for visualizing binary satellite data (compliant to CCSDS standard) which based on DFDL. The satellite data binary structure for a number of ESA Earth Observation Mission has been described using DFDL schemas and a java parser implemented which allow the application to access all different data fields. The binary DFDL schemas can be also edited by the user to describe custom type of satellite data (i.e. changing the structure) without changing the application. We are considering in future to release the java Read API stand alone for everybody to use and build their own applications. The release message with pointers to URL to download the application is attached here below: ***** Dear all, please be informed today we have released the (Space-to-Ground) S2G Data Viewer which can be freely downloaded from : http://eop-cfi.esa.int/S2G/s2g.html S2G data Viewer is a modern, multi platform, extendable application to inspect CCSDS data (CADU, TF/VCDU and SP) and example screenshot attached below. The application is currently configured to support the following ESA Earth Observation Mission (including the custom Source Packet Data Header): Sentinel-1 Sentinel-2 Sentinel-3 Swarm EarthCARE and it can be extended by the user to support different type of data or new missions. I invite you to download and use it to support your AIV activities and, if necessary, to send any feedback to s2g(a)eopp.esa.int ****** ----------------------------------------- Mr. Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: michele.zundo(a)esa.int This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email.

2 3

OGF DFDL WG Call Minutes 2012-04-05
by Steve Hanson 05 Apr '12

05 Apr '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is still at version 8 and may be found via https://forge.ogf.org/sf/projects/dfdl-wg Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] Issue 168 - Re: Fw: Escape scheme properties that can be expressions
by Steve Hanson 05 Apr '12

05 Apr '12

Agreed on DFDL WG call on 5th April 2012 to take an errata to improve the wording as below, but not to disallow relative exprs in defineFormat or defineEscapeScheme, as it prejudges usage. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: dfdl-wg(a)ogf.org Date: 05/04/2012 12:40 Subject: Re: Issue 168 - Re: [DFDL-WG] Fw: Escape scheme properties that can be expressions Mike I think the wording in 6.3.2 applies specifically to the anonymous dfdl:format. I don't think it was intended to apply to dfdl:format inside dfdl:defineFormat nor to dfdl:escapeScheme inside dfdl:defineEscapeScheme, presumably due to the reasons we discussed on last week's call - namely that by using dfdl:ref they can be applied directly to a specific object. Of course, dfdl:ref can be used on the anonymous dfdl:format too, So the 6.3.2 words should have said: "In the dfd:format annotation on the xs:schema, a DFDL property either declared locally or obtained via dfdl:ref and which uses a DFDL expressions to provide its value MAY NOT contain relative paths". So I think we still need to decide on dfdl:format inside dfdl:defineFormat nor to dfdl:escapeScheme inside dfdl:defineEscapeScheme. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Cc: dfdl-wg(a)ogf.org Date: 03/04/2012 15:45 Subject: Issue 168 - Re: [DFDL-WG] Fw: Escape scheme properties that can be expressions In the absence of a compelling reason to allow relative paths here, I think the conservative choice is to retain this rule from 6.3.2. However, I did think about this and my evaluation on this topic is that allowing relative paths makes things just a little more complicated analytically to insure one gets good performance, but as DFDL does not allow recursive types, it is a relatively simple analysis to convert any relative path use in DFDL into absolute paths. So I am not worried about there being performance impact. ...mikeb On Mon, Apr 2, 2012 at 7:30 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: We took action 168 on the last call to discuss the issue of relative paths in expressions for properties in global formats. 168 Relative path expressions in global annotations (Mike) 28/03: Mike to check that allowing relative paths does not cause performance problems for an implementation. 05/04: Just seen this in section 6.3.2. of the spec: "DFDL expressions that are used to provide the value of DFDL properties in the dfd:format annotation on the xs:schema MAY NOT contain relative paths". Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/04/2012 12:28 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 14/03/2012 16:09 Subject: Escape scheme properties that can be expressions The spec allows the escapeCharacter and escapeEscapeCharacter of DFDL escapeScheme to be expressions, in case the characters are supplied dynamically in the data stream. However given that escape schemes are defined globally, should these properties be permitted to take relative paths in the expression? Today the spec implicitly allows this, meaning an implementation can not entirely validate an escapeScheme annotation up front, but must validate escapeCharacter and escapeEscapeCharacter at each point of use. I'm not sure it makes much sense for the path to be anything than absolute. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Fw: Action 167: textNumberPatterns with P, V, # - allowable combinations
by Steve Hanson 05 Apr '12

05 Apr '12

Agreed on DFDL WG call 5th April 2012: A pattern with a V symbol must not have # symbols to the right of the V symbol. A pattern with P symbols at the left end must have no # symbols in the pattern. A pattern with P symbols at the right end has no restrictions. ICU padding and DFDL padding/trimming are independent mechanisms, may be used in conjunction. When ICU padding is used in pattern (* symbol) it is a schema definition error if P or V symbol in pattern. When ICU significant digits in pattern (@ symbol) it is a schema definition error if P or V symbol in pattern. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Cc: Richard Schofield/UK/IBM@IBMGB Date: 22/03/2012 11:01 Subject: Fw: Action 167: textNumberPatterns with P,V, # - allowable combinations Further to discussion on the call, here is what IBM COBOL manual says about PIC P. An assumed decimal scaling position. It is used to specify the location of an assumed decimal point when the point is not within the number that appears in the data item. The scaling position character P is not counted in the size of the data item. Scaling position characters are counted in determining the maximum number of digit positions (63) in numeric-edited items or in items that appear as arithmetic operands. The scaling position character P may appear only as a continuous string of Ps in the leftmost or rightmost digit positions within a PICTURE character-string. Because the scaling position character P implies an assumed decimal point (to the left of the Ps, if the Ps are leftmost PICTURE characters; to the right of the Ps, if the Ps are rightmost PICTURE characters), the assumed decimal point symbol, V, is redundant as either the leftmost or rightmost character within such a PICTURE description. In certain operations that reference a data item whose PICTURE character-string contains the symbol P, the algebraic value of the data item is used rather than the actual character representation of the data item. This algebraic value assumes the decimal point in the prescribed location and zero in place of the digit position specified by the symbol P. The size of the value is the number of digit positions represented by the PICTURE character-string. These operations are any of the following: Any operation requiring a numeric sending operand. A MOVE statement where the sending operand is numeric and its PICTURE character-string contains the symbol P. A MOVE statement where the sending operand is numeric-edited and its PICTURE character-string contains the symbol P and the receiving operand is numeric or numeric-edited. A comparison operation where both operands are numeric. In all other operations the digit positions specified with the symbol P are ignored and are not counted in the size of the operand. This implies that the scaling should be applied as a lexical operation on the data. In other words two COBOL fields, one with PIC PP9 and value '2' and one with PP999 and value '002' do not result in the same logical number. There is an equivalence between V and P. PP999 == V99999 and 999PP == 99999V == 99999. If we consider things in these terms the reasoning is simpler. To prevent # symbol zero suppression from changing the value, rule a) must apply and there must be no # to the right of the V. That restates our rules as: a) A pattern with a V symbol must not have # symbols to the right of the V symbol. b) A pattern with P symbols at the left end must have no # symbols in the pattern. c) A pattern with P symbols at the right end has no restrictions. There is another problem though. The number can be trimmed using the pad character from either or both ends depending on justification, before applying the number pattern. If the pad character is 0 then this can also cause 0's to be lost and result in mis-application of V and P symbols. I'm not sure there is much we can do about this. Modelers need to be careful when padding/trimming that they get the justification correct. For example, we typically think of numbers as being right justified, but for a number with Ps on the left, it is effectively left justified and should be modeled as such. We added errata 2.25 which prevented trimming from leaving an empty string. I am thinking that this errata should actually say that trimming must leave at least the minimum number of digits implied by the pattern, as an extra safeguard? We mustn't disallow trimming/padding altogether as it is used to remove spaces. The ICU pad character symbol * is used to provide a pad character when the data is shorter than the pattern. This is only used to pad when unparsing, it is not used to trim. But it might be safer to disallow P and V symbols when * is used? Reading the ICU description of significant digit symbol @, explicit decimal points are disallowed. I think we should disallow P and V symbols when @ symbol is used. Errata 2.28 should be updated. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 22/03/2012 09:22 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 21/03/2012 10:34 Subject: Action 167: textNumberPatterns with P,V, # - allowable combinations Last week we agreed that disallowing text number patterns that contained a # symbol and either a P or V symbol was too restrictive. Accordingly the following rules are proposed to control when # may be used in the same pattern as P or V to ensure an unambiguous pattern. a) Pattern must not have # symbols to the right of the V symbol. b) If pattern has P symbols at the left end, then there must be as many 0 symbols adjacent to the rightmost P symbol as there are P symbols. c) If pattern has P symbols at the right end, there are no restrictions. If a) or b) are violated it is a schema definition error. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

issue 145 - name dfdl:elementAlias
by Mike Beckerle 05 Apr '12

05 Apr '12

About the name dfdl:elementAlias It suggests that this is another element name, and in other contexts than a choice it might be used as well, and in addition it suggests that the dfdl:choiceBranchRef can evaluate to the element name, or to this alias. I think the definition of all this behavior should be narrower, and the names should suggest this. dfdl:choiceBranchRef dfdl:choiceBranchID seem like the right pairing. A choice branch ref can refer to exactly and only the choice branch ID, never the element name, and this ID is used only for choice branch ref purposes. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

Issue 145 - cobol 88 example with multiple definitions per 88
by Mike Beckerle 05 Apr '12

05 Apr '12

Cobol below illustrates how the 88 statement can define more than one constant to represent a selection. E.g., the definition for Consonant below. A single "Accept Char....When Consonant" test tests the char value against the entire specification including individual enums, and the ranges specified as well. This is the mechanism I've seen used to specify that several different enumerated codes correspond to the same record format. I've only seen the multiple enumerated constants in practice, not the range specifications: IDENTIFICATION DIVISION. PROGRAM-ID. Conditions. AUTHOR. Michael Coughlan. * An example program demonstrating the use of * condition names (level 88's). * The EVALUATE and PERFORM verbs are also used. DATA DIVISION. WORKING-STORAGE SECTION. 01 Char PIC X. 88 Vowel VALUE "a", "e", "i", "o", "u". 88 Consonant VALUE "b", "c", "d", "f", "g", "h" "j" THRU "n", "p" THRU "t", "v" THRU "z". 88 Digit VALUE "0" THRU "9". 88 ValidCharacter VALUE "a" THRU "z", "0" THRU "9". PROCEDURE DIVISION. Begin. DISPLAY "Enter lower case character or digit. No data ends.". ACCEPT Char. PERFORM UNTIL NOT ValidCharacter EVALUATE TRUE WHEN Vowel DISPLAY "The letter " Char " is a vowel." WHEN Consonant DISPLAY "The letter " Char " is a consonant." WHEN Digit DISPLAY Char " is a digit." WHEN OTHER DISPLAY "problems found" END-EVALUATE END-PERFORM STOP RUN. I suggest that tests as complicated as the ones above that include range specifications should turn into normal discriminators in DFDL. But what if it was just enumerated values, e.g., I take the ranges off.... 01 Char PIC XX. 88 BaseRecord VALUE "a1", "11", "52". 88 Ext-Record VALUE "bb", "b0" This is the example I've actually seen in data. So in this case we just need the ability for the elementAlias to be a list of potential values. Or we can just be satisfied with requiring each of a1, 11, and 52 to be a separate element with a different name, so the above becomes a choice of 6 things, or we can just require a regular old discriminator in any case more general than single-value dispatch. I wanted everyone to see the example, but I actually advocate the narrowest possible definition here. I think the dfdl:elementAlias should be exactly one value only, and users should use a discriminator with a test for anything more general. ...mikeb

2 1

Issue 168 - Re: Fw: Escape scheme properties that can be expressions
by Mike Beckerle 05 Apr '12

05 Apr '12

In the absence of a compelling reason to allow relative paths here, I think the conservative choice is to retain this rule from 6.3.2. However, I did think about this and my evaluation on this topic is that allowing relative paths makes things just a little more complicated analytically to insure one gets good performance, but as DFDL does not allow recursive types, it is a relatively simple analysis to convert any relative path use in DFDL into absolute paths. So I am not worried about there being performance impact. ...mikeb On Mon, Apr 2, 2012 at 7:30 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > We took action 168 on the last call to discuss the issue of relative paths > in expressions for properties in global formats. > > *168* > *Relative path expressions in global annotations (Mike)* > 28/03: Mike to check that allowing relative paths does not cause > performance problems for an implementation. > 05/04: > > > Just seen this in section 6.3.2. of the spec: "DFDL expressions that are > used to provide the value of DFDL properties in the dfd:format annotation > on the xs:schema MAY NOT contain relative paths". > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > ----- Forwarded by Steve Hanson/UK/IBM on 02/04/2012 12:28 ----- > > From: Steve Hanson/UK/IBM > To: dfdl-wg(a)ogf.org > Date: 14/03/2012 16:09 > Subject: Escape scheme properties that can be expressions > ------------------------------ > > > The spec allows the escapeCharacter and escapeEscapeCharacter of DFDL > escapeScheme to be expressions, in case the characters are supplied > dynamically in the data stream. > However given that escape schemes are defined globally, should these > properties be permitted to take relative paths in the expression? > Today the spec implicitly allows this, meaning an implementation can not > entirely validate an escapeScheme annotation up front, but must validate > escapeCharacter and escapeEscapeCharacter at each point of use. > I'm not sure it makes much sense for the path to be anything than absolute. > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > > > > > ------------------------------ > > * > * > > *Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > * > > > > > > > > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1

OGF DFDL WG Call Agenda 2012-04-05 - Call moved to Thurs & time changed !!
by Steve Hanson 02 Apr '12

02 Apr '12

Please find agenda attached for the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Call is on Thurs 2pm UK (9am Eastern) for this week only. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Escape scheme properties that can be expressions
by Steve Hanson 02 Apr '12

02 Apr '12

We took action 168 on the last call to discuss the issue of relative paths in expressions for properties in global formats. 168 Relative path expressions in global annotations (Mike) 28/03: Mike to check that allowing relative paths does not cause performance problems for an implementation. 05/04: Just seen this in section 6.3.2. of the spec: "DFDL expressions that are used to provide the value of DFDL properties in the dfd:format annotation on the xs:schema MAY NOT contain relative paths". Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/04/2012 12:28 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 14/03/2012 16:09 Subject: Escape scheme properties that can be expressions The spec allows the escapeCharacter and escapeEscapeCharacter of DFDL escapeScheme to be expressions, in case the characters are supplied dynamically in the data stream. However given that escape schemes are defined globally, should these properties be permitted to take relative paths in the expression? Today the spec implicitly allows this, meaning an implementation can not entirely validate an escapeScheme annotation up front, but must validate escapeCharacter and escapeEscapeCharacter at each point of use. I'm not sure it makes much sense for the path to be anything than absolute. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] [wg-all] New document published: WS-Disagreement
by Mike Beckerle 01 Apr '12

01 Apr '12

Somebody said this spec is really relevant to the DFDL group, but I can't seem to open the document on my TRS-80. The data file begins with hex 4170 7269 6c20 466f 6f6c 73. Can somebody help me figure out what this is? ...mikeb On Sun, Apr 1, 2012 at 4:00 PM, Andre Merzky <andre(a)merzky.net> wrote: > OGF Community: > > A new document has been published in the OGF series. All OGF > documents (including some that are open for public comment!) may be > found here: > > http://www.ogf.org/gf/docs/ > > * GFD-R.199 "WS-Disagreement" Via OGF Operations. > A. Merzky, A. Papaspyrou > April 1st 2012 > https://www.ogf.org/documents/GFD.199.pdf > > > Abstract: > > This document provides information to the grid community, proposing a > standard for a WS-Disagreement protocol (WS-NO), to communicate > disagreement > messages between components, and also between humans. > > > Andre Merzky, for OGF Operations > > -- > Nothing is ever easy... > -- > The wg-all mailing list is an umbrella list for all > OGF mailing lists. If you are subscribed to any OGF > mailing list, you will receive mails to wg-all. > > You can unsubscribe from this list on > > http://www.ogf.org/cgi-bin/perl/unsubscribe-wg-all.pl > > wg-all mailing list > wg-all(a)ogf.org > https://www.ogf.org/mailman/listinfo/wg-all -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

2 1