Hi Michele
We are looking forward to the discussion on S2G and DFDL on today's
DFDL-WG call.
So that other folk on the call can see what we have been discussing over
the past few weeks, I've listed the full set of issues in one place:
The original 4 issues which I noted are reproduced here...
1. DFDL does not have defaults for its properties. You must not use
defaults, instead use dfdl:format to apply values across all the xsd.
2. You are using dfdl:encoding and dfdl:byteOrder on complex elements to
control the encoding and byte order of child elements. This is not how
DFDL works, you need to use dfdl:format.
3. The namespace for DFDL in the xsds is incorrect - it is missing a
trailing slash - "http://www.ogf.org/dfdl/dfdl-1.0/"
4. There is no such property value as dfdl:lengthKind="expression" - it is
just "explicit" (DFDL knows it is an expression because of {..} )
(Your responses to these are further down the attached e-mail thread.)
Some other issues I have noted...
5. Section 3.1 in the pdf. The DFDL infoset shown is not a correct
infoset, as it contains DFDL property values instead of data values.
6. Section 5. dmx:representation. Better if the name was not
'representation' as that is too close to dfdl:representation. Suggest
dmx:displayRep.
7. Section 5.2. Time code representations. Are any of these sufficiently
common to justify including in the DFDL specification?
8. What does this DMX property do? dmx:concealable="true"
9. Attribute form of DFDL & DMX properties are mixed together in an
illegal way. Take the following example.
<xs:simpleType name="TypeCADUSyncMarker">
<xs:annotation>
<xs:appinfo source="
http://www.ogf.org/dfdl/dfdl-1.0">
<dmx:simpleType dfdl:lengthKind="explicit"
dfdl:lengthUnits="bytes" dfdl:length="4" dmx:representation="Hexadecimal"
/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:int"/>
</xs:simpleType>
a) The 'source' attribute says this is a DFDL appinfo annotation, yet the
content is a dmx:simpleType element.
b) The only place DFDL short form properties can appear is directly on a
schema object.
I would expect to see the following, which has separate appinfos for DFDL
and DMX. Note that the dmx: is not needed when encoded on the appinfo
element.
<xs:simpleType name="TypeCADUSyncMarker">
<xs:annotation>
<xs:appinfo source="
http://www.ogf.org/dfdl/dfdl-1.0">
<dfdl:simpleType lengthKind="explicit"
lengthUnits="bytes" length="4"/>
</xs:appinfo>
<xs:appinfo source="<S2G URL>">
<dmx:simpleType
representation="Hexadecimal"/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:int"/>
</xs:simpleType>
Or carry DMX properties on the schema object in short form:
<xs:simpleType name="TypeCADUSyncMarker"
dmx:representation="Hexadecimal">
<xs:annotation>
<xs:appinfo source="
http://www.ogf.org/dfdl/dfdl-1.0">
<dfdl:simpleType lengthKind="explicit"
lengthUnits="bytes" length="4"/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:int"/>
</xs:simpleType>
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh(a)uk.ibm.com
tel:+44-1962-815848
From: Steve Hanson/UK/IBM
To: Michele Zundo <michele.zundo(a)esa.int>
Date: 16/04/2012 10:22
Subject: Re: [DFDL-WG] DFDL based application released
Hi Michele
The invite I sent was a repeating invite, so you should have been added to
all future DFDL calls. That obviously didn't work, so how about we choose
Tues 8th May? I will write up a more formal review.
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh(a)uk.ibm.com
tel:+44-1962-815848
From: Michele Zundo <michele.zundo(a)esa.int>
To: Steve Hanson/UK/IBM@IBMGB
Date: 15/04/2012 11:10
Subject: Re: [DFDL-WG] DFDL based application released
Dear Steve,
I got reply to your comments from our contractor. We might discuss this at
the next call (I can try to involve the developer) or prior to
that (not next week as I'm fully booked).
btw I'm very happy that you are giving such high quality detailed review,
hopefully I can find enough resources on our side to pursue them as
necessary.
Michele
Point 1:
is a documentation error.
the "VERY old" DFDL standard comment seems to come from a documentation
error in S2G-DME-TEC-TNO014-1B.pdf. Table 2 should refer to [RD.1] Data
Format Description Language (DFDL) v1.0 Specification (January 31,2011),
as this is the DFDL standard used to support S2G development (this is also
the latest version of the document available at http://www.ogf.org/dfdl/)
Point 2:
A brief analysis of the implementation suggests that implementing
dfdl:format could be done with reasonable effort, This considers only the
use of dfdl:format to specify global level properties, as implementing the
ability to define new formats (through dfdl:defineFormat) to be referred
by other dfdl:elements would increase complexity and thus implementation
effort. We should understand if doing only this would be sufficient
(taking into account we are not implementing a full DFDL arising engine)
i.e. from current:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dmx="
http://www.deimos.com.pt/dmx/dmx-1.0" xmlns:dfdl="
http://www.ogf.org/dfdl/dfdl-1.0">
<xs:include schemaLocation="EarthCARES-BandTMCADUTypes.xsd"/>
<xs:element name="CADU" type="TypeCADU" dfdl:encoding="utf-8"
dfdl:byteOrder="bigEndian"/>
</xs:schema>
to the following (added a new annotation at xs:schema level, containing
the encoding and byteOrder properties; and removed the properties from the
CADU element)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dmx="
http://www.deimos.com.pt/dmx/dmx-1.0" xmlns:dfdl="
http://www.ogf.org/dfdl/dfdl-1.0/">
<xs:annotation>
<xs:appinfo source=”http://www.ogf.org/dfdl/”>
<dfdl:format encoding="utf-8" byteOrder="bigEndian"/>
</xs:appinfo>
</xs:annotation>
<xs:include schemaLocation="EarthCARES-BandTMCADUTypes.xsd"/>
<xs:element name="CADU" type="TypeCADU"/>
</xs:schema>
Point 3:
not a problem the / can be added already now with no impact to S2G
implementation.
Point 4:
DFDL parsing implementation S2G is simplified for the application needs
and not fully generic.
In the case of dfdl:lengthKind="expression", it was a way to circumvent
the "naive" DFDL expressions implementation setup for S2G. At some point I
considered allowing dfdl:lengthKind="explicit" for expression-like
lengths, but given the available time it was not feasible.
In terms of schemas, the only change would be to replace expression by
explicit. In terms of implementation, a quick-fix is still possible by
adding a few hard-coded evaluation rules. We instead would prefer to
design a full evaluation engine, in order to improve the coverage of the
DFDL expressions language which however needs much more effort.
Regards
Michele
PS I have only received one invitation for the 10th April (in the past).
When are the next one sent ?
On Apr 11, 2012, at 12:09 , Steve Hanson wrote:
Hi Michele
That's interesting. The Proposed Recommendation version was Jan 2011
(draft 45). The version used by your contractor was (according to
http://eop-cfi.esa.int/S2G/DOCS/S2G-DME-TEC-TNO014-1B.pdf) dated
12/03/2010 (draft 40). The differences between the drafts 40 & 45 are not
that great, so it should not have made much difference. It explains issue
3, but not issues 1, 2, 4.
That's all for now, I look forward to hearing from you when you are back
from vacation.
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh(a)uk.ibm.com
tel:+44-1962-815848
From: Michele Zundo <michele.zundo(a)esa.int>
To: Steve Hanson/UK/IBM@IBMGB
Date: 10/04/2012 18:40
Subject: Re: [DFDL-WG] DFDL based application released
Steven,
I will be happy to discuss the changes and take them into account in our
next contract as much as possible.
When we wrote the SOW for the activity (around beginning of 2011 so not so
long ago after all) I assume that was the version available on ogf web
site. Take into account I am not the developer who coded but I will
forward your comments to my contractor to see how to proceed. Take also
into account that we can ingest evolution of DFDL only when we have a
contract ongoing.
I will explain what we can (and cannot do ) at the call.
Regards
Inviato da iPhone
Il giorno 10 Apr 2012, alle ore 18:51, Steve Hanson <smh(a)uk.ibm.com> ha
scritto:
Hi Michele
I have looked at your documentation and example xsds. We will discuss in
more detail on the call, but my main observation is that the version of
the DFDL specification you were using is VERY old. The DFDL language has
changed in some important ways since that draft, so the way you use DFDL
in S2G is not correct.
The key issues are:
1. DFDL does not have defaults for its properties. You must not use
defaults, instead use dfdl:format to apply values across all the xsd.
2. You are using dfdl:encoding and dfdl:byteOrder on complex elements to
control the encoding and byte order of child elements. This is not how
DFDL works, you need to use dfdl:format.
3. The namespace for DFDL in the xsds is incorrect - it is missing a
trailing slash - "http://www.ogf.org/dfdl/dfdl-1.0/"
4. There is no such property value as dfdl:lengthKind="expression" - it is
just "explicit" (we know it is an expression because of {..} )
Please can you add a disclaimer to the S2G documentation as soon as
possible, to clarify that S2G is based on a draft version of DFDL from
2010 and not the OGF Proposed Recommendation?
On the call we can discuss ways to make your implementation more
conformant with the OGF PR version of the spec.
Thanks
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh(a)uk.ibm.com
tel:+44-1962-815848
From: Michele Zundo <michele.zundo(a)esa.int>
To: Steve Hanson/UK/IBM@IBMGB
Date: 10/04/2012 12:19
Subject: Re: [DFDL-WG] DFDL based application released
That is great. I will confim when I'm back at work next week with my
agenda in front.
Inviato da iPhone
Il giorno 10 Apr 2012, alle ore 11:29, Steve Hanson <smh(a)uk.ibm.com> ha
scritto:
Hi Michele
When I said "join a DFDL WG call" I meant just dialling into one of our
weekly calls, and explaining to the WG what you have created.
If you want to subsequently participate in the WG then that's great, but
that's a separate issue and entirely up to you.
I have added you as an invitee to the DFDL WG weekly calls. Please accept
those and choose one that suits you (other than Tues April 24th & May
1st).
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh(a)uk.ibm.com
tel:+44-1962-815848
From: Michele Zundo <michele.zundo(a)esa.int>
To: Steve Hanson/UK/IBM@IBMGB
Date: 06/04/2012 14:08
Subject: Re: [DFDL-WG] DFDL based application released
I would prefer having a phone chat first to understand the scope of the wg
activities to see if/how I can contribute.
Arranging meeting at such short notice with me being away for next week or
so is bit problematic.
Anyway when and where is the next meeting ?
Inviato da iPhone
Il giorno 6 Apr 2012, alle ore 14:20, Steve Hanson <smh(a)uk.ibm.com> ha
scritto:
Hi Michele
Are you back on Wed 18th April itself? If so we could move the DFDL WG
call that week to Wed. I have a meeting clash so that would help me too.
If that does not work then we are looking at Tues May 8th, because I am in
the USA on Tues April 24th & May 1st.
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh(a)uk.ibm.com
tel:+44-1962-815848
From: Michele Zundo <michele.zundo(a)esa.int>
To: Steve Hanson/UK/IBM@IBMGB
Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>
Date: 06/04/2012 11:12
Subject: Re: [DFDL-WG] DFDL based application released
Dear Steve,
I would be happy to participate taking into account I have quite limited
availability of time since this il not my main work line.
This week I'm on Easter holiday but we could talk after I'm back ( after
18 Apr) to make some planning. Shall we phone then ?
Inviato da iPhone
Il giorno 5 Apr 2012, alle ore 19:26, Steve Hanson <smh(a)uk.ibm.com> ha
scritto:
Hi Michele
Thanks for letting the DFDL WG know about your DFDL based application.
It's great that DFDL is being adopted in applications.
Would you like to join a DFDL WG call in the near future, for a discussion
on what you have created, and for us to feedback our findings?
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh(a)uk.ibm.com
tel:+44-1962-815848
From: Michele Zundo <michele.zundo(a)esa.int>
To: dfdl-wg(a)ogf.org
Date: 05/04/2012 13:43
Subject: [DFDL-WG] DFDL based application released
Sent by: dfdl-wg-bounces(a)ogf.org
Dear all,
you might be interested to know that yesterday we have freely released an
application for visualizing binary satellite data (compliant to CCSDS
standard)
which based on DFDL.
The satellite data binary structure for a number of ESA Earth Observation
Mission
has been described using DFDL schemas and a java parser implemented which
allow the application to access all different data fields.
The binary DFDL schemas can be also edited by the user to describe custom
type
of satellite data (i.e. changing the structure) without changing the
application.
We are considering in future to release the java Read API stand alone for
everybody to
use and build their own applications.
The release message with pointers to URL to download the application is
attached
here below:
*****
Dear all,
please be informed today we have released the (Space-to-Ground) S2G Data
Viewer which
can be freely downloaded from :
http://eop-cfi.esa.int/S2G/s2g.html
S2G data Viewer is a modern, multi platform, extendable application to
inspect CCSDS data (CADU, TF/VCDU and SP)
and example screenshot attached below.
The application is currently configured to support the following ESA Earth
Observation Mission
(including the custom Source Packet Data Header):
Sentinel-1
Sentinel-2
Sentinel-3
Swarm
EarthCARE
and it can be extended by the user to support different type of data or
new missions.
I invite you to download and use it to support your AIV activities and, if
necessary, to send any
feedback to s2g(a)eopp.esa.int
******
-----------------------------------------
Mr. Michele Zundo
Head of Ground System Definition and Verification Office
EOP-PEP
European Space Agency, ESTEC
e-mail: michele.zundo(a)esa.int
This message and any attachments are intended for the use of the addressee
or addressees only. The unauthorised disclosure, use, dissemination or
copying (either in whole or in part) of its content is not permitted. If
you received this message in error, please notify the sender and delete it
from your system. Emails can be altered and their integrity cannot be
guaranteed by the sender.
Please consider the environment before printing this email.
--
dfdl-wg mailing list
dfdl-wg(a)ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
This message and any attachments are intended for the use of the addressee
or addressees only. The unauthorised disclosure, use, dissemination or
copying (either in whole or in part) of its content is not permitted. If
you received this message in error, please notify the sender and delete it
from your system. Emails can be altered and their integrity cannot be
guaranteed by the sender.
Please consider the environment before printing this email.
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
This message and any attachments are intended for the use of the addressee
or addressees only. The unauthorised disclosure, use, dissemination or
copying (either in whole or in part) of its content is not permitted. If
you received this message in error, please notify the sender and delete it
from your system. Emails can be altered and their integrity cannot be
guaranteed by the sender.
Please consider the environment before printing this email.
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
This message and any attachments are intended for the use of the addressee
or addressees only. The unauthorised disclosure, use, dissemination or
copying (either in whole or in part) of its content is not permitted. If
you received this message in error, please notify the sender and delete it
from your system. Emails can be altered and their integrity cannot be
guaranteed by the sender.
Please consider the environment before printing this email.
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
This message and any attachments are intended for the use of the addressee
or addressees only. The unauthorised disclosure, use, dissemination or
copying (either in whole or in part) of its content is not permitted. If
you received this message in error, please notify the sender and delete it
from your system. Emails can be altered and their integrity cannot be
guaranteed by the sender.
Please consider the environment before printing this email.
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-----------------------------------------
Michele Zundo
Head of Ground System Definition and Verification Office
EOP-PEP
European Space Agency, ESTEC
e-mail: michele.zundo(a)esa.int
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Call cancelled, both Mike and Steve have availability problems.
----- Forwarded by Steve Hanson/UK/IBM on 02/05/2012 11:45 -----
From: Steve Hanson/UK/IBM
To: dfdl-wg(a)ogf.org
Date: 30/04/2012 14:18
Subject: OGF DFDL WG Call Agenda 2012-05-02
Please find agenda for the above call on GridForge at
https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/…
Note this call is on Wed not Tues and is an hour earlier.
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF DFDL Working Group
IBM SWG, Hursley, UK
smh(a)uk.ibm.com
tel:+44-1962-815848
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
