Hi Michele
We are looking forward to the discussion
on S2G and DFDL on today's DFDL-WG call.
So that other folk on the call can see
what we have been discussing over the past few weeks, I've listed the full
set of issues in one place:
The original 4 issues which I noted
are reproduced here...
1. DFDL does not have defaults
for its properties. You must not use defaults, instead use dfdl:format
to apply values across all the xsd.
2. You are using dfdl:encoding and dfdl:byteOrder on complex elements
to control the encoding and byte order of child elements. This is not how
DFDL works, you need to use dfdl:format.
3. The namespace for DFDL in the xsds is incorrect - it is missing
a trailing slash - "http://www.ogf.org/dfdl/dfdl-1.0/"
4. There is no such property value as dfdl:lengthKind="expression"
- it is just "explicit" (DFDL knows it is an expression because
of {..} )
(Your responses to these are further
down the attached e-mail thread.)
Some other issues I have noted...
5. Section 3.1 in the pdf. The
DFDL infoset shown is not a correct infoset, as it contains DFDL property
values instead of data values.
6. Section 5. dmx:representation.
Better if the name was not 'representation' as that is too close to dfdl:representation.
Suggest dmx:displayRep.
7. Section 5.2. Time code representations.
Are any of these sufficiently common to justify including in the DFDL specification?
8. What does this DMX property
do? dmx:concealable="true"
9. Attribute form of DFDL &
DMX properties are mixed together in an illegal way. Take the following
example.
<xs:simpleType
name="TypeCADUSyncMarker">
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/dfdl-1.0">
<dmx:simpleType
dfdl:lengthKind="explicit" dfdl:lengthUnits="bytes"
dfdl:length="4"
dmx:representation="Hexadecimal"/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:int"/>
</xs:simpleType>
a) The 'source' attribute says this
is a DFDL appinfo annotation, yet the content is a dmx:simpleType element.
b) The only place DFDL short form properties
can appear is directly on a schema object.
I would expect to see the following,
which has separate appinfos for DFDL and DMX. Note that the dmx: is not
needed when encoded on the appinfo element.
<xs:simpleType
name="TypeCADUSyncMarker">
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/dfdl-1.0">
<dfdl:simpleType lengthKind="explicit"
lengthUnits="bytes" length="4"/>
</xs:appinfo>
<xs:appinfo source="<S2G URL>">
<dmx:simpleType representation="Hexadecimal"/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:int"/>
</xs:simpleType>
Or carry DMX properties on the schema
object in short form:
<xs:simpleType
name="TypeCADUSyncMarker" dmx:representation="Hexadecimal">
<xs:annotation>
<xs:appinfo
source="http://www.ogf.org/dfdl/dfdl-1.0">
<dfdl:simpleType lengthKind="explicit"
lengthUnits="bytes" length="4"/>
</xs:appinfo>
</xs:annotation>
<xs:restriction base="xs:int"/>
</xs:simpleType>
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF
DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
From:
Steve Hanson/UK/IBM
To:
Michele Zundo <michele.zundo@esa.int>
Date:
16/04/2012 10:22
Subject:
Re: [DFDL-WG]
DFDL based application released
Hi Michele
The invite I sent was a repeating invite,
so you should have been added to all future DFDL calls. That obviously
didn't work, so how about we choose Tues 8th May? I will write up
a more formal review.
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF
DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
From:
Michele Zundo <michele.zundo@esa.int>
To:
Steve Hanson/UK/IBM@IBMGB
Date:
15/04/2012 11:10
Subject:
Re: [DFDL-WG]
DFDL based application released
Dear Steve,
I got reply to your comments from our contractor. We might
discuss this at the next call (I can try to involve the developer) or prior
to
that (not next week as I'm fully booked).
btw I'm very happy that you are giving such high quality
detailed review, hopefully I can find enough resources on our side to pursue
them as necessary.
Michele
Point 1:
is a documentation error.
the "VERY old"
DFDL standard comment seems to come from a documentation
error in S2G-DME-TEC-TNO014-1B.pdf. Table 2 should refer to [RD.1] Data
Format Description Language (DFDL) v1.0 Specification (January 31,2011),
as this is the DFDL standard used to support S2G development (this is also
the latest version of the document available at http://www.ogf.org/dfdl/).
Point 2:
A brief analysis of the implementation suggests that implementing
dfdl:format could be done with reasonable effort, This considers
only the use of dfdl:format to specify global level properties,
as implementing the ability to define new formats (through dfdl:defineFormat)
to be referred by other dfdl:elements would increase complexity
and thus implementation effort. We should understand if doing only this
would be sufficient (taking into account we are not implementing a full
DFDL arising engine) i.e. from current:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:dmx="http://www.deimos.com.pt/dmx/dmx-1.0"
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0">
<xs:include schemaLocation="EarthCARES-BandTMCADUTypes.xsd"/>
<xs:element name="CADU" type="TypeCADU"
dfdl:encoding="utf-8"
dfdl:byteOrder="bigEndian"/>
</xs:schema>
to the following (added a new annotation at xs:schema level, containing
the encoding and byteOrder properties; and removed the properties
from the CADU element)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:dmx="http://www.deimos.com.pt/dmx/dmx-1.0"
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/">
<xs:annotation>
<xs:appinfo source=”http://www.ogf.org/dfdl/”>
<dfdl:format encoding="utf-8"
byteOrder="bigEndian"/>
</xs:appinfo>
</xs:annotation>
<xs:include schemaLocation="EarthCARES-BandTMCADUTypes.xsd"/>
<xs:element name="CADU" type="TypeCADU"/>
</xs:schema>
Point 3:
not a problem the / can be added already now with
no impact to S2G implementation.
Point 4:
DFDL parsing implementation S2G is simplified for
the application needs and not fully generic.
In the case of dfdl:lengthKind="expression",
it was a way to circumvent the "naive" DFDL expressions implementation
setup for S2G. At some point I considered allowing dfdl:lengthKind="explicit"
for expression-like lengths, but given the available time it was not feasible.
In terms of schemas, the only change would be to replace expression
by explicit. In terms of implementation, a quick-fix is still possible
by adding a few hard-coded evaluation rules. We instead would prefer to
design a full evaluation engine, in order to improve the coverage of the
DFDL expressions language which however needs much more effort.
Regards
Michele
PS I have only received one invitation for the 10th April
(in the past). When are the next one sent ?
On Apr 11, 2012, at 12:09 , Steve Hanson wrote:
Hi Michele
That's interesting. The Proposed Recommendation version was Jan 2011 (draft
45). The version used by your contractor was (according to http://eop-cfi.esa.int/S2G/DOCS/S2G-DME-TEC-TNO014-1B.pdf)
dated 12/03/2010 (draft 40). The differences between the drafts 40
& 45 are not that great, so it should not have made much difference.
It explains issue 3, but not issues 1, 2, 4.
That's all for now, I look forward to hearing from you when you are back
from vacation.
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF
DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
From: Michele
Zundo <michele.zundo@esa.int>
To: Steve
Hanson/UK/IBM@IBMGB
Date: 10/04/2012
18:40
Subject: Re:
[DFDL-WG] DFDL based application released
Steven,
I will be happy to discuss the changes and take them into account in our
next contract as much as possible.
When we wrote the SOW for the activity (around beginning of 2011 so not
so long ago after all) I assume that was the version available on ogf web
site. Take into account I am not the developer who coded but I will forward
your comments to my contractor to see how to proceed. Take also into account
that we can ingest evolution of DFDL only when we have a contract ongoing.
I will explain what we can (and cannot do ) at the call.
Regards
Inviato da iPhone
Il giorno 10 Apr 2012, alle ore 18:51, Steve Hanson <smh@uk.ibm.com>
ha scritto:
Hi Michele
I have looked at your documentation and example xsds. We will discuss
in more detail on the call, but my main observation is that the version
of the DFDL specification you were using is VERY old. The DFDL language
has changed in some important ways since that draft, so the way you use
DFDL in S2G is not correct.
The key issues are:
1. DFDL does not have defaults for its properties. You must not
use defaults, instead use dfdl:format to apply values across all the xsd.
2. You are using dfdl:encoding and dfdl:byteOrder on complex elements
to control the encoding and byte order of child elements. This is not how
DFDL works, you need to use dfdl:format.
3. The namespace for DFDL in the xsds is incorrect - it is missing
a trailing slash - "http://www.ogf.org/dfdl/dfdl-1.0/"
4. There is no such property value as dfdl:lengthKind="expression"
- it is just "explicit" (we know it is an expression because
of {..} )
Please can you add a disclaimer to the S2G documentation as soon as possible,
to clarify that S2G is based on a draft version of DFDL from 2010 and not
the OGF Proposed Recommendation?
On the call we can discuss ways to make your implementation more conformant
with the OGF PR version of the spec.
Thanks
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF
DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
From: Michele
Zundo <michele.zundo@esa.int>
To: Steve
Hanson/UK/IBM@IBMGB
Date: 10/04/2012
12:19
Subject: Re:
[DFDL-WG] DFDL based application released
That is great. I will confim when I'm back at work next week with my agenda
in front.
Inviato da iPhone
Il giorno 10 Apr 2012, alle ore 11:29, Steve Hanson <smh@uk.ibm.com>
ha scritto:
Hi Michele
When I said "join a DFDL WG call" I meant just dialling into
one of our weekly calls, and explaining to the WG what you have created.
If you want to subsequently participate in the WG then that's great, but
that's a separate issue and entirely up to you.
I have added you as an invitee to the DFDL WG weekly calls. Please
accept those and choose one that suits you (other than Tues April 24th
& May 1st).
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF
DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
From: Michele
Zundo <michele.zundo@esa.int>
To: Steve
Hanson/UK/IBM@IBMGB
Date: 06/04/2012
14:08
Subject: Re:
[DFDL-WG] DFDL based application released
I would prefer having a phone chat first to understand the scope of the
wg activities to see if/how I can contribute.
Arranging meeting at such short notice with me being away for next week
or so is bit problematic.
Anyway when and where is the next meeting ?
Inviato da iPhone
Il giorno 6 Apr 2012, alle ore 14:20, Steve Hanson <smh@uk.ibm.com>
ha scritto:
Hi Michele
Are you back on Wed 18th April itself? If so we could move the DFDL
WG call that week to Wed. I have a meeting clash so that would help me
too.
If that does not work then we are looking at Tues May 8th, because I am
in the USA on Tues April 24th & May 1st.
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF
DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
From: Michele
Zundo <michele.zundo@esa.int>
To: Steve
Hanson/UK/IBM@IBMGB
Cc: "dfdl-wg@ogf.org"
<dfdl-wg@ogf.org>
Date: 06/04/2012
11:12
Subject: Re:
[DFDL-WG] DFDL based application released
Dear Steve,
I would be happy to participate taking into account I have quite limited
availability of time since this il not my main work line.
This week I'm on Easter holiday but we could talk after I'm back ( after
18 Apr) to make some planning. Shall we phone then ?
Inviato da iPhone
Il giorno 5 Apr 2012, alle ore 19:26, Steve Hanson <smh@uk.ibm.com>
ha scritto:
Hi Michele
Thanks for letting the DFDL WG know about your DFDL based application.
It's great that DFDL is being adopted in applications.
Would you like to join a DFDL WG call in the near future, for a discussion
on what you have created, and for us to feedback our findings?
Regards
Steve Hanson
Architect, Data Format Description Language (DFDL)
Co-Chair, OGF
DFDL Working Group
IBM SWG, Hursley, UK
smh@uk.ibm.com
tel:+44-1962-815848
From: Michele
Zundo <michele.zundo@esa.int>
To: dfdl-wg@ogf.org
Date: 05/04/2012
13:43
Subject: [DFDL-WG]
DFDL based application released
Sent by: dfdl-wg-bounces@ogf.org
Dear all,
you might be interested to know that yesterday we have freely released
an
application for visualizing binary satellite data (compliant to CCSDS standard)
which based on DFDL.
The satellite data binary structure for a number of ESA Earth Observation
Mission
has been described using DFDL schemas and a java parser implemented which
allow the application to access all different data fields.
The binary DFDL schemas can be also edited by the user to describe custom
type
of satellite data (i.e. changing the structure) without changing the application.
We are considering in future to release the java Read API stand alone for
everybody to
use and build their own applications.
The release message with pointers to URL to download the application is
attached
here below:
*****
Dear all,
please be informed today we have released the (Space-to-Ground) S2G
Data Viewer which
can be freely downloaded from :
http://eop-cfi.esa.int/S2G/s2g.html
S2G data Viewer is a modern, multi platform, extendable application to
inspect CCSDS data (CADU, TF/VCDU and SP)
and example screenshot attached below.
The application is currently configured to support the following ESA Earth
Observation Mission
(including the custom Source Packet Data Header):
Sentinel-1
Sentinel-2
Sentinel-3
Swarm
EarthCARE
and it can be extended by the user to support different type of data
or new missions.
I invite you to download and use it to support your AIV activities and,
if necessary, to send any
feedback to s2g@eopp.esa.int
******
-----------------------------------------
Mr. Michele Zundo
Head of Ground System Definition and Verification Office
EOP-PEP
European Space Agency, ESTEC
e-mail: michele.zundo@esa.int
This message and any attachments are intended for the use of the addressee
or addressees only. The unauthorised disclosure, use, dissemination or
copying (either in whole or in part) of its content is not permitted. If
you received this message in error, please notify the sender and delete
it from your system. Emails can be altered and their integrity cannot be
guaranteed by the sender.
Please consider the environment before printing this email.
--
dfdl-wg mailing list
dfdl-wg@ogf.org
https://www.ogf.org/mailman/listinfo/dfdl-wg
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
This message and any attachments are intended for the
use of the addressee or addressees only. The unauthorised disclosure, use,
dissemination or copying (either in whole or in part) of its content is
not permitted. If you received this message in error, please notify the
sender and delete it from your system. Emails can be altered and their
integrity cannot be guaranteed by the sender.
Please consider the environment before printing this email.
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
This message and any attachments are intended for the
use of the addressee or addressees only. The unauthorised disclosure, use,
dissemination or copying (either in whole or in part) of its content is
not permitted. If you received this message in error, please notify the
sender and delete it from your system. Emails can be altered and their
integrity cannot be guaranteed by the sender.
Please consider the environment before printing this email.
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
This message and any attachments are intended for the
use of the addressee or addressees only. The unauthorised disclosure, use,
dissemination or copying (either in whole or in part) of its content is
not permitted. If you received this message in error, please notify the
sender and delete it from your system. Emails can be altered and their
integrity cannot be guaranteed by the sender.
Please consider the environment before printing this email.
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
This message and any attachments are intended for the
use of the addressee or addressees only. The unauthorised disclosure, use,
dissemination or copying (either in whole or in part) of its content is
not permitted. If you received this message in error, please notify the
sender and delete it from your system. Emails can be altered and their
integrity cannot be guaranteed by the sender.
Please consider the environment before printing this email.
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
-----------------------------------------
Michele Zundo
Head of Ground System Definition and Verification Office
EOP-PEP
European Space Agency, ESTEC
e-mail: michele.zundo@esa.int
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU