- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Minutes 2012-05-08
by Steve Hanson 08 May '12

08 May '12

Please find minutes from the above call on GridForge at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

S2G use of DFDL - to be discussed on 8th May DFDL-WG call
by Steve Hanson 08 May '12

08 May '12

Hi Michele We are looking forward to the discussion on S2G and DFDL on today's DFDL-WG call. So that other folk on the call can see what we have been discussing over the past few weeks, I've listed the full set of issues in one place: The original 4 issues which I noted are reproduced here... 1. DFDL does not have defaults for its properties. You must not use defaults, instead use dfdl:format to apply values across all the xsd. 2. You are using dfdl:encoding and dfdl:byteOrder on complex elements to control the encoding and byte order of child elements. This is not how DFDL works, you need to use dfdl:format. 3. The namespace for DFDL in the xsds is incorrect - it is missing a trailing slash - "http://www.ogf.org/dfdl/dfdl-1.0/" 4. There is no such property value as dfdl:lengthKind="expression" - it is just "explicit" (DFDL knows it is an expression because of {..} ) (Your responses to these are further down the attached e-mail thread.) Some other issues I have noted... 5. Section 3.1 in the pdf. The DFDL infoset shown is not a correct infoset, as it contains DFDL property values instead of data values. 6. Section 5. dmx:representation. Better if the name was not 'representation' as that is too close to dfdl:representation. Suggest dmx:displayRep. 7. Section 5.2. Time code representations. Are any of these sufficiently common to justify including in the DFDL specification? 8. What does this DMX property do? dmx:concealable="true" 9. Attribute form of DFDL & DMX properties are mixed together in an illegal way. Take the following example. <xs:simpleType name="TypeCADUSyncMarker"> <xs:annotation> <xs:appinfo source=" http://www.ogf.org/dfdl/dfdl-1.0"> <dmx:simpleType dfdl:lengthKind="explicit" dfdl:lengthUnits="bytes" dfdl:length="4" dmx:representation="Hexadecimal" /> </xs:appinfo> </xs:annotation> <xs:restriction base="xs:int"/> </xs:simpleType> a) The 'source' attribute says this is a DFDL appinfo annotation, yet the content is a dmx:simpleType element. b) The only place DFDL short form properties can appear is directly on a schema object. I would expect to see the following, which has separate appinfos for DFDL and DMX. Note that the dmx: is not needed when encoded on the appinfo element. <xs:simpleType name="TypeCADUSyncMarker"> <xs:annotation> <xs:appinfo source=" http://www.ogf.org/dfdl/dfdl-1.0"> <dfdl:simpleType lengthKind="explicit" lengthUnits="bytes" length="4"/> </xs:appinfo> <xs:appinfo source="<S2G URL>"> <dmx:simpleType representation="Hexadecimal"/> </xs:appinfo> </xs:annotation> <xs:restriction base="xs:int"/> </xs:simpleType> Or carry DMX properties on the schema object in short form: <xs:simpleType name="TypeCADUSyncMarker" dmx:representation="Hexadecimal"> <xs:annotation> <xs:appinfo source=" http://www.ogf.org/dfdl/dfdl-1.0"> <dfdl:simpleType lengthKind="explicit" lengthUnits="bytes" length="4"/> </xs:appinfo> </xs:annotation> <xs:restriction base="xs:int"/> </xs:simpleType> Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Michele Zundo <michele.zundo(a)esa.int> Date: 16/04/2012 10:22 Subject: Re: [DFDL-WG] DFDL based application released Hi Michele The invite I sent was a repeating invite, so you should have been added to all future DFDL calls. That obviously didn't work, so how about we choose Tues 8th May? I will write up a more formal review. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Date: 15/04/2012 11:10 Subject: Re: [DFDL-WG] DFDL based application released Dear Steve, I got reply to your comments from our contractor. We might discuss this at the next call (I can try to involve the developer) or prior to that (not next week as I'm fully booked). btw I'm very happy that you are giving such high quality detailed review, hopefully I can find enough resources on our side to pursue them as necessary. Michele Point 1: is a documentation error. the "VERY old" DFDL standard comment seems to come from a documentation error in S2G-DME-TEC-TNO014-1B.pdf. Table 2 should refer to [RD.1] Data Format Description Language (DFDL) v1.0 Specification (January 31,2011), as this is the DFDL standard used to support S2G development (this is also the latest version of the document available at http://www.ogf.org/dfdl/) Point 2: A brief analysis of the implementation suggests that implementing dfdl:format could be done with reasonable effort, This considers only the use of dfdl:format to specify global level properties, as implementing the ability to define new formats (through dfdl:defineFormat) to be referred by other dfdl:elements would increase complexity and thus implementation effort. We should understand if doing only this would be sufficient (taking into account we are not implementing a full DFDL arising engine) i.e. from current: <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dmx=" http://www.deimos.com.pt/dmx/dmx-1.0" xmlns:dfdl=" http://www.ogf.org/dfdl/dfdl-1.0"> <xs:include schemaLocation="EarthCARES-BandTMCADUTypes.xsd"/> <xs:element name="CADU" type="TypeCADU" dfdl:encoding="utf-8" dfdl:byteOrder="bigEndian"/> </xs:schema> to the following (added a new annotation at xs:schema level, containing the encoding and byteOrder properties; and removed the properties from the CADU element) <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dmx=" http://www.deimos.com.pt/dmx/dmx-1.0" xmlns:dfdl=" http://www.ogf.org/dfdl/dfdl-1.0/"> <xs:annotation> <xs:appinfo source=”http://www.ogf.org/dfdl/”> <dfdl:format encoding="utf-8" byteOrder="bigEndian"/> </xs:appinfo> </xs:annotation> <xs:include schemaLocation="EarthCARES-BandTMCADUTypes.xsd"/> <xs:element name="CADU" type="TypeCADU"/> </xs:schema> Point 3: not a problem the / can be added already now with no impact to S2G implementation. Point 4: DFDL parsing implementation S2G is simplified for the application needs and not fully generic. In the case of dfdl:lengthKind="expression", it was a way to circumvent the "naive" DFDL expressions implementation setup for S2G. At some point I considered allowing dfdl:lengthKind="explicit" for expression-like lengths, but given the available time it was not feasible. In terms of schemas, the only change would be to replace expression by explicit. In terms of implementation, a quick-fix is still possible by adding a few hard-coded evaluation rules. We instead would prefer to design a full evaluation engine, in order to improve the coverage of the DFDL expressions language which however needs much more effort. Regards Michele PS I have only received one invitation for the 10th April (in the past). When are the next one sent ? On Apr 11, 2012, at 12:09 , Steve Hanson wrote: Hi Michele That's interesting. The Proposed Recommendation version was Jan 2011 (draft 45). The version used by your contractor was (according to http://eop-cfi.esa.int/S2G/DOCS/S2G-DME-TEC-TNO014-1B.pdf) dated 12/03/2010 (draft 40). The differences between the drafts 40 & 45 are not that great, so it should not have made much difference. It explains issue 3, but not issues 1, 2, 4. That's all for now, I look forward to hearing from you when you are back from vacation. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Date: 10/04/2012 18:40 Subject: Re: [DFDL-WG] DFDL based application released Steven, I will be happy to discuss the changes and take them into account in our next contract as much as possible. When we wrote the SOW for the activity (around beginning of 2011 so not so long ago after all) I assume that was the version available on ogf web site. Take into account I am not the developer who coded but I will forward your comments to my contractor to see how to proceed. Take also into account that we can ingest evolution of DFDL only when we have a contract ongoing. I will explain what we can (and cannot do ) at the call. Regards Inviato da iPhone Il giorno 10 Apr 2012, alle ore 18:51, Steve Hanson <smh(a)uk.ibm.com> ha scritto: Hi Michele I have looked at your documentation and example xsds. We will discuss in more detail on the call, but my main observation is that the version of the DFDL specification you were using is VERY old. The DFDL language has changed in some important ways since that draft, so the way you use DFDL in S2G is not correct. The key issues are: 1. DFDL does not have defaults for its properties. You must not use defaults, instead use dfdl:format to apply values across all the xsd. 2. You are using dfdl:encoding and dfdl:byteOrder on complex elements to control the encoding and byte order of child elements. This is not how DFDL works, you need to use dfdl:format. 3. The namespace for DFDL in the xsds is incorrect - it is missing a trailing slash - "http://www.ogf.org/dfdl/dfdl-1.0/" 4. There is no such property value as dfdl:lengthKind="expression" - it is just "explicit" (we know it is an expression because of {..} ) Please can you add a disclaimer to the S2G documentation as soon as possible, to clarify that S2G is based on a draft version of DFDL from 2010 and not the OGF Proposed Recommendation? On the call we can discuss ways to make your implementation more conformant with the OGF PR version of the spec. Thanks Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Date: 10/04/2012 12:19 Subject: Re: [DFDL-WG] DFDL based application released That is great. I will confim when I'm back at work next week with my agenda in front. Inviato da iPhone Il giorno 10 Apr 2012, alle ore 11:29, Steve Hanson <smh(a)uk.ibm.com> ha scritto: Hi Michele When I said "join a DFDL WG call" I meant just dialling into one of our weekly calls, and explaining to the WG what you have created. If you want to subsequently participate in the WG then that's great, but that's a separate issue and entirely up to you. I have added you as an invitee to the DFDL WG weekly calls. Please accept those and choose one that suits you (other than Tues April 24th & May 1st). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Date: 06/04/2012 14:08 Subject: Re: [DFDL-WG] DFDL based application released I would prefer having a phone chat first to understand the scope of the wg activities to see if/how I can contribute. Arranging meeting at such short notice with me being away for next week or so is bit problematic. Anyway when and where is the next meeting ? Inviato da iPhone Il giorno 6 Apr 2012, alle ore 14:20, Steve Hanson <smh(a)uk.ibm.com> ha scritto: Hi Michele Are you back on Wed 18th April itself? If so we could move the DFDL WG call that week to Wed. I have a meeting clash so that would help me too. If that does not work then we are looking at Tues May 8th, because I am in the USA on Tues April 24th & May 1st. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 06/04/2012 11:12 Subject: Re: [DFDL-WG] DFDL based application released Dear Steve, I would be happy to participate taking into account I have quite limited availability of time since this il not my main work line. This week I'm on Easter holiday but we could talk after I'm back ( after 18 Apr) to make some planning. Shall we phone then ? Inviato da iPhone Il giorno 5 Apr 2012, alle ore 19:26, Steve Hanson <smh(a)uk.ibm.com> ha scritto: Hi Michele Thanks for letting the DFDL WG know about your DFDL based application. It's great that DFDL is being adopted in applications. Would you like to join a DFDL WG call in the near future, for a discussion on what you have created, and for us to feedback our findings? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: dfdl-wg(a)ogf.org Date: 05/04/2012 13:43 Subject: [DFDL-WG] DFDL based application released Sent by: dfdl-wg-bounces(a)ogf.org Dear all, you might be interested to know that yesterday we have freely released an application for visualizing binary satellite data (compliant to CCSDS standard) which based on DFDL. The satellite data binary structure for a number of ESA Earth Observation Mission has been described using DFDL schemas and a java parser implemented which allow the application to access all different data fields. The binary DFDL schemas can be also edited by the user to describe custom type of satellite data (i.e. changing the structure) without changing the application. We are considering in future to release the java Read API stand alone for everybody to use and build their own applications. The release message with pointers to URL to download the application is attached here below: ***** Dear all, please be informed today we have released the (Space-to-Ground) S2G Data Viewer which can be freely downloaded from : http://eop-cfi.esa.int/S2G/s2g.html S2G data Viewer is a modern, multi platform, extendable application to inspect CCSDS data (CADU, TF/VCDU and SP) and example screenshot attached below. The application is currently configured to support the following ESA Earth Observation Mission (including the custom Source Packet Data Header): Sentinel-1 Sentinel-2 Sentinel-3 Swarm EarthCARE and it can be extended by the user to support different type of data or new missions. I invite you to download and use it to support your AIV activities and, if necessary, to send any feedback to s2g(a)eopp.esa.int ****** ----------------------------------------- Mr. Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: michele.zundo(a)esa.int This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: michele.zundo(a)esa.int Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-05-08
by Steve Hanson 08 May '12

08 May '12

Please find agenda for the above call on GridForge at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: OGF DFDL WG Call Agenda 2012-05-02 - cancelled
by Steve Hanson 02 May '12

02 May '12

Call cancelled, both Mike and Steve have availability problems. ----- Forwarded by Steve Hanson/UK/IBM on 02/05/2012 11:45 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 30/04/2012 14:18 Subject: OGF DFDL WG Call Agenda 2012-05-02 Please find agenda for the above call on GridForge at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… Note this call is on Wed not Tues and is an hour earlier. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Cancelled: OGF DFDL Working Group weekly call
by Steve Hanson 02 May '12

02 May '12

Mike and Steve both have problems making this week's call so cancelling.

1 0

OGF DFDL WG Call Agenda 2012-05-02
by Steve Hanson 30 Apr '12

30 Apr '12

Please find agenda for the above call on GridForge at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… Note this call is on Wed not Tues and is an hour earlier. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Rescheduled: OGF DFDL Working Group weekly call (2 May 14:00 GDT)
by Steve Hanson 24 Apr '12

24 Apr '12

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl GridForge DFDL: http://forge.ogf.org/projects/dfdl-wg/ Changing the day of the call for the week commencing 30th April, so that Mike and Steve can attend. Also allocated an extra half hour as we have a few things to cover.

1 0

OGF DFDL WG Call Minutes 2012-04-17
by Steve Hanson 20 Apr '12

20 Apr '12

Please find minutes from the above call on GridForge at https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… The minutes cover the call on 17th plus follow-on call on 24th. Errata document is still at version 8 and may be found via https://forge.ogf.org/sf/projects/dfdl-wg Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group weekly call
by smh＠uk.ibm.com 20 Apr '12

20 Apr '12

Cancelled: OGF DFDL Working Group weekly call 24/04/2012 - (Repeats) Chair: Steve Hanson/UK/IBM Sent By: Steve_Hanson/UK/IBM%IBMGB Location: IBM Hursley DE2K09 Steve_Hanson Steve Hanson has cancelled this meeting. Your calendar will be updated to reflect this change. Repeats This entry repeats Required: mbeckerle.dfdl(a)gmail.com, Stephanie Fetzer/Charlotte/IBM@IBMUS, Suman Kalia/Toronto/IBM@IBMCA, Tim Kimber/UK/IBM@IBMGB Optional: adam.fox(a)nrl.navy.mil, a.mumfor(a)radium.ncsc.mil, boyd.fletcher(a)us.army.mil, bradley.r.sexton(a)gmail.com, dfdl-wg(a)ogf.org, michele.zundo(a)esa.int, remcgrat(a)illinois.edu

1 0

Next DFDL WG meetings
by Steve Hanson 20 Apr '12

20 Apr '12

Tues 24th April - cancelled Tues 1st May - moved to Wed 2nd May, time & agenda to follow Regards Steve Hanson Strategy, Common Transformation & DFDL Co-Chair, OGF DFDL WG IBM SWG, Hursley, UK, smh(a)uk.ibm.com, tel +44-(0)1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0