- dfdl-wg - lists.ogf.org

Re: [DFDL-WG] [DFDL]: First Release of DFDL4S Parser Library
by Steve Hanson 11 May '15

11 May '15

Dear Michele I can have a call at 16:00 CET tomorrow but if you want other people from DFDL Working Group to attend then it should be at 16:00 UK (GMT+1 = BST). This is an hour behind CET. Please email webex details to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Maurizio De Bartolomei <Maurizio.De.Bartolomei(a)esa.int>, Montserrat Piñol <mpinol(a)eopp.esa.int>, "Rui Mestre (DME)" <rui.mestre(a)deimos.com.pt> Date: 11/05/2015 13:05 Subject: Re: [DFDL]: First Release of DFDL4S Parser Library Dear Steve, 1) I can confirm that we have used version 207 of the DFDL as reference. 2) The DFDL4S implementation and the delta between the Core DFDL spec and what has been implemented is documented in 3) Schemas can be easily exported by the S2G app, but I attach some for reference (they are all inside the jar) 3) We can have a teleconf Tomorrow at 16:00 CET if you want, Should I set up a Webex to share the screen ? Regards Michele On 06 May 2015, at 10:49 , Steve Hanson <smh(a)uk.ibm.com> wrote: Dear Michele Good to hear from you. Thank you for letting me know about the latest developments with S2G Data Viewer. I am sure the DFDL Working Group would be interested in a demo of S2G. What is your availability on Tuesdays at 4pm-5pm UK? Meantime do you have a document that you could send which summarises which features of the DFDL language you have used, and some example schemas? Please note that the DFDL 1.0 specification was re-published in October 2014, replacing and obsoleting the version from January 2011. Does S2G comply with the October 2014 version ( https://www.ogf.org/documents/GFD.207.pdf) ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Montserrat Piñol <mpinol(a)eopp.esa.int>, Maurizio De Bartolomei <Maurizio.De.Bartolomei(a)esa.int>, "Rui Mestre (DME)" < rui.mestre(a)deimos.com.pt> Date: 29/04/2015 17:38 Subject: [DFDL]: First Release of DFDL4S Parser Library Dear Steve, we talked some time ago when we presented the tool we developed at ESA to inspect satellite TM and which made use of DFDL description for the binary data and we have included a number of your comments at the time. Since then we have further evolved the tool (called S2G Data Viewer) and produced as well a standalone version of a Java library which performs the function of extracting data field from binary data based on a DFDL description. This library (called DFDL4S i.e. DFDL for Space) is now available to anybody interested. It implements only the subset of the Core capability that we need but I think it might be nevertheless interesting for the DFDL community. In future we plan to expand to write capability the same library so user can serialise their data as per DFDL description and generate realistically looking (simulated) satellite data. The application and the library are available for download at http:eop-cfi.esa.int If there is interest we could arrange a short demo of the app (using webex) or I couldd ask the developer to show the command line test of the DFDL4S library. We are very happy/willing to share results/code/etc. Best Regards PS we are currently thinking to build “transformation” that can ingest different type of descriptive representation and generate the corresponding DFDL schemas. ESA - European Space Agency Michele Zundo Head of Ground System Definition and Verification Office System Support Division Earth Observation Project Department Directorate of Earth Observation Programmes ESTEC Kepleerlaan 1, 2200AZ Noordwijk, Netherlands michele.zundo(a)esa.int | www.esa.int T. +31 71 565 3924 | F +31 71 565 3191 This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: michele.zundo(a)esa.int Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2

Meeting invitation: S2G and DFDL demo
by Michele Zundo 11 May '15

11 May '15

Hello , Michele Zundo invites you to attend this online meeting. Topic: S2G and DFDL demo Date: Tuesday, 12. May 2015 Time: 15:00, GMT Summer Time (London, GMT+01:00) Meeting Number: 843 184 272 Meeting Password: toto1234 ------------------------------------------------------- To join the online meeting (Now from mobile devices!) ------------------------------------------------------- 1. Go to https://esa-meeting.webex.com/esa-meeting/j.php?MTID=me3a599ee88e3b896337d8… 2. If requested, enter your name and email address. 3. If a password is required, enter the meeting password: toto1234 4. Click "Join". To view in other time zones or languages, please click the link: https://esa-meeting.webex.com/esa-meeting/j.php?MTID=ma20877926dd694b83b532… ------------------------------------------------------- To join the audio conference only ------------------------------------------------------- Call-in toll number (UK): +44-203-478-5289 Global call-in numbers: https://esa-meeting.webex.com/esa-meeting/globalcallin.php?serviceType=MC&E… Access code:843 184 272 ------------------------------------------------------- For assistance ------------------------------------------------------- 1. Go to https://esa-meeting.webex.com/esa-meeting/mc 2. On the left navigation bar, click "Support". You can contact me at: michele.zundo(a)esa.int To add this meeting to your calendar program (for example Microsoft Outlook), click this link: https://esa-meeting.webex.com/esa-meeting/j.php?MTID=m70df310e4c695d305bed8… The playback of UCF (Universal Communications Format) rich media files requires appropriate players. To view this type of rich media files in the meeting, please check whether you have the players installed on your computer by going to https://esa-meeting.webex.com/esa-meeting/systemdiagnosis.php. Sign up for a free trial of WebEx http://www.webex.com/go/mcemfreetrial http://www.webex.com CCP:+442034785289x843184272# IMPORTANT NOTICE: This WebEx service includes a feature that allows audio and any documents and other materials exchanged or viewed during the session to be recorded. By joining this session, you automatically consent to such recordings. If you do not consent to the recording, discuss your concerns with the meeting host prior to the start of the recording or do not join the session. Please note that any such recordings may be subject to discovery in the event of litigation.

1 0

Need clarity: when unparsing dfdl:length expression is evaluated or not?
by Mike Beckerle 07 May '15

07 May '15

I know we discussed this extensively, and there are notes that the IBM DFDL implementation evaluates a dfdl:length expression when unparsing. But searching the mail history, the trackers, etc. I cannot find a clear definitive statement of what we finally decided as to whether this is the official behavior or not. I cannot find language in the DFDL spec that says that the dfdl:length expression is or is not evaluated. (No such problem for dfdl:occursCountKind 'expression'. There we are very clear it is NOT evaluated when unparsing.) There is this statement in the discussion of dfdl:contentLength function: "When unparsing with dfdl:lengthKind "explicit", the calculation of dfdl:contentLength() returns the value of the dfdl:length property." It does not say whether this forces the expression (if dfdl:length is an expression) to be evaluated or not, though it implies it must be evaluated because otherwise it would be undefined in that situation. Is this the only situation during unparsing when the dfdl:length expression is evaluated? ...mike Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 2

Re: [DFDL-WG] [DFDL]: First Release of DFDL4S Parser Library
by Steve Hanson 06 May '15

06 May '15

Dear Michele Good to hear from you. Thank you for letting me know about the latest developments with S2G Data Viewer. I am sure the DFDL Working Group would be interested in a demo of S2G. What is your availability on Tuesdays at 4pm-5pm UK? Meantime do you have a document that you could send which summarises which features of the DFDL language you have used, and some example schemas? Please note that the DFDL 1.0 specification was re-published in October 2014, replacing and obsoleting the version from January 2011. Does S2G comply with the October 2014 version ( https://www.ogf.org/documents/GFD.207.pdf) ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Montserrat Piñol <mpinol(a)eopp.esa.int>, Maurizio De Bartolomei <Maurizio.De.Bartolomei(a)esa.int>, "Rui Mestre (DME)" <rui.mestre(a)deimos.com.pt> Date: 29/04/2015 17:38 Subject: [DFDL]: First Release of DFDL4S Parser Library Dear Steve, we talked some time ago when we presented the tool we developed at ESA to inspect satellite TM and which made use of DFDL description for the binary data and we have included a number of your comments at the time. Since then we have further evolved the tool (called S2G Data Viewer) and produced as well a standalone version of a Java library which performs the function of extracting data field from binary data based on a DFDL description. This library (called DFDL4S i.e. DFDL for Space) is now available to anybody interested. It implements only the subset of the Core capability that we need but I think it might be nevertheless interesting for the DFDL community. In future we plan to expand to write capability the same library so user can serialise their data as per DFDL description and generate realistically looking (simulated) satellite data. The application and the library are available for download at http:eop-cfi.esa.int If there is interest we could arrange a short demo of the app (using webex) or I couldd ask the developer to show the command line test of the DFDL4S library. We are very happy/willing to share results/code/etc. Best Regards PS we are currently thinking to build “transformation” that can ingest different type of descriptive representation and generate the corresponding DFDL schemas. ESA - European Space Agency Michele Zundo Head of Ground System Definition and Verification Office System Support Division Earth Observation Project Department Directorate of Earth Observation Programmes ESTEC Kepleerlaan 1, 2200AZ Noordwijk, Netherlands michele.zundo(a)esa.int | www.esa.int T. +31 71 565 3924 | F +31 71 565 3191 This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

3 6

OGF DFDL WG Call Minutes 2015-04-28
by Steve Hanson 29 Apr '15

29 Apr '15

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13421?download= Next call is 12th May. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] complex nillable ES restriction - more
by Steve Hanson 28 Apr '15

28 Apr '15

Some more on this ... Here's the words from section 13.15 of the spec: DFDL allows elements of complex type to be nillable. However, to avoid the concept of a complex element having a value, which does not exist in DFDL, the only permissible nil value is the empty string, represented by the DFDL %ES; entity and using dfdl:nilKind 'literalValue'. Here's the original words from experience document #2 on this. Note the lengthKind 'implicit' restriction. For a complex element, length plus initiator and terminator enables the nil representation to be established (It is a schema definition error if a complex element is nillable ‘true’ and lengthKind ‘implicit’), but all other representations can only be determined by descending into the complex type for the element. If the descent returns successfully (that is, no unsuppressed processing error occurs) then the other representations may be established. So ... if you can establish the length of the complex element up front then potentially you could check against any nil literal value. Such a check would be different from that for a simple element though, as pad characters are trimmed first, and those properties do not apply to complex elements. I am not sure about lengthKind 'delimited'. With the current ES only rule, the parser can look at the next byte and if it immediately finds an in-scope delimiter, it can match ES. If any nil literal value is allowed, the parser is going to have to scan. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 28/04/2015 09:23 Subject: Re: [DFDL-WG] complex nillable ES restriction It's not an ad-hoc restriction. It's because a complex element has no value, so there are no DFDL properties to describe a value. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 27/04/2015 19:06 Subject: [DFDL-WG] complex nillable ES restriction Sent by: dfdl-wg-bounces(a)ogf.org Right now complex elements can be nillable, but we have this ad-hoc restriction that says the nilValue can only be %ES;. The rationale for this is unclear to me. Can we review what the rationale for this restriction was? Does anyone recall? We have a format where the literal nil value for the complex type wants to be "%WSP*;-%WSP*;" that is, a hyphen, but with surrounding whitespace absorbed. We can model this a different way, but the natural thing to do is to model it as suggested. (This also just happens to run on Daffodil - because we're not detecting this ad-hoc restriction - a bug) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

spec bug - nillable properties say they're only allowed on elements of simple type.
by Mike Beckerle 28 Apr '15

28 Apr '15

The various nil-related properties all have this at the bottom of their descriptions: Annotation: dfdl:element(simpleType) The "(simpleType)" part should be dropped. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

complex nillable ES restriction
by Mike Beckerle 28 Apr '15

28 Apr '15

Right now complex elements can be nillable, but we have this ad-hoc restriction that says the nilValue can only be %ES;. The rationale for this is unclear to me. Can we review what the rationale for this restriction was? Does anyone recall? We have a format where the literal nil value for the complex type wants to be "%WSP*;-%WSP*;" that is, a hyphen, but with surrounding whitespace absorbed. We can model this a different way, but the natural thing to do is to model it as suggested. (This also just happens to run on Daffodil - because we're not detecting this ad-hoc restriction - a bug) Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Action 228: TDML file with tutorial content - mock-up
by Mike Beckerle 27 Apr '15

27 Apr '15

This is a mock up of how a TDML file might be turned into a self-contained executable from which tutorial HTML can be directly generated. I started from tutorial 6. The HTML file should be viewable in any browser. Keep in mind it lacks a corresponding stylesheet, so it's rather ugly as HTML pages go, but it does use this very nice javascript formatter for the data and XML sections which does colorizing (which can be customized as to colors/shading, etc.). I am not sure how to reproduce the bolding of the interesting lines that is done in the original schemas of the tutorial. Nor is there a robust way to refer to line numbers without counting them. But I still think this is pretty interesting. I think the next exercise would be trying to use HTML Table nesting to provide richer data descriptions for binary data where nested boxes are really necessary. I'm thinking along the lines of: <document tutorialBoxes="[ ]">[...abc][def[ghi]jkl][mnop][qrs...]</document> translating the brackets there into boxes that nest in the expected manner. If data contains brackets, then the tutoriaBoxes attribute would be set to use different characters not found in the data. I want to have there be some way to provide data pictures which show nested boxes without a lot of effort.

1 0

OGF DFDL WG Call Agenda 2015-04-28
by Steve Hanson 27 Apr '15

27 Apr '15

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13421?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0