May 2015 - dfdl-wg - lists.ogf.org

still no need for fn:error
by Mike Beckerle 03 Aug '16

03 Aug '16

Here's the example as suggested by Jonathan of how to issue an error that I previously was suggesting needed fn:error. I'm convinced this is sufficient and we can avoid the need for fn:error now. <xs:element name="magic_number" type="ex:uint32" dfdl:byteOrder="bigEndian"> <xs:annotation> <xs:appinfo source="*http://www.ogf.org/dfdl/*<http://www.ogf.org/dfdl/dfdl-1.0/> "> <dfdl:assert test="{ (xs:unsignedInt(.) eq *dfdl:hUInt('0xa1b2c3d4')*) | (xs:unsignedInt(.) eq *dfdl:hUInt('0xd4c3b2a1')*) }" message="{ fn:concat( 'Magic number ', dfdl:hexBinary(dfdl:unsignedInt(.)), ' was not 0xA1B2C3D4 (for bigEndian) or 0xD4C3B2A1 (for littleEndian).' }" /> <dfdl:setVariable ref="ex:bOrd">{ xs:unsignedInt(.) }<dfdl:setVariable> </xs:appinfo> </xs:annotation> </xs:element> -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 11

clarification on choice unparsing and empty sequences
by Mike Beckerle 11 Aug '15

11 Aug '15

The spec language around how choice branches are resolved when unparsing suggests that each branch must have an element in it somewhere. On unparsing there is the question of how one identifies the appropriate schema choice branch corresponding to the data in the infoset. This is complicated by the fact that the children may not be elements. They may themselves be sequences or choices.The selection of the choice branch is as follows: The element in the infoset is used to search the choice branches in the schema, in schema definition order, but without looking inside any complex elements. If the element occurs in a branch, then that branch is selected and if subsequently a processing error occurs, this selection is not revisited (that is, there is no backtracking). To avoid any unintended behavior, all the children of a choice can be modeled as elements. However, that passage of the spec seems incomplete now to me. Though at the time it was written and reviewed it seemed to be the solution to the issue. It does imply that each model group in a choice branch has to have an element somewhere, but I couldn't find that statement explicitly in the spec, and the spec does say empty model groups are specifically allowed. 14.1 Empty Sequences A sequence having no children is syntactically legal in DFDL. In the data stream, such a sequence can have non-zero length *LeftFraming* and *RightFraming* regions, but the SequenceContent region in between must be empty. It is a processing error if the SequenceContent region of an empty sequence has non-zero length when parsing. This leaves open the issue of hidden groups e.g., <sequence dfdl:hiddenGroupRef="mygroup"/> is an empty sequence? Or does the hidden group count as if there really are children of this sequence? I suspect the latter, but need to see how others have interpreted this. XML schema does not define an empty sequence that is the content model of a complex type definition as effective content so any DFDL annotations on such a construct would be ignored. It is a schema definition error if the empty sequence is the content model of a complex type, or if a complex type has nothing in its content model at all. That makes clear that both these are SDE: <complexType><sequence/></complexType> <complexType></complexType> But it leaves many scenarios unclear still. Consider this schema fragment: .... <choice> <sequence dfdl:terminator=";"/> <element name="foo"/> </choice> <element name="bar"/> .... If we are unparsing and the infoset is just <bar> then we can compute that finding <bar> in the infoset means the first branch is selected. So we would unparse that and output a ";". However, if there is true ambiguity like: <choice> <sequence dfdl:terminator=";"/> <sequence dfdl:terminator="/"/> <element name="foo"/> </choice> <element name="bar"/> That's effectively saying on parse either a ";" or a "/" may be found. On unparse, there is nothing to guide which choice branch other than telling us if <bar> is found, that the <foo> element branch is NOT selected. However, it is ok to just output the first always (;) if the infoset has a <bar> element. However, I wanted to check and see what others interpretation of the spec is for this issue. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call 2015-05-26 - cancelled
by Steve Hanson 26 May '15

26 May '15

Steve not available. Next meeting is June 9th. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: DFDL Working Group Call
by Steve Hanson 26 May '15

26 May '15

Steve unable to make the call

1 0

OGF DFDL WG Call Minutes 2015-05-12
by Steve Hanson 20 May '15

20 May '15

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13427?download= Next call is 26th May. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2015-05-12
by Steve Hanson 11 May '15

11 May '15

Please note change to usual format. Topic: S2G and DFDL demo Date: Tuesday, 12. May 2015 Time: 16:00, GMT Summer Time (London, GMT+01:00) Meeting Number: 843 184 272 Meeting Password: toto1234 ------------------------------------------------------- To join the online meeting (Now from mobile devices!) ------------------------------------------------------- 1. Go to https://esa-meeting.webex.com/esa-meeting/j.php?MTID=me3a599ee88e3b896337d8… 2. If requested, enter your name and email address. 3. If a password is required, enter the meeting password: toto1234 4. Click "Join". To view in other time zones or languages, please click the link: https://esa-meeting.webex.com/esa-meeting/j.php?MTID=ma20877926dd694b83b532… ------------------------------------------------------- To join the audio conference only ------------------------------------------------------- Call-in toll number (UK): +44-203-478-5289 Global call-in numbers: https://esa-meeting.webex.com/esa-meeting/globalcallin.php?serviceType=MC&E… Access code:843 184 272 ------------------------------------------------------- For assistance ------------------------------------------------------- 1. Go to https://esa-meeting.webex.com/esa-meeting/mc 2. On the left navigation bar, click "Support". Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Confirmed: DFDL Working Group Call (9 Jun 16:00 GDT in IBM Hursley DE3S09)
by Steve Hanson 11 May '15

11 May '15

OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg DFDL Getting Started: http://www.ibm.com/developerworks/library/se-dfdl/ DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Call details changed just for this week, to accommodate demp of S2G Data Viewer, a satellite data parsing tool that uses a small subset of DFDL. Topic: S2G and DFDL demo Date: Tuesday, 12. May 2015 Time: 16:00, GMT Summer Time (London, GMT+01:00) Meeting Number: 843 184 272 Meeting Password: toto1234 ------------------------------------------------------- To join the online meeting (Now from mobile devices!) ------------------------------------------------------- 1. Go to https://esa-meeting.webex.com/esa-meeting/j.php?MTID=me3a599ee88e3b896337d8… 2. If requested, enter your name and email address. 3. If a password is required, enter the meeting password: toto1234 4. Click "Join". To view in other time zones or languages, please click the link: https://esa-meeting.webex.com/esa-meeting/j.php?MTID=ma20877926dd694b83b532… ------------------------------------------------------- To join the audio conference only ------------------------------------------------------- Call-in toll number (UK): +44-203-478-5289 Global call-in numbers: https://esa-meeting.webex.com/esa-meeting/globalcallin.php?serviceType=MC&E… Access code:843 184 272 ------------------------------------------------------- For assistance ------------------------------------------------------- 1. Go to https://esa-meeting.webex.com/esa-meeting/mc 2. On the left navigation bar, click "Support".

1 0

Fwd: (Forward to attendees) Meeting rescheduled: S2G and DFDL demo
by Michele Zundo 11 May '15

11 May '15

here with the right time > Begin forwarded message: > > From: Michele Zundo <messenger(a)webex.com> > Subject: (Forward to attendees) Meeting rescheduled: S2G and DFDL demo > Date: 11 May 2015 16:02:28 CEST > To: michele.zundo(a)esa.int > Reply-To: michele.zundo(a)esa.int > > > **** You can forward this email invitation to attendees **** > > Hello , > > Michele Zundo changed the time for this online meeting. > > Topic: S2G and DFDL demo > Date: Tuesday, 12. May 2015 > Time: 16:00, GMT Summer Time (London, GMT+01:00) > Meeting Number: 843 184 272 > Meeting Password: toto1234 > > > ------------------------------------------------------- > To join the online meeting (Now from mobile devices!) > ------------------------------------------------------- > 1. Go to https://esa-meeting.webex.com/esa-meeting/j.php?MTID=m6fb2895e83484bf4bc082… <https://esa-meeting.webex.com/esa-meeting/j.php?MTID=m6fb2895e83484bf4bc082…> > 2. If requested, enter your name and email address. > 3. If a password is required, enter the meeting password: toto1234 > 4. Click "Join". > > To view in other time zones or languages, please click the link: > https://esa-meeting.webex.com/esa-meeting/j.php?MTID=m67ac6d2d2d74f27f9c177… <https://esa-meeting.webex.com/esa-meeting/j.php?MTID=m67ac6d2d2d74f27f9c177…> > > ------------------------------------------------------- > To join the audio conference only > ------------------------------------------------------- > Call-in toll number (UK): +44-203-478-5289 > Global call-in numbers: https://esa-meeting.webex.com/esa-meeting/globalcallin.php?serviceType=MC&E… <https://esa-meeting.webex.com/esa-meeting/globalcallin.php?serviceType=MC&E…> > > Access code:843 184 272 > > ------------------------------------------------------- > For assistance > ------------------------------------------------------- > 1. Go to https://esa-meeting.webex.com/esa-meeting/mc <https://esa-meeting.webex.com/esa-meeting/mc> > 2. On the left navigation bar, click "Support". > > You can contact me at: > michele.zundo(a)esa.int <mailto:michele.zundo@esa.int> > > > To update this meeting to your calendar program (for example Microsoft Outlook), click this link: > https://esa-meeting.webex.com/esa-meeting/j.php?MTID=m48fb87157eef88b251782… <https://esa-meeting.webex.com/esa-meeting/j.php?MTID=m48fb87157eef88b251782…> > > > WebEx will automatically setup Meeting Manager for Windows the first time you join a meeting. To save time, you can setup prior to the meeting by clicking this link: > https://esa-meeting.webex.com/esa-meeting/meetingcenter/mcsetup.php <https://esa-meeting.webex.com/esa-meeting/meetingcenter/mcsetup.php> > > > The playback of UCF (Universal Communications Format) rich media files requires appropriate players. To view this type of rich media files in the meeting, please check whether you have the players installed on your computer by going to https://esa-meeting.webex.com/esa-meeting/systemdiagnosis.php <https://esa-meeting.webex.com/esa-meeting/systemdiagnosis.php>. > > Sign up for a free trial of WebEx > http://www.webex.com/go/mcemfreetrial <http://www.webex.com/go/mcemfreetrial> > > http://www.webex.com <http://www.webex.com/> > > CCP:+442034785289x843184272# > > IMPORTANT NOTICE: This WebEx service includes a feature that allows audio and any documents and other materials exchanged or viewed during the session to be recorded. By joining this session, you automatically consent to such recordings. If you do not consent to the recording, discuss your concerns with the meeting host prior to the start of the recording or do not join the session. Please note that any such recordings may be subject to discovery in the event of litigation. ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: michele.zundo(a)esa.int This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email.

1 0

Meeting rescheduled: S2G and DFDL demo
by Michele Zundo 11 May '15

11 May '15

Hello , Michele Zundo changed the time for this online meeting. Topic: S2G and DFDL demo Date: Tuesday, 12. May 2015 Time: 16:00, GMT Summer Time (London, GMT+01:00) Meeting Number: 843 184 272 Meeting Password: toto1234 ------------------------------------------------------- To join the online meeting (Now from mobile devices!) ------------------------------------------------------- 1. Go to https://esa-meeting.webex.com/esa-meeting/j.php?MTID=me3a599ee88e3b896337d8… 2. If requested, enter your name and email address. 3. If a password is required, enter the meeting password: toto1234 4. Click "Join". To view in other time zones or languages, please click the link: https://esa-meeting.webex.com/esa-meeting/j.php?MTID=ma20877926dd694b83b532… ------------------------------------------------------- To join the audio conference only ------------------------------------------------------- Call-in toll number (UK): +44-203-478-5289 Global call-in numbers: https://esa-meeting.webex.com/esa-meeting/globalcallin.php?serviceType=MC&E… Access code:843 184 272 ------------------------------------------------------- For assistance ------------------------------------------------------- 1. Go to https://esa-meeting.webex.com/esa-meeting/mc 2. On the left navigation bar, click "Support". You can contact me at: michele.zundo(a)esa.int To update this meeting to your calendar program (for example Microsoft Outlook), click this link: https://esa-meeting.webex.com/esa-meeting/j.php?MTID=ma7a7c17623138cef68c2f… WebEx will automatically setup Meeting Manager for Windows the first time you join a meeting. To save time, you can setup prior to the meeting by clicking this link: https://esa-meeting.webex.com/esa-meeting/meetingcenter/mcsetup.php The playback of UCF (Universal Communications Format) rich media files requires appropriate players. To view this type of rich media files in the meeting, please check whether you have the players installed on your computer by going to https://esa-meeting.webex.com/esa-meeting/systemdiagnosis.php. Sign up for a free trial of WebEx http://www.webex.com/go/mcemfreetrial http://www.webex.com CCP:+442034785289x843184272# IMPORTANT NOTICE: This WebEx service includes a feature that allows audio and any documents and other materials exchanged or viewed during the session to be recorded. By joining this session, you automatically consent to such recordings. If you do not consent to the recording, discuss your concerns with the meeting host prior to the start of the recording or do not join the session. Please note that any such recordings may be subject to discovery in the event of litigation.

1 0

Re: [DFDL-WG] [DFDL]: First Release of DFDL4S Parser Library
by Steve Hanson 11 May '15

11 May '15

Dear Michele I can have a call at 16:00 CET tomorrow but if you want other people from DFDL Working Group to attend then it should be at 16:00 UK (GMT+1 = BST). This is an hour behind CET. Please email webex details to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Maurizio De Bartolomei <Maurizio.De.Bartolomei(a)esa.int>, Montserrat Piñol <mpinol(a)eopp.esa.int>, "Rui Mestre (DME)" <rui.mestre(a)deimos.com.pt> Date: 11/05/2015 13:05 Subject: Re: [DFDL]: First Release of DFDL4S Parser Library Dear Steve, 1) I can confirm that we have used version 207 of the DFDL as reference. 2) The DFDL4S implementation and the delta between the Core DFDL spec and what has been implemented is documented in 3) Schemas can be easily exported by the S2G app, but I attach some for reference (they are all inside the jar) 3) We can have a teleconf Tomorrow at 16:00 CET if you want, Should I set up a Webex to share the screen ? Regards Michele On 06 May 2015, at 10:49 , Steve Hanson <smh(a)uk.ibm.com> wrote: Dear Michele Good to hear from you. Thank you for letting me know about the latest developments with S2G Data Viewer. I am sure the DFDL Working Group would be interested in a demo of S2G. What is your availability on Tuesdays at 4pm-5pm UK? Meantime do you have a document that you could send which summarises which features of the DFDL language you have used, and some example schemas? Please note that the DFDL 1.0 specification was re-published in October 2014, replacing and obsoleting the version from January 2011. Does S2G comply with the October 2014 version ( https://www.ogf.org/documents/GFD.207.pdf) ? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Michele Zundo <michele.zundo(a)esa.int> To: Steve Hanson/UK/IBM@IBMGB Cc: Montserrat Piñol <mpinol(a)eopp.esa.int>, Maurizio De Bartolomei <Maurizio.De.Bartolomei(a)esa.int>, "Rui Mestre (DME)" < rui.mestre(a)deimos.com.pt> Date: 29/04/2015 17:38 Subject: [DFDL]: First Release of DFDL4S Parser Library Dear Steve, we talked some time ago when we presented the tool we developed at ESA to inspect satellite TM and which made use of DFDL description for the binary data and we have included a number of your comments at the time. Since then we have further evolved the tool (called S2G Data Viewer) and produced as well a standalone version of a Java library which performs the function of extracting data field from binary data based on a DFDL description. This library (called DFDL4S i.e. DFDL for Space) is now available to anybody interested. It implements only the subset of the Core capability that we need but I think it might be nevertheless interesting for the DFDL community. In future we plan to expand to write capability the same library so user can serialise their data as per DFDL description and generate realistically looking (simulated) satellite data. The application and the library are available for download at http:eop-cfi.esa.int If there is interest we could arrange a short demo of the app (using webex) or I couldd ask the developer to show the command line test of the DFDL4S library. We are very happy/willing to share results/code/etc. Best Regards PS we are currently thinking to build “transformation” that can ingest different type of descriptive representation and generate the corresponding DFDL schemas. ESA - European Space Agency Michele Zundo Head of Ground System Definition and Verification Office System Support Division Earth Observation Project Department Directorate of Earth Observation Programmes ESTEC Kepleerlaan 1, 2200AZ Noordwijk, Netherlands michele.zundo(a)esa.int | www.esa.int T. +31 71 565 3924 | F +31 71 565 3191 This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ----------------------------------------- Michele Zundo Head of Ground System Definition and Verification Office EOP-PEP European Space Agency, ESTEC e-mail: michele.zundo(a)esa.int Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 2