April 2010 - ogsa-dmi-wg

Repost: OGSA-DMI 2010-04-22 Meeting Minutes (embedded within this email for convenience)
by Steve Crouch 23 Apr '10

23 Apr '10

OGSA DMI Skype Telecon - 22 April 2010, 8am BST/GMT+1 ----------------------------------------------------- Present: Mario Antonioletti, EPCC [MAA] Steve Crouch, University of Southampton [STC] David Meredith, STFC [DM] Agenda: 1. Agenda bashing 2. Actions review 3. Progress update 4. Opening up an interaction channel with EGI for data movement -- Response from Steven Newhouse begin_of_the_skype_highlighting end_of_the_skype_highlighting about OGSA-DMI work (esp. bulk data copying) 5. DMI strawman -- Feedback required from group -- Define timeline for merging DMI strawman into func.spec. 6. Phil.Trans.A DTS paper (deadline for camera-ready version Monday 26th) 7. Preliminary OGF29 discussion 8. Planning 9. AOB 10. Date of next meeting New Actions: ------------ [STC] Set up a Google poll for next call - and post to the OGSA-DMI list Completed Actions ----------------- [DM] Clarify a few points on strawman slides (inc. JSDL approaches, in-line vs element references) [STC] After strawman slides updated, send slides to S.Newhouse for feedback and canvassing requirements [SM] To send round link to DMI implementation on SourceForge Ongoing Actions: ---------------- [STC] Draft an extension to OGSA-DMI charter to encompass bulk data copying activity - Edit the version on the GridForge website - the master version [DM/SM] Look at functional spec and evaluate necessary changes w.r.t. strawman - And post up an example [STC] Query Ravi about involvement with OGSA-DMI to engage Globus - and get a technical contact [DM/STC] Ask Andrew Grimshaw and Mark Morgan begin_of_the_skype_highlighting end_of_the_skype_highlighting University of Virginia about: - Element references with BulkDataCopy - DEPRs and EPRs - opinions on how best to proceed - Send presentation to them but summarise in bullet points in the email; include corrorobative statements from S.Newhouse [DM/SM] Improve the strawman document in collaboration - Catch up required to merge changes and discuss points - Need to change functional spec - Couple of days effort - Feedback is important [DM] Upload strawman document to the gridforge in order to have more visibility to other participants - Effectively the presentation uploaded to OGF28 is a first draft - Functional spec - change directly? 3. Progress Update Consensus to table the progress update until next time. 4. Opening up an interaction channel with EGI for data movement S.Newhouse email reply circulated to list with his permission... DMI - abstractions include sources and sinks; which could be anything including databases. Opportunity for scope creep - would we want to consider transformations; from one data type to another? iRODS - how does it fits with DMI and DTS? M.Atkinson - moving computation _to_ the data. What are data sources and data sinks? Perhaps we should look further into classifying these? e.g. OGSA-DAI, SRM, etc. at the moment, goes back to EPR/DEPRs and how to specify the metadata within other data within it? e.g. for DBMS - perhaps could put in SQL direct to access data? Or, could have an abstraction mechanism to group data together at the source end and use that within the DEPR/EPR. Discussion of how to access multiple files within a source; specify DBMS data access requirements in the transfer requirements. Keep tabs on the EMI/UMD process for adopting software for the DTS for its possible inclusion (virtual or otherwise) within the UMD. Data requirements from EGI - data session at the EGI tech forum; perhaps we should have representative at the forum from OGSA-DMI. 5. DMI strawman Consensus to table the DMI strawman discussion until next time. 6. Phil.Trans.A DTS paper DTS paper - D.Meredith to trim down the paper and send out a revised version hopefully today for review. 7. Preliminary OGF29 discussion S.Crouch very likely going so far. Need to do some preliminary work to bring others in. Ask A.Grimshaw about his attendance (when emailing about EPRS/DEPRS and element references). 8. Planning 9. AOB 10. Date of next meeting New action: [STC] Set up a Google poll for next call and post to the OGSA-DMI list.

1 0

Scheduling Doodle for next OGSA-DMI Teleconference
by Steve Crouch 22 Apr '10

22 Apr '10

Hi All, I've set up a Doodle to schedule the next OGSA-DMI teleconference as per my action from the meeting this morning: http://www.doodle.com/9gefe36th5nw9n53 11am BST/GMT+1 is a good time for the Australian Contingent, so have used this for a basis for the possible slots over that week (the Monday is a Bank Holiday in England, so have missed that day out). Cheers Steve -- Dr Stephen Crouch, Software Architect, OMII-UK, School of Electronics and Computer Science, Room 4067, Level 4, Building 32, University of Southampton, Highfield, Southampton SO17 1BJ Tel: +44 (0)23 8059 8787 EMail: s.crouch(a)omii.ac.uk Fax: +44 (0)23 8059 3045 WWW: http://www.ecs.soton.ac.uk/~stc

1 0

OGSA-DMI 2010-04-22 Meeting Minutes
by Steve Crouch 22 Apr '10

22 Apr '10

Hi All, The meeting minutes from the 8am BST/GMT+1 OGSA-DMI call this morning can be found at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.ogsa-dmi-… Please let me know of any corrections, Cheers Steve -- Dr Stephen Crouch, Software Architect, OMII-UK, School of Electronics and Computer Science, Room 4067, Level 4, Building 32, University of Southampton, Highfield, Southampton SO17 1BJ Tel: +44 (0)23 8059 8787 EMail: s.crouch(a)omii.ac.uk Fax: +44 (0)23 8059 3045 WWW: http://www.ecs.soton.ac.uk/~stc

1 0

[Fwd: RE: OGSA-DMI approaches and feedback (& EGI)]
by Steve Crouch 22 Apr '10

22 Apr '10

See below for the reply from Steven about our current work and linking in with the EGI and their requirements... Cheers Steve -------- Original Message -------- Subject: RE: OGSA-DMI approaches and feedback (& EGI) Date: Mon, 29 Mar 2010 15:14:00 +0200 From: Steven Newhouse <Steven.Newhouse(a)cern.ch> To: s.crouch(a)omii.ac.uk <s.crouch(a)omii.ac.uk> CC: david meredith <david.meredith(a)stfc.ac.uk>, Steven Newhouse <steven.newhouse(a)egi.eu> References: <4BACDE02.8090703(a)omii.ac.uk> Hi Steve, > When you have a moment, I would greatly appreciate any feedback you > could offer on some of our current OGSA-DMI work, particularly from the > point of view of EGI and its requirements... At a high-level yes... but I regret I do not have the bandwidth for a detailed examination. Clearly, data is becoming increasingly important. I say data instead of files. Clearly, underpinning the data there is a need to move collections of files about. There are three interfaces here - coordination, transport & storage. Support of SRM that plugs whatever transport layer is used down into the underlying storage layer is important considering the diversity of the different storage systems. Transport - for files I'm not sure there is anything better than GridFTP. At the coordination layer there are implementations - FTS and RFT - I'm not sure if anything has appeared beyond DMI in this space over the last few years. > What has emerged is the common need for specifying bulk data copying, > from potentially multiple sources to multiple sinks, using different > protocols and credentials per copy. Keep in mind that files are only one of the sources of data. I can see scenarios where you might want to extract and replicate structured data sources (or subsets of the data) at different sites around the infrastructure so that they are available locally for intensive computation. > Attached is a presentation given by David Meredith at OGF28, that > provides an overview of our current efforts, and provides more detail. > If you could comment or provide suggestions on the slides that would be > most helpful. More generally where does this DMI activity now sit in relation to the functionality offered by iRODS? How would accessing data through this interfaces (which is seeing some widescale deployment) work? > Also definitely worth mentioning, there is an OMII-UK CSP (remember > them? :) called DataMINX. It's funded until the end of this year and > with David Meredith as PI, aims to deliver a scalable data transfer > service (DTS), based on these open standards and core requirements. That's interesting. As you know EGI will be offering an open process in incorporating services from national and European projects into its Unified Middleware Distribution - which may be virtual inclusion through a software repository or physical inclusion in a distribution. > Also, we'd ideally like to keep tabs on and investigate such data > requirements that emerge from EGI as it progresses - do you have any > thoughts on this? Specifically at the moment - no. But the EC this morning mentioned data as a big concern for them (and therefore by definition EGI) and the frequent ESFRI meetings that I go to mention this as a concern. I can imagine a 'data session' taking place at the EGI technical forum in September in Amsterdam to try and carry forward some activity here. Steven -- Dr Stephen Crouch, Software Architect, OMII-UK, School of Electronics and Computer Science, Room 4067, Level 4, Building 32, University of Southampton, Highfield, Southampton SO17 1BJ Tel: +44 (0)23 8059 8787 EMail: s.crouch(a)omii.ac.uk Fax: +44 (0)23 8059 3045 WWW: http://www.ecs.soton.ac.uk/~stc

1 0

Proposed agenda for Skype call tomorrow: 8am BST/GMT+1 on Thurs 22nd April
by Steve Crouch 21 Apr '10

21 Apr '10

Hi All, See below a proposed agenda for the planned Skype call at 8am BST (GMT+1) on Thursday 22th April (tomorrow): - Agenda bashing - Actions review - Progress update - Opening up an interaction channel with EGI for data movement -- Response from Steven Newhouse about OGSA-DMI work (esp. bulk data copying) - DMI strawman -- Feedback required from group -- Define timeline for merging DMI strawman into func.spec. - Phil.Trans.A DTS paper (deadline for camera-ready version Monday 26th) - Preliminary OGF29 discussion - Planning - AOB - Date of next meeting Anyone interested in joining the skype call please contact me on my skype id: s.crouch2000 . Please feel free to mail the list with any other items you wish to discuss, or amendments. The previous meeting minutes (8am, 25th March) can be found at: http://forge.gridforum.org/sf/go/doc15943?nav=1 Ongoing Actions: ---------------- [STC] Draft an extension to OGSA-DMI charter to encompass bulk data copying activity [DM] Clarify a few points on strawman slides (inc. JSDL approaches, in-line vs element references) [STC] After strawman slides updated, send slides to S.Newhouse for feedback and canvassing requirements [DM/SM] Look at functional spec and evaluate necessary changes w.r.t. strawman [STC] Query Ravi about involvement with OGSA-DMI to engage Globus - and get a technical contact [DM/STC] Ask Andrew Grimshaw and Mark Morgan University of Virginia about: - Element references with BulkDataCopy - DEPRs and EPRs - opinions on how best to proceed [SM] To send round link to DMI implementation on SourceForge [SM] Improve the strawman document in collaboration with DM - Catch up required to merge changes and discuss points - Need to change functional spec [DM] Upload strawman document to the gridforge in order to have more visibility to other participants - Effectively the presentation uploaded to OGF28 is a first draft - Functional spec - change directly? -- Dr Stephen Crouch, Software Architect, OMII-UK, School of Electronics and Computer Science, Room 4067, Level 4, Building 32, University of Southampton, Highfield, Southampton SO17 1BJ Tel: +44 (0)23 8059 8787 EMail: s.crouch(a)omii.ac.uk Fax: +44 (0)23 8059 3045 WWW: http://www.ecs.soton.ac.uk/~stc

1 0