See below for the reply from Steven about our current work and linking in with the EGI and their requirements... Cheers Steve -------- Original Message -------- Subject: RE: OGSA-DMI approaches and feedback (& EGI) Date: Mon, 29 Mar 2010 15:14:00 +0200 From: Steven Newhouse <Steven.Newhouse@cern.ch> To: s.crouch@omii.ac.uk <s.crouch@omii.ac.uk> CC: david meredith <david.meredith@stfc.ac.uk>, Steven Newhouse <steven.newhouse@egi.eu> References: <4BACDE02.8090703@omii.ac.uk> Hi Steve,
When you have a moment, I would greatly appreciate any feedback you could offer on some of our current OGSA-DMI work, particularly from the point of view of EGI and its requirements...
At a high-level yes... but I regret I do not have the bandwidth for a detailed examination. Clearly, data is becoming increasingly important. I say data instead of files. Clearly, underpinning the data there is a need to move collections of files about. There are three interfaces here - coordination, transport & storage. Support of SRM that plugs whatever transport layer is used down into the underlying storage layer is important considering the diversity of the different storage systems. Transport - for files I'm not sure there is anything better than GridFTP. At the coordination layer there are implementations - FTS and RFT - I'm not sure if anything has appeared beyond DMI in this space over the last few years.
What has emerged is the common need for specifying bulk data copying, from potentially multiple sources to multiple sinks, using different protocols and credentials per copy.
Keep in mind that files are only one of the sources of data. I can see scenarios where you might want to extract and replicate structured data sources (or subsets of the data) at different sites around the infrastructure so that they are available locally for intensive computation.
Attached is a presentation given by David Meredith at OGF28, that provides an overview of our current efforts, and provides more detail. If you could comment or provide suggestions on the slides that would be most helpful.
More generally where does this DMI activity now sit in relation to the functionality offered by iRODS? How would accessing data through this interfaces (which is seeing some widescale deployment) work?
Also definitely worth mentioning, there is an OMII-UK CSP (remember them? :) called DataMINX. It's funded until the end of this year and with David Meredith as PI, aims to deliver a scalable data transfer service (DTS), based on these open standards and core requirements.
That's interesting. As you know EGI will be offering an open process in incorporating services from national and European projects into its Unified Middleware Distribution - which may be virtual inclusion through a software repository or physical inclusion in a distribution.
Also, we'd ideally like to keep tabs on and investigate such data requirements that emerge from EGI as it progresses - do you have any thoughts on this?
Specifically at the moment - no. But the EC this morning mentioned data as a big concern for them (and therefore by definition EGI) and the frequent ESFRI meetings that I go to mention this as a concern. I can imagine a 'data session' taking place at the EGI technical forum in September in Amsterdam to try and carry forward some activity here. Steven -- Dr Stephen Crouch, Software Architect, OMII-UK, School of Electronics and Computer Science, Room 4067, Level 4, Building 32, University of Southampton, Highfield, Southampton SO17 1BJ Tel: +44 (0)23 8059 8787 EMail: s.crouch@omii.ac.uk Fax: +44 (0)23 8059 3045 WWW: http://www.ecs.soton.ac.uk/~stc