October 2009 - ogsa-dmi-wg

PGI and DataMINX Data Transfer Service
by Mario Antonioletti 27 Oct '09

27 Oct '09

Hi, Apologies if you get this more than once - Dave tried to email it to the list earlier today and it appears not to have got through so trying from my account to see if this makes a difference. Mario -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ---------- Forwarded message ---------- Date: Tue, 27 Oct 2009 11:48:40 -0000 From: david.meredith(a)stfc.ac.uk Hi folks, Can I introduce our DataMINX project (http://code.google.com/p/dtsproject ). We are currently involved with a data transfer service for supporting a range of different protocols and would like to adopt and potentially contribute to a relevant OGF spec if appropriate. DMI and JSDL file staging have their strengths and weaknesses in this area, and perhaps there's an opportunity to exploit these strengths within each standard in some way (Steve, hope you don't mind me re-using your words here). DataMINX is a UK-Australia collaboration currently involving the UK NGS, OMII-UK, ARCS and Intersect. May I also point you to our wiki (http://code.google.com/p/dtsproject/wiki/DMI_JSDL_Issues ) where we have put together a summary of some JSDL/DMI issues for use in a bulk data transfer type service. The wiki presents a couple of draft proposals (DMIB and JSDL Data Transfer). Please note both are merely rough proposals with the aim of capturing requirements and for compare/contrast/creating-conjecture (hopefully lots to discuss). For brevity, I have summarised the proposals below, but please do refer to the wiki as there are lots of considerations/requirements and discussion points. Kind regards, Dave Meredith (on behalf of the DataMinx team) DMIB The first proposal ('DMIB' where we have used 'B' to indicate Bulk-transfer) re-uses the existing DMI elements in a new document-literal wrapped style message model. This does not modify the existing DMI (except for the addition of some new <xsd:any> extensions), rather, it wraps together the SourceDEPR and SinkDEPR elements (both <wsa:EndpointReferenceTypeS>) and the <dmi:TransferRequirements> into a single wrapping element. This 'wrapping' element can be defined multiple times in a single request packet in order to define a bulk transfer. (The <dmi:DataLocations>, <dmi:Data> elements are not modified and are re-used as per the dmi spec). Semantically, this appears to be similar to defining multiple source and target JSDL DataStaging elements. JSDL Data Transfer The second draft proposal, defines a <DataTransfer> element (rather than DataStaging). This is used to define some extra elements and modifies the format of the <DataStaging> element to be more suited to bulk transfers, since staging data to/from an intermediary compute resource may not appropriate here (implementation detail). Additions include: a repositioned Credential element, new abstract URIProperties element, client defined sub-transfer id (is used in conjunction with the overall bulk transfer id for drilling down to determine status of sub-transfers), added a modified <dmi:TransferRequirements/> element that adds the <jsdl:CreationFlag/>. -----Original Message----- From: Steve Crouch [mailto:s.crouch@omii.ac.uk] Sent: 26 October 2009 10:18 To: Morris Riedel Cc: Meredith, David (STFC,DL,ESC); gerson galang; Alex Arana; Peter Turner; Mario Antonioletti; Shahbaz Memon Subject: PGI and DataMINX Data Transfer Service Hi Morris, You may recall from the PGI telecon we had on the 2nd October we discussed a set of JSDL requirements for datastaging that have come out of the Australian DataMINX project (including Credential and additional 'URI properties' elements within Source and Target elements). As a recap, they have secured funding to to develop a scalable, production-level, heavy-use Data Transfer Service for deployment, and wish to adopt the relevant OGF standard(s) for the service and contribute to these standards. They're also currently pursuing requirements for an OGSA-DMI rendering within the DMI WG. I've cc'd Peter Turner, PI of the DataMINX project, David Meredith from STFC (UK), Gerson Galang from ARCS, and Alex Arana from Intersect who are involved in the project. Previously, David's been heavily involved in developing the JSDL parameter sweep extension. They can elaborate with far greater precision what is required here. I've also cc'd Mario Antonioletti and Shahbaz Memon from the OGSA-DMI WG, where some progress has been made already within DMI. What we've realised is that both DMI and JSDL have their strengths and weaknesses in this area, and perhaps there's an opportunity to exploit these strengths within each standard in some way. I recall you were planning to come up with a JSDL profile that includes the requirements we discussed e.g. from SRM (amongst others). Are you aware at this stage what your plans are for developing the profile? If it's ok, if you could let us know your plans for the profile and how we could be involved that would be great. Many thanks, Cheers Steve -- Dr Stephen Crouch, Software Architect, OMII-UK, School of Electronics and Computer Science, Room 4067, Level 4, Building 32, University of Southampton, Highfield, Southampton SO17 1BJ Tel: +44 (0)23 8059 8787 EMail: s.crouch(a)omii.ac.uk Fax: +44 (0)23 8059 3045 WWW: http://www.ecs.soton.ac.uk/~stc -- Scanned by iCritical.

1 0

DMI Telcon Minutes
by Mario Antonioletti 26 Oct '09

26 Oct '09

Please let me know if there are corrections/omissions. DMI Telcon - 26 October 2009 ============================= Present ------- Mario Antonioletti, EPCC Peter Turner, University of Sydney Steve Crouch, Southampton Shahbaz Memon, FZJ David Meredith, STFC Gerson Galang, VeRSI Actions: -------- [MAA] Give everybody on this call author privileges once on Grid Forge and joined the OGSA-DMI WG. [MAA] Look at Shahbaz's WSRF DMI rendering. [SM] Find out if adding xsd:any to all complex elements would have a negative impact. [DM] Add suggested changes to the text in the functional spec. [SC] Make introductions to get some DMI/JSDL interaction going. [DM] Reply to Allen's email. [SM] Come up with a straw man proposal for the bulk transfer operation. DONM: 8am UK time (GMT) 23rd of November. --- MAA put a new version of the extensions on document. Very minor changes to this version though. DM updated the content on the DTS wiki: http://code.google.com/p/dtsproject/wiki/DMI_JSDL_Issues Have not migrated to Grid Forge for the time being as Grid Forge has been unstable. For the time being will continue working on this platform - Dave will give access to folks to be able to make edits on the wiki. DM will have more time to DMI based work in December. Will be busy up to that point. SC gave a heads up about a PGI WG telcon for JSDL. They are thinking of moving credentials into the messages for JSDL2 and BES2. For now Morris will try to specify this in a profile which will act as input for JSDL2. SM thought it might be a good idea to make a composite profile for DMI and JSDL to try and bring the two groups closer together. Discussed email sent out by SM: http://www.ogf.org/pipermail/ogsa-dmi-wg/2009-October/000512.html - JSDL src & target & transfer requirements File staging example. JSDL xsd:any can be used to embed transfer requirements. Present in only one. Could be made mandatory. - Resource identification. Job resource key for a single identifier for a bulk transfer operation. Embed multiple jobs in the request. MA noted that we still need to determine what the scope for the work that is going to be done is going to be. What the granularity of the control of the individual transfers is going to be. Seems to be that we shall try to see if we can work at the finest level of granularity and then see how far that takes us. There was a discussion about how to identify the sub-transfers. DM keen on having a system like: job_id:sub_job_id where the sub_job_id can be specified by the client to have a meaningful name. SM not keen on doing this - the server should specify the sub_job_id. MA suggested using aliasing. DM thought that the Schema could be used to specify that the sub_job_id was unique - which had been SM's concern. - Monitoring and Control of transfers SM thinks we can remove a port type for data transfers and change the status methods - possibly only have one operation and parameterise the inputs to specify the change of state. In PGI they moved the control management to the factory. DM would be happy with this approach as it would make it easier to have a RESTful approach SM is going to come up with a straw man proposal using pseudo schema which he's going to send to the mailing list for discussion. DONM: 8am UK time (GMT) 23rd of November. +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+ -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Fwd: dmi question
by Shahbaz Memon 26 Oct '09

26 Oct '09

here is David's email with my response. ---------- Forwarded message ---------- From: Shahbaz Memon <m.memon(a)fz-juelich.de> Date: Thu, Oct 22, 2009 at 2:21 PM Subject: Re: dmi question To: david.meredith(a)stfc.ac.uk Hi Dave, My apologies for delay. > Quick question: If I remember correctly, at the last DMI phone meeting you mentioned that you had been working on an approach to define data transfers using a DMI-like approach that catered for multiple data sources and sinks? If true, do you have any more details or schema examples in preparation for next weeks DMI phone meeting ? I have gone through the schemas on google code. For initiating many transfers, GetDataTransferInstance would yield some requirements for managing them in one go. From schema point of view it would be more optimal to reuse the elements of existing DMI spec. With reference to last telecon, I had some thoughts to avoid major changes in DMI spec. I am not sure if it is trivial or within the DMI scope. But still my comments are as follows; - to use same DataEPR element from DMI as compare to mjsdl:Source or mjsdl:Targets: We can reuse them as they are already in the spec, except the absence of transfer requirements. I am not sure whether its feasible to have individual or group transferrequirements. It seems that you want to apply transferreqs for set of transfers, may be we can also put an optional transferreqs for individual transfer ones (in order to provide flexibility). - jobResourcekey is a single identifier returned from the getdatatransfer operation, but it is pointing to multiple jobs. I.M.H.O If we return ref of each transfer, so that each one can be traceable and monitored, rather than a composite id. My proposition is that we can return all the transfer-ids and a composite-id (group or transaction-id) together to distinguish group of transfers. Generally, this composite-id can also be queried to fetch the bytes-transferred , upon which server returns total bytes-transferred of their children. - Now question comes after the client has already sent the request and intend to monitor transfers (individual / composite). In PGI we did the same work, we came to the conclusion of removing Job interface and re-using the factory interface for managing multiple jobs. Its completely fine here too, because of the overhead of managing multiple web service instances generated within a single request. In PGI, we have introduced the "changestatus" method which takes set of job eprs and transition them to certain state e.g (Pending->Ready) or (Ready->Start) phase. Now the question comes in DMI, if we are taking group transfers as one atomic job, then to manage its state is not a issue. The complexity lies when the group contains multiple individual transfers, and to track or upgrade each of the status will be tricky. Normally, the users of Grid job submission interface always expects that job returns a single status, though there are multiple stage-ins/outs (with parallel different state transitions) are attached. In dataminx, the current schema depicts the use of jsdls. Specifically, in our case, DMI is a data transfer interface, i.e. more orientation towards the data features. And, end users are not taking transfers as grid jobs, though they consider them as data jobs where each transfer (with certain properties) can be steered or monitored. In grid/computing jobs, users are not so much concerned with the attached source->target transfers, rather they are interested in job per se. May be we can discuss them more in our upcoming telecon. We can also see what more can be done to make DMI a simple and clean interface and progress the document uploaded by Mario. Best Regards, Shahbaz Memon On Thu, Oct 22, 2009 at 1:21 PM, <david.meredith(a)stfc.ac.uk> wrote: > Hi Shabaz, > > Quick question: If I remember correctly, at the last DMI phone meeting you mentioned that you had been working on an approach to define data transfers using a DMI-like approach that catered for multiple data sources and sinks? If true, do you have any more details or schema examples in preparation for next weeks DMI phone meeting ? > > To date, I haven't really had much time to think about this, but this should change soon (I am currently in the process of handing off my current projects so that I can concentrate on DataMINX/DMI/transfers - scheduled to start Dec 1st). > > Kind regards, > Dave > > > ------------------------ > David Meredith > STFC eScience Centre > Daresbury Laboratory > Warrington Cheshire > WA4 4AD > Tel: 01925 603762(Direct Line) > email: david.meredith(a)stfc.ac.uk > > -- > Scanned by iCritical. > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ >

1 0

Reminder Telcon.
by Mario Antonioletti 25 Oct '09

25 Oct '09

Hi, Looks like it was an extra hour in bed so 8am will now seem like 9am. Anyhow will be having a telcon at 8am UK time (GMT) to discuss progress (or lack of). Usual numbers apply: Call details ============ Participant access code: 23057332 then # Primary dial-in number: 0844 561 0081 or if dialling internationally +44 2078193600 Or local access alternatives: USA West Coast 4089616553 USA East Coast 7183541169 Germany (0)6950070811 Finland (0)969379595 Australia 0282239343 Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+ -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Postponed Telcon
by Mario Antonioletti 14 Oct '09

14 Oct '09

Folks, We were scheduled to have a telcon this Friday (16th October) this has now been rescheduled for Monday 26th of October at 8am UK time. Note that clocks go back on Sunday 25th in the UK so we will be working at GMT time or to UK folks it will seem like 7am despite the fact that it will really be 8am. Will try to remember to send a reminder and an agenda nearer the time. Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+ -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Notes from today's Telcon
by Mario Antonioletti 01 Oct '09

01 Oct '09

Hi, Apologies these are a bit rough and only give an indication of what was discussed. If folks that attended the call would like to supplement these notes then please do so. Mario ---- DMI Telcon - 01 October 2009 ============================ Present ------- Mario Antonioletti, EPCC Peter Turner, University of Sydney Steve Crouch, Southampton Shahbaz Memon, FZJ David Meredith, STFC Gerson Galang, VeRSI Actions: -------- [MAA] Give everybody on this call author privileges once on Grid Forge and joined the OGSA-DMI WG. [MAA] Look at Shahbaz's WSRF DMI rendering. [SM] Find out if adding xsd:any to all complex elements would have a negative impact. [DM] Add suggested changes to the text in the functional spec. Completed Actions: ------------------ [MAA] To come up with a template document to explore if required functionality can be put in a spec addendum/new spec. Have started a rough document. http://forge.gridforum.org/sf/go/doc15764?nav=1 [MAA] Ask Allen to comment on the proposal on the wiki. Did, Allen sent out comments. [DM] Enumerate the extensibility points required in the DMI spec. <dmi:Data/> element, add <xsd:any/> to all dmi complex types? [MAA] Try and get Steve Crouch involved. Steve said that he would be able to attend. --- MAA put a very rough document to explore the extensions being proposed by the DataMinx folks on the OGSA-DMI WG pages. Discussed the proposed extensions to the existing element, mainly adding an xsd:any to the <dmi:Data/> element. Possibly do this for all complex elements. Has an impact on interoperability but can make these optional extensions and later standardise through profiles. Discussed the fact that that DataMinx do not use an explicit factory - instance pattern. They use JMS to post and distribute messages. Simplest scenario from/to JMS queues syphoned off to back end nodes that actually do the transport - they want to make sure the implementation is agnostic of the underlying data movement protocol. However this does not preclude the fact that they use a factory like interface. Currently have a slight schism in the group over the encapsulation of multiple data transfers by a DTI. Allen's points were discussed on the call and Dave is going to post a reply in the near future. The group is going to use a mixture of the GridForge document mario posted and the GridForge wiki to continue exploring these possible extensions. Dave keen to have the data transfers represented by documents. Mario said that that could turn into a data movement workflow and it could fall outside the remit of OGSA-DMI. The group will try to explore this further. Date of next call: October 16th 9am UK time. +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+ -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Document
by Mario Antonioletti 01 Oct '09

01 Oct '09

Folks, I have finally put up a very rough document - still more like a skeleton than anything else. It contains modified versions of David Meredith's text and is available at: http://forge.gridforum.org/sf/go/doc15764?nav=1 I would like to keep the token for a little bit longer before passing it on to someone else. I would like to note that this document, being the zeroth iteration, does not have any group consesus as yet. Would like this to become a strawman once complete for the extensions that are being proposed. Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+ -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0