here is David's email with my response. ---------- Forwarded message ---------- From: Shahbaz Memon <m.memon@fz-juelich.de> Date: Thu, Oct 22, 2009 at 2:21 PM Subject: Re: dmi question To: david.meredith@stfc.ac.uk Hi Dave, My apologies for delay.
Quick question: If I remember correctly, at the last DMI phone meeting you mentioned that you had been working on an approach to define data transfers using a DMI-like approach that catered for multiple data sources and sinks? If true, do you have any more details or schema examples in preparation for next weeks DMI phone meeting ?
I have gone through the schemas on google code. For initiating many transfers, GetDataTransferInstance would yield some requirements for managing them in one go. From schema point of view it would be more optimal to reuse the elements of existing DMI spec. With reference to last telecon, I had some thoughts to avoid major changes in DMI spec. I am not sure if it is trivial or within the DMI scope. But still my comments are as follows; - to use same DataEPR element from DMI as compare to mjsdl:Source or mjsdl:Targets: We can reuse them as they are already in the spec, except the absence of transfer requirements. I am not sure whether its feasible to have individual or group transferrequirements. It seems that you want to apply transferreqs for set of transfers, may be we can also put an optional transferreqs for individual transfer ones (in order to provide flexibility). - jobResourcekey is a single identifier returned from the getdatatransfer operation, but it is pointing to multiple jobs. I.M.H.O If we return ref of each transfer, so that each one can be traceable and monitored, rather than a composite id. My proposition is that we can return all the transfer-ids and a composite-id (group or transaction-id) together to distinguish group of transfers. Generally, this composite-id can also be queried to fetch the bytes-transferred , upon which server returns total bytes-transferred of their children. - Now question comes after the client has already sent the request and intend to monitor transfers (individual / composite). In PGI we did the same work, we came to the conclusion of removing Job interface and re-using the factory interface for managing multiple jobs. Its completely fine here too, because of the overhead of managing multiple web service instances generated within a single request. In PGI, we have introduced the "changestatus" method which takes set of job eprs and transition them to certain state e.g (Pending->Ready) or (Ready->Start) phase. Now the question comes in DMI, if we are taking group transfers as one atomic job, then to manage its state is not a issue. The complexity lies when the group contains multiple individual transfers, and to track or upgrade each of the status will be tricky. Normally, the users of Grid job submission interface always expects that job returns a single status, though there are multiple stage-ins/outs (with parallel different state transitions) are attached. In dataminx, the current schema depicts the use of jsdls. Specifically, in our case, DMI is a data transfer interface, i.e. more orientation towards the data features. And, end users are not taking transfers as grid jobs, though they consider them as data jobs where each transfer (with certain properties) can be steered or monitored. In grid/computing jobs, users are not so much concerned with the attached source->target transfers, rather they are interested in job per se. May be we can discuss them more in our upcoming telecon. We can also see what more can be done to make DMI a simple and clean interface and progress the document uploaded by Mario. Best Regards, Shahbaz Memon On Thu, Oct 22, 2009 at 1:21 PM, <david.meredith@stfc.ac.uk> wrote:
Hi Shabaz,
Quick question: If I remember correctly, at the last DMI phone meeting you mentioned that you had been working on an approach to define data transfers using a DMI-like approach that catered for multiple data sources and sinks? If true, do you have any more details or schema examples in preparation for next weeks DMI phone meeting ?
To date, I haven't really had much time to think about this, but this should change soon (I am currently in the process of handing off my current projects so that I can concentrate on DataMINX/DMI/transfers - scheduled to start Dec 1st).
Kind regards, Dave
------------------------ David Meredith STFC eScience Centre Daresbury Laboratory Warrington Cheshire WA4 4AD Tel: 01925 603762(Direct Line) email: david.meredith@stfc.ac.uk
-- Scanned by iCritical.
------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------