Hi folx, we recently started a discussion thread about workflow support in SAGA, off this list. This mail tries to move that thread onto this list, in order to gauge interest and feedback from the wider community. So, please feel free to jump in! Below are excerpts from the thread (resorted, reformatted, shortened), as starting point. Best regards, Andre.
On Fri, Jun 13, 2008, yasir mehmood wrote:
A colleague of mine (Irfan Habib) is writing a pipeline service as a client to the glueing service which passes it (or plan to pass) a workflow instead of a job. He is concerned about the workflow support in Saga Java Implementations. Could you please confirm this ?
Thilo Kielmann wrote:
Dear Yasir,
The SAGA API (and thus also its implementation in Java) does not deal with workflows, only with individual jobs. You have to construct workflows yourself, using SAGA jobs, containers, etc.
Quoting [Irfan Habib] (Jun 13 2008):
Dear Thilo,
Thank you for your kind reply, may I just ask one more question.
As you have stated that the SAGA API does not deal with workflows only individual jobs, so in that case if I want to execute a workflow through SAGA, I have to externally handle the execution of the workflow and invoke SAGA only for execution of individual jobs?
-----Original Message----- From: Andre Merzky [mailto:andre@merzky.net]
That is correct. What is missing in SAGA for basic workflow support are job dependencies. That is what you need to maintain manually.
Please have a look at the saga::task::container class - that class is supposed to make it easier to manage large numbers of tasks/jobs.
If you have interest in that topic, we could discuss to draft a saga package which adds WF like dependencies to tasks... This is on our roadmap anyway, but so far with low priority, due to lack of interest and use cases.
Best, Andre.
Quoting [Ashiq Anjum] (Jun 17 2008):
Dear Andre,
If this is the case then SAGA introduces a large overhead: The applications (for example the Pipeline service and other services we intend to use) would have to have a middle component which would get the workflow, break it into constituent parts, and invoke the API for each job separately, this component in this scenario would be a enactment engine. In this scenario the Grid middleware at any given time would be scheduling one single job, rather then an entire workflow.
I think this is a bottleneck and users can not do fine grained scheduling of compute and data intensive jobs without this feature. The purpose behind SAGA was to facilitate user by enabling them to transparently access the Grid. But if the users has to code themselves for the workflow dependencies, this will be much more complex for the users and will go against the 20:80 slogan for SAGA. I think this feature will be very helpful to be included in the future releases.
I was also under the impression that glite adapter is already available. Do you have some idea that how far is this from a stable release?
Moreover, did some one submit jobs (using Java implementation of SAGA) on a real execution environment (eg. condor, SGE etc)? It looks like most of the testing has been made using fork based execution but submitting jobs on condor/SGE will require range of JSDL parameters to be sent from the client application through the SAGA API. Can we create JSDL's for complex jobs to be executed on remote execution environments (eg Condor, SGE)?
Thanks,
Best Regards Ashiq Anjum
Dear Ashiq,
we are aware (painfully so) of the fact that SAGA does not yet cover the full scope of programming models used in Grid environments. And yes, workflows are an obvious gap, but so are others: p2p, messaging, checkpoint/recovery, transactions, resorce discovery, service discovery, ...
The list is long, and we are working on a number of those items. As for workflows: we simply did not have enough input from the workflow community in order to make an informed decision on how a workflow oriented job API should look like. It might be enough to simply accept a workflow description and to submit it to an enactor - but which language to use? At the time, it was _very_ unclear if and what workflow language would dominate the scene (I have no idea how the situation is now - is BPAL 'the one'?). And idealy we would also like to express workflows programmatically, too, i.e. as dependencies between saga::job instances.
I have two more comments slightly less defensive, and more constructive I hope :-)
First, writing a workflow enactor in SAGA would be cool, and potentially useful for a range of people and pojects! I understand that this is not on your / Yasirs roadmap, and probably very distracting and time consuming. But you may want to check with other projects if they would be interested in joining forces.
Second, this might be good timing to actually add workflow support to the SAGA standard! We would very much appreciate your input on that topic (although I would like to move this thread to the saga-rg@ogf.org mailing list then). Is that something you would consider spending time on?
These are my toughts on the topic so far - I'd be happy to get your feedback.
Thanks, Andre.
-- Nothing is ever easy.