Parameter Sweep and BES Factory
Hi! I'm just wondering -- is there any agreed view on how Parameter Sweep Extension should interact with BES Factory job submission? My main concern is whether the submission should return an activity that hides all those individual tasks that the parameter sweep produced (say, 1000 jobs from one JSDL...) or should it be possible to somehow treat them individually? The problem with the former is that it would only be possible to get a some kind of heuristic view of the state of the activity (say, the activity is in running state if at least one of the individual tasks has moved to this state). The problem with the later is, in my understanding, the fact that the CreateActivity yields one and only one ActivityIdentifier. -- Piotr Domagalski
I would treat this as a two part problem... The current model in BES and JSDL is that one JSDL document passed into the CreateActivity yields one EPR to reference/manage it. I don't see any compelling reason to move away from that. To deal with the state representation of the many parameter sweep jobs I would look to use the BES state extension model to capture the details of the number of queued, running or finished parameter sweep tasks WITHIN the running state. I think this would need to be captured in a short profile document along the lines of the HPCBP File Staging extension. If you want to manage each task within an activity there is an undefined Activity port type in the BES specification that you could look to define for individual parameter sweep tasks if you really wanted to... but I'm not convinced of the use case and need for this. Steven
-----Original Message----- From: jsdl-wg-bounces@ogf.org [mailto:jsdl-wg-bounces@ogf.org] On Behalf Of Piotr Domagalski Sent: Tuesday, July 22, 2008 10:32 AM To: jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: [jsdl-wg] Parameter Sweep and BES Factory
Hi!
I'm just wondering -- is there any agreed view on how Parameter Sweep Extension should interact with BES Factory job submission? My main concern is whether the submission should return an activity that hides all those individual tasks that the parameter sweep produced (say, 1000 jobs from one JSDL...) or should it be possible to somehow treat them individually?
The problem with the former is that it would only be possible to get a some kind of heuristic view of the state of the activity (say, the activity is in running state if at least one of the individual tasks has moved to this state).
The problem with the later is, in my understanding, the fact that the CreateActivity yields one and only one ActivityIdentifier.
-- Piotr Domagalski -- jsdl-wg mailing list jsdl-wg@ogf.org http://www.ogf.org/mailman/listinfo/jsdl-wg
On Tue, Jul 22, 2008 at 8:18 PM, Steven Newhouse <Steven.Newhouse@microsoft.com> wrote:
The current model in BES and JSDL is that one JSDL document passed into the CreateActivity yields one EPR to reference/manage it. I don't see any compelling reason to move away from that.
On one hand, this could be the easiest solution to this problem -- it would break the WSDL, though. On the other, this could be bad in terms of performance. I'd rather not see CreateActivity response with thousands of EPRs... In fact, I see one use case of using parameter sweep to be the performance gain of hiding the details of a complex job (in terms of numerous tasks).
To deal with the state representation of the many parameter sweep jobs I would look to use the BES state extension model to capture the details of the number of queued, running or finished parameter sweep tasks WITHIN the running state. I think this would need to be captured in a short profile document along the lines of the HPCBP File Staging extension.
That looks like a sensible approach. I'd be happy to see such a profile in the near future...
If you want to manage each task within an activity there is an undefined Activity port type in the BES specification that you could look to define for individual parameter sweep tasks if you really wanted to... but I'm not convinced of the use case and need for this.
Neither do I. My main concern is monitoring the state of such an activity, which could take a long time because of many tasks and having some "heuristic state" could be not an option. -- Piotr Domagalski
Alternatively, a "job manager" - could understand the JSDL extensions and make a sequence of calls on one or more BES instances. I'd like to keep the BES's from becoming too complex. A
-----Original Message----- From: ogsa-bes-wg-bounces@ogf.org [mailto:ogsa-bes-wg-bounces@ogf.org] On Behalf Of Steven Newhouse Sent: Tuesday, July 22, 2008 2:19 PM To: Piotr Domagalski; jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: Re: [OGSA-BES-WG] [jsdl-wg] Parameter Sweep and BES Factory
I would treat this as a two part problem...
The current model in BES and JSDL is that one JSDL document passed into the CreateActivity yields one EPR to reference/manage it. I don't see any compelling reason to move away from that.
To deal with the state representation of the many parameter sweep jobs I would look to use the BES state extension model to capture the details of the number of queued, running or finished parameter sweep tasks WITHIN the running state. I think this would need to be captured in a short profile document along the lines of the HPCBP File Staging extension.
If you want to manage each task within an activity there is an undefined Activity port type in the BES specification that you could look to define for individual parameter sweep tasks if you really wanted to... but I'm not convinced of the use case and need for this.
Steven
-----Original Message----- From: jsdl-wg-bounces@ogf.org [mailto:jsdl-wg-bounces@ogf.org] On Behalf Of Piotr Domagalski Sent: Tuesday, July 22, 2008 10:32 AM To: jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: [jsdl-wg] Parameter Sweep and BES Factory
Hi!
I'm just wondering -- is there any agreed view on how Parameter Sweep Extension should interact with BES Factory job submission? My main concern is whether the submission should return an activity that hides all those individual tasks that the parameter sweep produced (say, 1000 jobs from one JSDL...) or should it be possible to somehow treat them individually?
The problem with the former is that it would only be possible to get a some kind of heuristic view of the state of the activity (say, the activity is in running state if at least one of the individual tasks has moved to this state).
The problem with the later is, in my understanding, the fact that the CreateActivity yields one and only one ActivityIdentifier.
-- Piotr Domagalski -- jsdl-wg mailing list jsdl-wg@ogf.org http://www.ogf.org/mailman/listinfo/jsdl-wg
-- ogsa-bes-wg mailing list ogsa-bes-wg@ogf.org http://www.ogf.org/mailman/listinfo/ogsa-bes-wg
Hi all, I am delighted to see the Parameter Sweep extension getting grounded and connected with other Standards and Profiles. To drop my quick thoughts n this - and I know Steve will wince on this one - but a combining Parameter Sweep with the current BES interface is a perfect use case for WSRF ServiceGroup - in a WSRF rendering of BES: Instead of returning thousands of EPRs to all individual jobs of the sweep, the BES container that understands and accepts Parameter Sweeps may return the EPR to the ServiceGroup. Maybe the *concepts* (and not necessarily the WSRF rendering) of ServiceGroup might be of some use here ... Cheers, Michel On 22 Jul 2008, at 20:31, Andrew Grimshaw wrote:
Alternatively, a "job manager" - could understand the JSDL extensions and make a sequence of calls on one or more BES instances. I'd like to keep the BES's from becoming too complex.
A
-----Original Message----- From: ogsa-bes-wg-bounces@ogf.org [mailto:ogsa-bes-wg- bounces@ogf.org] On Behalf Of Steven Newhouse Sent: Tuesday, July 22, 2008 2:19 PM To: Piotr Domagalski; jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: Re: [OGSA-BES-WG] [jsdl-wg] Parameter Sweep and BES Factory
I would treat this as a two part problem...
The current model in BES and JSDL is that one JSDL document passed into the CreateActivity yields one EPR to reference/manage it. I don't see any compelling reason to move away from that.
To deal with the state representation of the many parameter sweep jobs I would look to use the BES state extension model to capture the details of the number of queued, running or finished parameter sweep tasks WITHIN the running state. I think this would need to be captured in a short profile document along the lines of the HPCBP File Staging extension.
If you want to manage each task within an activity there is an undefined Activity port type in the BES specification that you could look to define for individual parameter sweep tasks if you really wanted to... but I'm not convinced of the use case and need for this.
Steven
-----Original Message----- From: jsdl-wg-bounces@ogf.org [mailto:jsdl-wg-bounces@ogf.org] On Behalf Of Piotr Domagalski Sent: Tuesday, July 22, 2008 10:32 AM To: jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: [jsdl-wg] Parameter Sweep and BES Factory
Hi!
I'm just wondering -- is there any agreed view on how Parameter Sweep Extension should interact with BES Factory job submission? My main concern is whether the submission should return an activity that hides all those individual tasks that the parameter sweep produced (say, 1000 jobs from one JSDL...) or should it be possible to somehow treat them individually?
The problem with the former is that it would only be possible to get a some kind of heuristic view of the state of the activity (say, the activity is in running state if at least one of the individual tasks has moved to this state).
The problem with the later is, in my understanding, the fact that the CreateActivity yields one and only one ActivityIdentifier.
-- Piotr Domagalski -- jsdl-wg mailing list jsdl-wg@ogf.org http://www.ogf.org/mailman/listinfo/jsdl-wg
-- ogsa-bes-wg mailing list ogsa-bes-wg@ogf.org http://www.ogf.org/mailman/listinfo/ogsa-bes-wg
-- ogsa-bes-wg mailing list ogsa-bes-wg@ogf.org http://www.ogf.org/mailman/listinfo/ogsa-bes-wg
-- Michel Drescher Fujitsu Laboratories of Europe, Ltd. Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE Reg. No. 4153469 +44 20 8606 4834 Michel.Drescher@uk.fujitsu.com
+1 Steve has anyone looked at Enumeration being used in this way? Can you filter the stream of results from an enumeration to extract the ERP's pointing to the services you want? On 22 Jul 2008, at 21:41, Michel Drescher wrote:
Hi all,
I am delighted to see the Parameter Sweep extension getting grounded and connected with other Standards and Profiles.
To drop my quick thoughts n this - and I know Steve will wince on this one - but a combining Parameter Sweep with the current BES interface is a perfect use case for WSRF ServiceGroup - in a WSRF rendering of BES: Instead of returning thousands of EPRs to all individual jobs of the sweep, the BES container that understands and accepts Parameter Sweeps may return the EPR to the ServiceGroup.
Maybe the *concepts* (and not necessarily the WSRF rendering) of ServiceGroup might be of some use here ...
Cheers, Michel
On 22 Jul 2008, at 20:31, Andrew Grimshaw wrote:
Alternatively, a "job manager" - could understand the JSDL extensions and make a sequence of calls on one or more BES instances. I'd like to keep the BES's from becoming too complex.
A
-----Original Message----- From: ogsa-bes-wg-bounces@ogf.org [mailto:ogsa-bes-wg- bounces@ogf.org] On Behalf Of Steven Newhouse Sent: Tuesday, July 22, 2008 2:19 PM To: Piotr Domagalski; jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: Re: [OGSA-BES-WG] [jsdl-wg] Parameter Sweep and BES Factory
I would treat this as a two part problem...
The current model in BES and JSDL is that one JSDL document passed into the CreateActivity yields one EPR to reference/manage it. I don't see any compelling reason to move away from that.
To deal with the state representation of the many parameter sweep jobs I would look to use the BES state extension model to capture the details of the number of queued, running or finished parameter sweep tasks WITHIN the running state. I think this would need to be captured in a short profile document along the lines of the HPCBP File Staging extension.
If you want to manage each task within an activity there is an undefined Activity port type in the BES specification that you could look to define for individual parameter sweep tasks if you really wanted to... but I'm not convinced of the use case and need for this.
Steven
-----Original Message----- From: jsdl-wg-bounces@ogf.org [mailto:jsdl-wg-bounces@ogf.org] On Behalf Of Piotr Domagalski Sent: Tuesday, July 22, 2008 10:32 AM To: jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: [jsdl-wg] Parameter Sweep and BES Factory
Hi!
I'm just wondering -- is there any agreed view on how Parameter Sweep Extension should interact with BES Factory job submission? My main concern is whether the submission should return an activity that hides all those individual tasks that the parameter sweep produced (say, 1000 jobs from one JSDL...) or should it be possible to somehow treat them individually?
The problem with the former is that it would only be possible to get a some kind of heuristic view of the state of the activity (say, the activity is in running state if at least one of the individual tasks has moved to this state).
The problem with the later is, in my understanding, the fact that the CreateActivity yields one and only one ActivityIdentifier.
-- Piotr Domagalski -- jsdl-wg mailing list jsdl-wg@ogf.org http://www.ogf.org/mailman/listinfo/jsdl-wg
-- ogsa-bes-wg mailing list ogsa-bes-wg@ogf.org http://www.ogf.org/mailman/listinfo/ogsa-bes-wg
-- ogsa-bes-wg mailing list ogsa-bes-wg@ogf.org http://www.ogf.org/mailman/listinfo/ogsa-bes-wg
-- Michel Drescher Fujitsu Laboratories of Europe, Ltd. Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE Reg. No. 4153469
+44 20 8606 4834 Michel.Drescher@uk.fujitsu.com
-- ogsa-bes-wg mailing list ogsa-bes-wg@ogf.org http://www.ogf.org/mailman/listinfo/ogsa-bes-wg
Take care: Dr. David Snelling < David . Snelling . UK . Fujitsu . com > Fujitsu Laboratories of Europe Limited Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE Reg. No. 4153469 +44-208-606-4649 (Office) +44-7768-807526 (Mobile)
All, Service groups are interesting - I have an alternative that we have already implemented - use RNS ... then one can use standard directory ops to look at the jobs. A
-----Original Message----- From: Michel Drescher [mailto:Michel.Drescher@uk.fujitsu.com] Sent: Tuesday, July 22, 2008 4:41 PM To: Andrew Grimshaw Cc: 'Steven Newhouse'; 'Piotr Domagalski'; jsdl-wg@ogf.org; ogsa-bes- wg@ogf.org Subject: Re: [OGSA-BES-WG] [jsdl-wg] Parameter Sweep and BES Factory
Hi all,
I am delighted to see the Parameter Sweep extension getting grounded and connected with other Standards and Profiles.
To drop my quick thoughts n this - and I know Steve will wince on this one - but a combining Parameter Sweep with the current BES interface is a perfect use case for WSRF ServiceGroup - in a WSRF rendering of BES: Instead of returning thousands of EPRs to all individual jobs of the sweep, the BES container that understands and accepts Parameter Sweeps may return the EPR to the ServiceGroup.
Maybe the *concepts* (and not necessarily the WSRF rendering) of ServiceGroup might be of some use here ...
Cheers, Michel
On 22 Jul 2008, at 20:31, Andrew Grimshaw wrote:
Alternatively, a "job manager" - could understand the JSDL extensions and make a sequence of calls on one or more BES instances. I'd like to keep the BES's from becoming too complex.
A
-----Original Message----- From: ogsa-bes-wg-bounces@ogf.org [mailto:ogsa-bes-wg- bounces@ogf.org] On Behalf Of Steven Newhouse Sent: Tuesday, July 22, 2008 2:19 PM To: Piotr Domagalski; jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: Re: [OGSA-BES-WG] [jsdl-wg] Parameter Sweep and BES Factory
I would treat this as a two part problem...
The current model in BES and JSDL is that one JSDL document passed into the CreateActivity yields one EPR to reference/manage it. I don't see any compelling reason to move away from that.
To deal with the state representation of the many parameter sweep jobs I would look to use the BES state extension model to capture the details of the number of queued, running or finished parameter sweep tasks WITHIN the running state. I think this would need to be captured in a short profile document along the lines of the HPCBP File Staging extension.
If you want to manage each task within an activity there is an undefined Activity port type in the BES specification that you could look to define for individual parameter sweep tasks if you really wanted to... but I'm not convinced of the use case and need for this.
Steven
-----Original Message----- From: jsdl-wg-bounces@ogf.org [mailto:jsdl-wg-bounces@ogf.org] On Behalf Of Piotr Domagalski Sent: Tuesday, July 22, 2008 10:32 AM To: jsdl-wg@ogf.org; ogsa-bes-wg@ogf.org Subject: [jsdl-wg] Parameter Sweep and BES Factory
Hi!
I'm just wondering -- is there any agreed view on how Parameter Sweep Extension should interact with BES Factory job submission? My main concern is whether the submission should return an activity that hides all those individual tasks that the parameter sweep produced (say, 1000 jobs from one JSDL...) or should it be possible to somehow treat them individually?
The problem with the former is that it would only be possible to get a some kind of heuristic view of the state of the activity (say, the activity is in running state if at least one of the individual tasks has moved to this state).
The problem with the later is, in my understanding, the fact that the CreateActivity yields one and only one ActivityIdentifier.
-- Piotr Domagalski -- jsdl-wg mailing list jsdl-wg@ogf.org http://www.ogf.org/mailman/listinfo/jsdl-wg
-- ogsa-bes-wg mailing list ogsa-bes-wg@ogf.org http://www.ogf.org/mailman/listinfo/ogsa-bes-wg
-- ogsa-bes-wg mailing list ogsa-bes-wg@ogf.org http://www.ogf.org/mailman/listinfo/ogsa-bes-wg
-- Michel Drescher Fujitsu Laboratories of Europe, Ltd. Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE Reg. No. 4153469
+44 20 8606 4834 Michel.Drescher@uk.fujitsu.com
If you want the granularity of seeing and managing individual jobs submit the workload as individual jobs. If you're submitting a parametric job with many individual tasks I'm not sure why (beyond an overall progress meter) you want any managebility of the individual tasks. If you don't need to do something the problem goes away! No need for service groups or RNS in this particular area. I'm sure they both have many uses elsewhere... along with WS-Enumeration... which I have not looked at for use in this area. May be sometime... Steven
participants (5)
-
Andrew Grimshaw -
David Snelling -
Michel Drescher -
Piotr Domagalski -
Steven Newhouse