JobService.submitJob() query
Hi, Can anybody clear up this issue for me? I may be missing something in the spec. Thanks, Graeme -3.31 How is the resource manager endpoint for JobService.submitJob() specified? The endpoint is specified by an argument of JobService.runJob(), but is not _obvious_ for submitJob(). Is the endpoint a property of the instance of JobService (with runJob() a 'static' method of the class), or is the endpoint specified as an attribute of JobDefinition? I assume that the contents of the JobDefinition attribute 'SAGA_HostList' is intended to be passed to the resource manager (rather than specify the resource manager itself); for example mapping to the 'CandidateHosts' element of a JSDL document. Is an additional attribute within JobDefinition required to specify the endpoint?
Uhm, I think you got us there. I don't see any way to specify the resource either, so its left completely up to the backend to schedule the job. I am not sure if that was intended. Chris, are we missing something? Did we intend to leave resource specification out? Can't really be, as we have it in run_job as Graeme points out... 'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run. Andre. Quoting [Graeme Pound] (Feb 20 2006):
Hi,
Can anybody clear up this issue for me? I may be missing something in the spec.
Thanks, Graeme
-3.31 How is the resource manager endpoint for JobService.submitJob() specified?
The endpoint is specified by an argument of JobService.runJob(), but is not _obvious_ for submitJob(). Is the endpoint a property of the instance of JobService (with runJob() a 'static' method of the class), or is the endpoint specified as an attribute of JobDefinition?
I assume that the contents of the JobDefinition attribute 'SAGA_HostList' is intended to be passed to the resource manager (rather than specify the resource manager itself); for example mapping to the 'CandidateHosts' element of a JSDL document. Is an additional attribute within JobDefinition required to specify the endpoint?
-- "So much time, so little to do..." -- Garfield
Andre,
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
I do not follow you here. 'SAGA_HostList' is an attribute of JobDefinition (not JobInfo), therefore it should not be read only. Graeme Andre Merzky wrote:
Uhm, I think you got us there. I don't see any way to specify the resource either, so its left completely up to the backend to schedule the job. I am not sure if that was intended.
Chris, are we missing something? Did we intend to leave resource specification out? Can't really be, as we have it in run_job as Graeme points out...
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Hi,
Can anybody clear up this issue for me? I may be missing something in the spec.
Thanks, Graeme
-3.31 How is the resource manager endpoint for JobService.submitJob() specified?
The endpoint is specified by an argument of JobService.runJob(), but is not _obvious_ for submitJob(). Is the endpoint a property of the instance of JobService (with runJob() a 'static' method of the class), or is the endpoint specified as an attribute of JobDefinition?
I assume that the contents of the JobDefinition attribute 'SAGA_HostList' is intended to be passed to the resource manager (rather than specify the resource manager itself); for example mapping to the 'CandidateHosts' element of a JSDL document. Is an additional attribute within JobDefinition required to specify the endpoint?
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource. The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description. Sorry for creating confusion... Andre. Quoting [Graeme Pound] (Feb 20 2006):
Date: Mon, 20 Feb 2006 16:24:32 +0000 From: Graeme Pound <G.E.POUND@soton.ac.uk> To: Andre Merzky <andre@merzky.net> CC: SAGA RG <saga-rg@ggf.org> Subject: Re: [saga-rg] JobService.submitJob() query
Andre,
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
I do not follow you here. 'SAGA_HostList' is an attribute of JobDefinition (not JobInfo), therefore it should not be read only.
Graeme
Andre Merzky wrote:
Uhm, I think you got us there. I don't see any way to specify the resource either, so its left completely up to the backend to schedule the job. I am not sure if that was intended.
Chris, are we missing something? Did we intend to leave resource specification out? Can't really be, as we have it in run_job as Graeme points out...
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Hi,
Can anybody clear up this issue for me? I may be missing something in the spec.
Thanks, Graeme
-3.31 How is the resource manager endpoint for JobService.submitJob() specified?
The endpoint is specified by an argument of JobService.runJob(), but is not _obvious_ for submitJob(). Is the endpoint a property of the instance of JobService (with runJob() a 'static' method of the class), or is the endpoint specified as an attribute of JobDefinition?
I assume that the contents of the JobDefinition attribute 'SAGA_HostList' is intended to be passed to the resource manager (rather than specify the resource manager itself); for example mapping to the 'CandidateHosts' element of a JSDL document. Is an additional attribute within JobDefinition required to specify the endpoint?
-- "So much time, so little to do..." -- Garfield
Andre, I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads: SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF) This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document. The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition. Graeme Andre Merzky wrote:
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource.
The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description.
Sorry for creating confusion...
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Date: Mon, 20 Feb 2006 16:24:32 +0000 From: Graeme Pound <G.E.POUND@soton.ac.uk> To: Andre Merzky <andre@merzky.net> CC: SAGA RG <saga-rg@ggf.org> Subject: Re: [saga-rg] JobService.submitJob() query
Andre,
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run. I do not follow you here. 'SAGA_HostList' is an attribute of JobDefinition (not JobInfo), therefore it should not be read only.
Graeme
Andre Merzky wrote:
Uhm, I think you got us there. I don't see any way to specify the resource either, so its left completely up to the backend to schedule the job. I am not sure if that was intended.
Chris, are we missing something? Did we intend to leave resource specification out? Can't really be, as we have it in run_job as Graeme points out...
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Hi,
Can anybody clear up this issue for me? I may be missing something in the spec.
Thanks, Graeme
-3.31 How is the resource manager endpoint for JobService.submitJob() specified?
The endpoint is specified by an argument of JobService.runJob(), but is not _obvious_ for submitJob(). Is the endpoint a property of the instance of JobService (with runJob() a 'static' method of the class), or is the endpoint specified as an attribute of JobDefinition?
I assume that the contents of the JobDefinition attribute 'SAGA_HostList' is intended to be passed to the resource manager (rather than specify the resource manager itself); for example mapping to the 'CandidateHosts' element of a JSDL document. Is an additional attribute within JobDefinition required to specify the endpoint?
Hi Graeme, I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess... I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right? Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible. Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource). One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess). But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion? Thanks, Andre. Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme
Andre Merzky wrote:
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource.
The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description.
Sorry for creating confusion...
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Date: Mon, 20 Feb 2006 16:24:32 +0000 From: Graeme Pound <G.E.POUND@soton.ac.uk> To: Andre Merzky <andre@merzky.net> CC: SAGA RG <saga-rg@ggf.org> Subject: Re: [saga-rg] JobService.submitJob() query
Andre,
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run. I do not follow you here. 'SAGA_HostList' is an attribute of JobDefinition (not JobInfo), therefore it should not be read only.
Graeme
Andre Merzky wrote:
Uhm, I think you got us there. I don't see any way to specify the resource either, so its left completely up to the backend to schedule the job. I am not sure if that was intended.
Chris, are we missing something? Did we intend to leave resource specification out? Can't really be, as we have it in run_job as Graeme points out...
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Hi,
Can anybody clear up this issue for me? I may be missing something in the spec.
Thanks, Graeme
-3.31 How is the resource manager endpoint for JobService.submitJob() specified?
The endpoint is specified by an argument of JobService.runJob(), but is not _obvious_ for submitJob(). Is the endpoint a property of the instance of JobService (with runJob() a 'static' method of the class), or is the endpoint specified as an attribute of JobDefinition?
I assume that the contents of the JobDefinition attribute 'SAGA_HostList' is intended to be passed to the resource manager (rather than specify the resource manager itself); for example mapping to the 'CandidateHosts' element of a JSDL document. Is an additional attribute within JobDefinition required to specify the endpoint?
-- "So much time, so little to do..." -- Garfield
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
I would go for the resource manager as the thing being specified. The notion of specifying a host seems strange with large clusters & resource brokers. If the underlying middleware exposes each underlying host as an execution target... then effectively each host has its own 'resource manager' capability.
The host in run_job would then specify the target resource ("" for 'any resource' I guess).
On its own I'd consider this an invalid argument. If there are additional attributes being provided to qualify resource selection... maybe OK. But the "" I would see as input to a resource broker which would select a resource manager as the execution target. Not a capability embedded into the current SAGA layer. My 2p. Steven -- ---------------------------------------------------------------- Dr Steven Newhouse Tel:+44 (0)2380 598789 Director, Open Middleware Infrastructure Institute-UK (OMII-UK) c/o Suite 6005, Faraday Building (B21), Highfield Campus, Southampton University, Highfield, Southampton, SO17 1BJ, UK
Hi Steven, You are right, specifying the RM instead of target host makes sense in the cases you mention. Fine with me. Thanks, Andre. Quoting [Steven Newhouse] (Feb 22 2006):
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
I would go for the resource manager as the thing being specified. The notion of specifying a host seems strange with large clusters & resource brokers. If the underlying middleware exposes each underlying host as an execution target... then effectively each host has its own 'resource manager' capability.
The host in run_job would then specify the target resource ("" for 'any resource' I guess).
On its own I'd consider this an invalid argument. If there are additional attributes being provided to qualify resource selection... maybe OK. But the "" I would see as input to a resource broker which would select a resource manager as the execution target. Not a capability embedded into the current SAGA layer. My 2p.
Steven
-- "So much time, so little to do..." -- Garfield
Andre, I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager. The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager") I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers. I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings) Graeme Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme
Andre Merzky wrote:
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource.
The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description.
Sorry for creating confusion...
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Date: Mon, 20 Feb 2006 16:24:32 +0000 From: Graeme Pound <G.E.POUND@soton.ac.uk> To: Andre Merzky <andre@merzky.net> CC: SAGA RG <saga-rg@ggf.org> Subject: Re: [saga-rg] JobService.submitJob() query
Andre,
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run. I do not follow you here. 'SAGA_HostList' is an attribute of JobDefinition (not JobInfo), therefore it should not be read only.
Graeme
Andre Merzky wrote:
Uhm, I think you got us there. I don't see any way to specify the resource either, so its left completely up to the backend to schedule the job. I am not sure if that was intended.
Chris, are we missing something? Did we intend to leave resource specification out? Can't really be, as we have it in run_job as Graeme points out...
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Hi,
Can anybody clear up this issue for me? I may be missing something in the spec.
Thanks, Graeme
-3.31 How is the resource manager endpoint for JobService.submitJob() specified?
The endpoint is specified by an argument of JobService.runJob(), but is not _obvious_ for submitJob(). Is the endpoint a property of the instance of JobService (with runJob() a 'static' method of the class), or is the endpoint specified as an attribute of JobDefinition?
I assume that the contents of the JobDefinition attribute 'SAGA_HostList' is intended to be passed to the resource manager (rather than specify the resource manager itself); for example mapping to the 'CandidateHosts' element of a JSDL document. Is an additional attribute within JobDefinition required to specify the endpoint?
I agree that version 3 is least flexible. Stevens comment also indicated that 1 or 2 would be appropriate IIUC, with host defining the resource manager contact, not the target host. I would think that (1) is more flexible, and better in sync with run_job. For example it would allow to submit a job to a number of gram job managers (i.e. to a number of hosts) w/o the need for creating n job_service objects. So, I'd vote for (1). Cheers, Andre. Quoting [Graeme Pound] (Feb 22 2006):
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme
Andre Merzky wrote:
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource.
The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description.
Sorry for creating confusion...
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Date: Mon, 20 Feb 2006 16:24:32 +0000 From: Graeme Pound <G.E.POUND@soton.ac.uk> To: Andre Merzky <andre@merzky.net> CC: SAGA RG <saga-rg@ggf.org> Subject: Re: [saga-rg] JobService.submitJob() query
Andre,
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run. I do not follow you here. 'SAGA_HostList' is an attribute of JobDefinition (not JobInfo), therefore it should not be read only.
Graeme
Andre Merzky wrote:
Uhm, I think you got us there. I don't see any way to specify the resource either, so its left completely up to the backend to schedule the job. I am not sure if that was intended.
Chris, are we missing something? Did we intend to leave resource specification out? Can't really be, as we have it in run_job as Graeme points out...
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
Andre.
Quoting [Graeme Pound] (Feb 20 2006): >Hi, > >Can anybody clear up this issue for me? I may be missing something in >the spec. > >Thanks, >Graeme > > >-3.31 How is the resource manager endpoint for JobService.submitJob() >specified? > >The endpoint is specified by an argument of JobService.runJob(), but >is not _obvious_ for submitJob(). Is the endpoint a property of the >instance of JobService (with runJob() a 'static' method of the >class), or is the endpoint specified as an attribute of JobDefinition? > >I assume that the contents of the JobDefinition attribute >'SAGA_HostList' is intended to be passed to the resource manager >(rather than specify the resource manager itself); for example >mapping to the 'CandidateHosts' element of a JSDL document. Is an >additional attribute within JobDefinition required to specify the >endpoint?
-- "So much time, so little to do..." -- Garfield
I'll finally provide some context first .... As Andre mentioned in a previous email, we had left the choice of resource manager up to the implementation of the library, and didn't expose it up into the API layer. Feedback indicates that this was a mistake. :-) I can go for either 1 or 2, with a preference for 2. The reason is that to me the JobService represents the service endpoint to a job scheduler or resource manager. That said, I'm not religious about it. -- Chris On 22/2/06 01:27, "Graeme Pound" <G.E.POUND@soton.ac.uk> wrote:
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme
Andre Merzky wrote:
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource.
The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description.
Sorry for creating confusion...
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Date: Mon, 20 Feb 2006 16:24:32 +0000 From: Graeme Pound <G.E.POUND@soton.ac.uk> To: Andre Merzky <andre@merzky.net> CC: SAGA RG <saga-rg@ggf.org> Subject: Re: [saga-rg] JobService.submitJob() query
Andre,
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run. I do not follow you here. 'SAGA_HostList' is an attribute of JobDefinition (not JobInfo), therefore it should not be read only.
Graeme
Andre Merzky wrote:
Uhm, I think you got us there. I don't see any way to specify the resource either, so its left completely up to the backend to schedule the job. I am not sure if that was intended.
Chris, are we missing something? Did we intend to leave resource specification out? Can't really be, as we have it in run_job as Graeme points out...
'SAGA_HostList' is on the job, and read only - it gives information on where the job _is_ running, not where it _should_ run.
Andre.
Quoting [Graeme Pound] (Feb 20 2006): > Hi, > > Can anybody clear up this issue for me? I may be missing something in > the spec. > > Thanks, > Graeme > > > -3.31 How is the resource manager endpoint for JobService.submitJob() > specified? > > The endpoint is specified by an argument of JobService.runJob(), but is > not _obvious_ for submitJob(). Is the endpoint a property of the > instance of JobService (with runJob() a 'static' method of the class), > or is the endpoint specified as an attribute of JobDefinition? > > I assume that the contents of the JobDefinition attribute > 'SAGA_HostList' is intended to be passed to the resource manager > (rather than specify the resource manager itself); for example mapping > to the 'CandidateHosts' element of a JSDL document. Is an additional > attribute within JobDefinition required to specify the endpoint?
Hi Chris, Graeme, if we go for 2, we should actually remove the RM contact from run_job, as that would conflict with the RM contact given on the job_service creation. Cheers, Andre. Quoting [Christopher Smith] (Mar 06 2006):
I'll finally provide some context first ....
As Andre mentioned in a previous email, we had left the choice of resource manager up to the implementation of the library, and didn't expose it up into the API layer. Feedback indicates that this was a mistake. :-)
I can go for either 1 or 2, with a preference for 2. The reason is that to me the JobService represents the service endpoint to a job scheduler or resource manager. That said, I'm not religious about it.
-- Chris
On 22/2/06 01:27, "Graeme Pound" <G.E.POUND@soton.ac.uk> wrote:
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme
Andre Merzky wrote:
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource.
The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description.
Sorry for creating confusion...
Andre.
Quoting [Graeme Pound] (Feb 20 2006):
Date: Mon, 20 Feb 2006 16:24:32 +0000 From: Graeme Pound <G.E.POUND@soton.ac.uk> To: Andre Merzky <andre@merzky.net> CC: SAGA RG <saga-rg@ggf.org> Subject: Re: [saga-rg] JobService.submitJob() query
Andre,
> 'SAGA_HostList' is on the job, and read only - it gives > information on where the job _is_ running, not where it > _should_ run. I do not follow you here. 'SAGA_HostList' is an attribute of JobDefinition (not JobInfo), therefore it should not be read only.
Graeme
Andre Merzky wrote: > Uhm, I think you got us there. I don't see any way to > specify the resource either, so its left completely up to > the backend to schedule the job. I am not sure if that was > intended. > > Chris, are we missing something? Did we intend to leave > resource specification out? Can't really be, as we have it > in run_job as Graeme points out... > > 'SAGA_HostList' is on the job, and read only - it gives > information on where the job _is_ running, not where it > _should_ run. > > Andre. > > > Quoting [Graeme Pound] (Feb 20 2006): >> Hi, >> >> Can anybody clear up this issue for me? I may be missing something in >> the spec. >> >> Thanks, >> Graeme >> >> >> -3.31 How is the resource manager endpoint for JobService.submitJob() >> specified? >> >> The endpoint is specified by an argument of JobService.runJob(), but is >> not _obvious_ for submitJob(). Is the endpoint a property of the >> instance of JobService (with runJob() a 'static' method of the class), >> or is the endpoint specified as an attribute of JobDefinition? >> >> I assume that the contents of the JobDefinition attribute >> 'SAGA_HostList' is intended to be passed to the resource manager >> (rather than specify the resource manager itself); for example mapping >> to the 'CandidateHosts' element of a JSDL document. Is an additional >> attribute within JobDefinition required to specify the endpoint? >
-- "So much time, so little to do..." -- Garfield
run_job doesn't have an RM contact, but is actually specifying the end resource to run on (a FQDN of a compute node). It's there mostly to support things like ssh backends. -- Chris On 6/3/06 13:59, "Andre Merzky" <andre@merzky.net> wrote:
Hi Chris, Graeme,
if we go for 2, we should actually remove the RM contact from run_job, as that would conflict with the RM contact given on the job_service creation.
Cheers, Andre.
Quoting [Christopher Smith] (Mar 06 2006):
I'll finally provide some context first ....
As Andre mentioned in a previous email, we had left the choice of resource manager up to the implementation of the library, and didn't expose it up into the API layer. Feedback indicates that this was a mistake. :-)
I can go for either 1 or 2, with a preference for 2. The reason is that to me the JobService represents the service endpoint to a job scheduler or resource manager. That said, I'm not religious about it.
-- Chris
On 22/2/06 01:27, "Graeme Pound" <G.E.POUND@soton.ac.uk> wrote:
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme
Andre Merzky wrote:
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource.
The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description.
Sorry for creating confusion...
Andre.
Quoting [Graeme Pound] (Feb 20 2006): > Date: Mon, 20 Feb 2006 16:24:32 +0000 > From: Graeme Pound <G.E.POUND@soton.ac.uk> > To: Andre Merzky <andre@merzky.net> > CC: SAGA RG <saga-rg@ggf.org> > Subject: Re: [saga-rg] JobService.submitJob() query > > Andre, > > >> 'SAGA_HostList' is on the job, and read only - it gives >> information on where the job _is_ running, not where it >> _should_ run. > I do not follow you here. 'SAGA_HostList' is an attribute of > JobDefinition (not JobInfo), therefore it should not be read only. > > Graeme > > > Andre Merzky wrote: >> Uhm, I think you got us there. I don't see any way to >> specify the resource either, so its left completely up to >> the backend to schedule the job. I am not sure if that was >> intended. >> >> Chris, are we missing something? Did we intend to leave >> resource specification out? Can't really be, as we have it >> in run_job as Graeme points out... >> >> 'SAGA_HostList' is on the job, and read only - it gives >> information on where the job _is_ running, not where it >> _should_ run. >> >> Andre. >> >> >> Quoting [Graeme Pound] (Feb 20 2006): >>> Hi, >>> >>> Can anybody clear up this issue for me? I may be missing something in >>> the spec. >>> >>> Thanks, >>> Graeme >>> >>> >>> -3.31 How is the resource manager endpoint for JobService.submitJob() >>> specified? >>> >>> The endpoint is specified by an argument of JobService.runJob(), but >>> is >>> not _obvious_ for submitJob(). Is the endpoint a property of the >>> instance of JobService (with runJob() a 'static' method of the class), >>> or is the endpoint specified as an attribute of JobDefinition? >>> >>> I assume that the contents of the JobDefinition attribute >>> 'SAGA_HostList' is intended to be passed to the resource manager >>> (rather than specify the resource manager itself); for example mapping >>> to the 'CandidateHosts' element of a JSDL document. Is an additional >>> attribute within JobDefinition required to specify the endpoint? >>
Hi Chris, Quoting [Christopher Smith] (Mar 07 2006):
run_job doesn't have an RM contact, but is actually specifying the end resource to run on (a FQDN of a compute node). It's there mostly to support things like ssh backends.
For SSH or GRAM, what would be the difference between a "resource management contact" and a the "end resource"? Would there be any difference? If the "RM contact" is "my_rm://rm.grid.net:123/", what does the "end resource" be used for: as a hint for the RM where to run on? Thanks, Andre.
-- Chris
On 6/3/06 13:59, "Andre Merzky" <andre@merzky.net> wrote:
Hi Chris, Graeme,
if we go for 2, we should actually remove the RM contact from run_job, as that would conflict with the RM contact given on the job_service creation.
Cheers, Andre.
Quoting [Christopher Smith] (Mar 06 2006):
I'll finally provide some context first ....
As Andre mentioned in a previous email, we had left the choice of resource manager up to the implementation of the library, and didn't expose it up into the API layer. Feedback indicates that this was a mistake. :-)
I can go for either 1 or 2, with a preference for 2. The reason is that to me the JobService represents the service endpoint to a job scheduler or resource manager. That said, I'm not religious about it.
-- Chris
On 22/2/06 01:27, "Graeme Pound" <G.E.POUND@soton.ac.uk> wrote:
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme
Andre Merzky wrote: > Oops, you are right! My wrong - mixed it up with > ExecutionHosts. Well, then I really was off target: > SAGA_HostList is then indeed what you should use to specify > the target resource. > > The run_job would, in its simpliest implementation, create a > job description with SAGA_HostList set to the specified > endpoint, and do a submit_job on that description. > > Sorry for creating confusion... > > Andre. > > > Quoting [Graeme Pound] (Feb 20 2006): >> Date: Mon, 20 Feb 2006 16:24:32 +0000 >> From: Graeme Pound <G.E.POUND@soton.ac.uk> >> To: Andre Merzky <andre@merzky.net> >> CC: SAGA RG <saga-rg@ggf.org> >> Subject: Re: [saga-rg] JobService.submitJob() query >> >> Andre, >> >> >>> 'SAGA_HostList' is on the job, and read only - it gives >>> information on where the job _is_ running, not where it >>> _should_ run. >> I do not follow you here. 'SAGA_HostList' is an attribute of >> JobDefinition (not JobInfo), therefore it should not be read only. >> >> Graeme >> >> >> Andre Merzky wrote: >>> Uhm, I think you got us there. I don't see any way to >>> specify the resource either, so its left completely up to >>> the backend to schedule the job. I am not sure if that was >>> intended. >>> >>> Chris, are we missing something? Did we intend to leave >>> resource specification out? Can't really be, as we have it >>> in run_job as Graeme points out... >>> >>> 'SAGA_HostList' is on the job, and read only - it gives >>> information on where the job _is_ running, not where it >>> _should_ run. >>> >>> Andre. >>> >>> >>> Quoting [Graeme Pound] (Feb 20 2006): >>>> Hi, >>>> >>>> Can anybody clear up this issue for me? I may be missing something in >>>> the spec. >>>> >>>> Thanks, >>>> Graeme >>>> >>>> >>>> -3.31 How is the resource manager endpoint for JobService.submitJob() >>>> specified? >>>> >>>> The endpoint is specified by an argument of JobService.runJob(), but >>>> is >>>> not _obvious_ for submitJob(). Is the endpoint a property of the >>>> instance of JobService (with runJob() a 'static' method of the class), >>>> or is the endpoint specified as an attribute of JobDefinition? >>>> >>>> I assume that the contents of the JobDefinition attribute >>>> 'SAGA_HostList' is intended to be passed to the resource manager >>>> (rather than specify the resource manager itself); for example mapping >>>> to the 'CandidateHosts' element of a JSDL document. Is an additional >>>> attribute within JobDefinition required to specify the endpoint? >>> > >
-- "So much time, so little to do..." -- Garfield
Ultimately the host specified in run_job turns into the SAGA_Hostlist attribute with only the one name specified. Yes ... I suppose the RM endpoint is needed somewhere for run_job, but then it's starting to get more complicated, and thus you have to start questioning whether it's needed ...... etc, etc -- Chris On 6/3/06 15:15, "Andre Merzky" <andre@merzky.net> wrote:
Hi Chris,
Quoting [Christopher Smith] (Mar 07 2006):
run_job doesn't have an RM contact, but is actually specifying the end resource to run on (a FQDN of a compute node). It's there mostly to support things like ssh backends.
For SSH or GRAM, what would be the difference between a "resource management contact" and a the "end resource"? Would there be any difference?
If the "RM contact" is "my_rm://rm.grid.net:123/", what does the "end resource" be used for: as a hint for the RM where to run on?
Thanks, Andre.
-- Chris
On 6/3/06 13:59, "Andre Merzky" <andre@merzky.net> wrote:
Hi Chris, Graeme,
if we go for 2, we should actually remove the RM contact from run_job, as that would conflict with the RM contact given on the job_service creation.
Cheers, Andre.
Quoting [Christopher Smith] (Mar 06 2006):
I'll finally provide some context first ....
As Andre mentioned in a previous email, we had left the choice of resource manager up to the implementation of the library, and didn't expose it up into the API layer. Feedback indicates that this was a mistake. :-)
I can go for either 1 or 2, with a preference for 2. The reason is that to me the JobService represents the service endpoint to a job scheduler or resource manager. That said, I'm not religious about it.
-- Chris
On 22/2/06 01:27, "Graeme Pound" <G.E.POUND@soton.ac.uk> wrote:
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006): > Andre, > > I do not think that the attribute 'SAGA_HostList' of > JobDefinition is appropriate either. The description of > 'SAGA_HostList' in the strawman API reads: > > SAGA_HostList > - A list of host names, or host group names, which can be > considered by the resource manager as candidate hosts for > the job. Whether or not the job actually ends up running > on one of the hosts in the list, is solely at the > discretion of the resource manager. Vector of strings. > (JSDL, LSF) > > This attribute should be used to pass information to the > resource manager, NOT to specify the resource manager. For > example it maps to the "CandidateHosts" element of a JSDL > document. > > The first argument of runJob() is the "host name or IP > address of the endpoint which will accept and run the > job". This argument is not defined for submitJob(), nor is > it defined as an attribute of JobDefinition. > > Graeme
> > Andre Merzky wrote: >> Oops, you are right! My wrong - mixed it up with >> ExecutionHosts. Well, then I really was off target: >> SAGA_HostList is then indeed what you should use to specify >> the target resource. >> >> The run_job would, in its simpliest implementation, create a >> job description with SAGA_HostList set to the specified >> endpoint, and do a submit_job on that description. >> >> Sorry for creating confusion... >> >> Andre. >> >> >> Quoting [Graeme Pound] (Feb 20 2006): >>> Date: Mon, 20 Feb 2006 16:24:32 +0000 >>> From: Graeme Pound <G.E.POUND@soton.ac.uk> >>> To: Andre Merzky <andre@merzky.net> >>> CC: SAGA RG <saga-rg@ggf.org> >>> Subject: Re: [saga-rg] JobService.submitJob() query >>> >>> Andre, >>> >>> >>>> 'SAGA_HostList' is on the job, and read only - it gives >>>> information on where the job _is_ running, not where it >>>> _should_ run. >>> I do not follow you here. 'SAGA_HostList' is an attribute of >>> JobDefinition (not JobInfo), therefore it should not be read only. >>> >>> Graeme >>> >>> >>> Andre Merzky wrote: >>>> Uhm, I think you got us there. I don't see any way to >>>> specify the resource either, so its left completely up to >>>> the backend to schedule the job. I am not sure if that was >>>> intended. >>>> >>>> Chris, are we missing something? Did we intend to leave >>>> resource specification out? Can't really be, as we have it >>>> in run_job as Graeme points out... >>>> >>>> 'SAGA_HostList' is on the job, and read only - it gives >>>> information on where the job _is_ running, not where it >>>> _should_ run. >>>> >>>> Andre. >>>> >>>> >>>> Quoting [Graeme Pound] (Feb 20 2006): >>>>> Hi, >>>>> >>>>> Can anybody clear up this issue for me? I may be missing something >>>>> in >>>>> the spec. >>>>> >>>>> Thanks, >>>>> Graeme >>>>> >>>>> >>>>> -3.31 How is the resource manager endpoint for >>>>> JobService.submitJob() >>>>> specified? >>>>> >>>>> The endpoint is specified by an argument of JobService.runJob(), but >>>>> is >>>>> not _obvious_ for submitJob(). Is the endpoint a property of the >>>>> instance of JobService (with runJob() a 'static' method of the >>>>> class), >>>>> or is the endpoint specified as an attribute of JobDefinition? >>>>> >>>>> I assume that the contents of the JobDefinition attribute >>>>> 'SAGA_HostList' is intended to be passed to the resource manager >>>>> (rather than specify the resource manager itself); for example >>>>> mapping >>>>> to the 'CandidateHosts' element of a JSDL document. Is an additional >>>>> attribute within JobDefinition required to specify the endpoint? >>>> >> >>
This is confusing and must be clarified. Some points: - I had assumed that the 'host' argument for job_service.run_job() corresponded to the resource manager/web service interface to which the job would be submitted (and not to the final machine upon which the job will be run). - The implementations should not decide the resource manager. That would be equivalent to the command line tool 'globus-job-submit' having a single GRAM job manager hard-coded. Nor are Job brokers a widely adopted solution at present. I believe that the user expects/should be able to specify the resource manager. - I do not think that it is important for the user to be able to specify the machine upon which the job is run, for the most part this is up to the resource manager. The 'HostList' attribute of job_definition is sufficient. - These terms should be defined in the API to avoid confusion Graeme Christopher Smith wrote:
Ultimately the host specified in run_job turns into the SAGA_Hostlist attribute with only the one name specified.
Yes ... I suppose the RM endpoint is needed somewhere for run_job, but then it's starting to get more complicated, and thus you have to start questioning whether it's needed ...... etc, etc
-- Chris
On 6/3/06 15:15, "Andre Merzky" <andre@merzky.net> wrote:
Hi Chris,
Quoting [Christopher Smith] (Mar 07 2006):
run_job doesn't have an RM contact, but is actually specifying the end resource to run on (a FQDN of a compute node). It's there mostly to support things like ssh backends. For SSH or GRAM, what would be the difference between a "resource management contact" and a the "end resource"? Would there be any difference?
If the "RM contact" is "my_rm://rm.grid.net:123/", what does the "end resource" be used for: as a hint for the RM where to run on?
Thanks, Andre.
-- Chris
On 6/3/06 13:59, "Andre Merzky" <andre@merzky.net> wrote:
Hi Chris, Graeme,
if we go for 2, we should actually remove the RM contact from run_job, as that would conflict with the RM contact given on the job_service creation.
Cheers, Andre.
Quoting [Christopher Smith] (Mar 06 2006):
I'll finally provide some context first ....
As Andre mentioned in a previous email, we had left the choice of resource manager up to the implementation of the library, and didn't expose it up into the API layer. Feedback indicates that this was a mistake. :-)
I can go for either 1 or 2, with a preference for 2. The reason is that to me the JobService represents the service endpoint to a job scheduler or resource manager. That said, I'm not religious about it.
-- Chris
On 22/2/06 01:27, "Graeme Pound" <G.E.POUND@soton.ac.uk> wrote:
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote: > Hi Graeme, > > I am not much of an expert in resource thingies, so what I > say should be taken with a grain of salt. Or rather with a > spoon I guess... > > I assumed that run_job ("myhost", ...) would run the job on > myhost. But what you say (and what the spec implies I > think, after reading again) is that myhost is specifying the > resource manager contact, not the target host. Right? > > Well, then we have the ability to specify a resource manager > contact for run_job, and a target resource on job_submit. > But no combination seems possible. > > Frankly, I think that does not make sense. For one the > semantics between job_run and job_submit should not be that > different (job_run was supposed to be a shortcut for > job_submit). Secondly, there are clearly use cases for both > versions (resource manager contact and target resource). > > One solution would be to specify the resource manager > contact in the constructor of the job_service, as that is > supposed to represent the resource and job manager anyway I > guess. The host in run_job would then specify the target > resource ("" for 'any resource' I guess). > > But again, I am not on firm ground in resource management, > and rather would not like to mess w/o feedback of someone > more knowledgeable. What is your opinion? > > Thanks, Andre. > > > > Quoting [Graeme Pound] (Feb 21 2006): >> Andre, >> >> I do not think that the attribute 'SAGA_HostList' of >> JobDefinition is appropriate either. The description of >> 'SAGA_HostList' in the strawman API reads: >> >> SAGA_HostList >> - A list of host names, or host group names, which can be >> considered by the resource manager as candidate hosts for >> the job. Whether or not the job actually ends up running >> on one of the hosts in the list, is solely at the >> discretion of the resource manager. Vector of strings. >> (JSDL, LSF) >> >> This attribute should be used to pass information to the >> resource manager, NOT to specify the resource manager. For >> example it maps to the "CandidateHosts" element of a JSDL >> document. >> >> The first argument of runJob() is the "host name or IP >> address of the endpoint which will accept and run the >> job". This argument is not defined for submitJob(), nor is >> it defined as an attribute of JobDefinition. >> >> Graeme >> Andre Merzky wrote: >>> Oops, you are right! My wrong - mixed it up with >>> ExecutionHosts. Well, then I really was off target: >>> SAGA_HostList is then indeed what you should use to specify >>> the target resource. >>> >>> The run_job would, in its simpliest implementation, create a >>> job description with SAGA_HostList set to the specified >>> endpoint, and do a submit_job on that description. >>> >>> Sorry for creating confusion... >>> >>> Andre. >>> >>> >>> Quoting [Graeme Pound] (Feb 20 2006): >>>> Date: Mon, 20 Feb 2006 16:24:32 +0000 >>>> From: Graeme Pound <G.E.POUND@soton.ac.uk> >>>> To: Andre Merzky <andre@merzky.net> >>>> CC: SAGA RG <saga-rg@ggf.org> >>>> Subject: Re: [saga-rg] JobService.submitJob() query >>>> >>>> Andre, >>>> >>>> >>>>> 'SAGA_HostList' is on the job, and read only - it gives >>>>> information on where the job _is_ running, not where it >>>>> _should_ run. >>>> I do not follow you here. 'SAGA_HostList' is an attribute of >>>> JobDefinition (not JobInfo), therefore it should not be read only. >>>> >>>> Graeme >>>> >>>> >>>> Andre Merzky wrote: >>>>> Uhm, I think you got us there. I don't see any way to >>>>> specify the resource either, so its left completely up to >>>>> the backend to schedule the job. I am not sure if that was >>>>> intended. >>>>> >>>>> Chris, are we missing something? Did we intend to leave >>>>> resource specification out? Can't really be, as we have it >>>>> in run_job as Graeme points out... >>>>> >>>>> 'SAGA_HostList' is on the job, and read only - it gives >>>>> information on where the job _is_ running, not where it >>>>> _should_ run. >>>>> >>>>> Andre. >>>>> >>>>> >>>>> Quoting [Graeme Pound] (Feb 20 2006): >>>>>> Hi, >>>>>> >>>>>> Can anybody clear up this issue for me? I may be missing something >>>>>> in >>>>>> the spec. >>>>>> >>>>>> Thanks, >>>>>> Graeme >>>>>> >>>>>> >>>>>> -3.31 How is the resource manager endpoint for >>>>>> JobService.submitJob() >>>>>> specified? >>>>>> >>>>>> The endpoint is specified by an argument of JobService.runJob(), but >>>>>> is >>>>>> not _obvious_ for submitJob(). Is the endpoint a property of the >>>>>> instance of JobService (with runJob() a 'static' method of the >>>>>> class), >>>>>> or is the endpoint specified as an attribute of JobDefinition? >>>>>> >>>>>> I assume that the contents of the JobDefinition attribute >>>>>> 'SAGA_HostList' is intended to be passed to the resource manager >>>>>> (rather than specify the resource manager itself); for example >>>>>> mapping >>>>>> to the 'CandidateHosts' element of a JSDL document. Is an additional >>>>>> attribute within JobDefinition required to specify the endpoint? >>> > >
Andre Merzky wrote:
Hi Chris, Graeme,
if we go for 2, we should actually remove the RM contact from run_job, as that would conflict with the RM contact given on the job_service creation.
Cheers, Andre.
Andre, This provoked my question about job_service.run_job() being a 'static' Java method for which job_service need not be instantiated. However this may be beyond the scope of the language independent SAGA API. Graeme
Quoting [Christopher Smith] (Mar 06 2006):
I'll finally provide some context first ....
As Andre mentioned in a previous email, we had left the choice of resource manager up to the implementation of the library, and didn't expose it up into the API layer. Feedback indicates that this was a mistake. :-)
I can go for either 1 or 2, with a preference for 2. The reason is that to me the JobService represents the service endpoint to a job scheduler or resource manager. That said, I'm not religious about it.
-- Chris
On 22/2/06 01:27, "Graeme Pound" <G.E.POUND@soton.ac.uk> wrote:
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme Andre Merzky wrote:
Oops, you are right! My wrong - mixed it up with ExecutionHosts. Well, then I really was off target: SAGA_HostList is then indeed what you should use to specify the target resource.
The run_job would, in its simpliest implementation, create a job description with SAGA_HostList set to the specified endpoint, and do a submit_job on that description.
Sorry for creating confusion...
Andre.
Quoting [Graeme Pound] (Feb 20 2006): > Date: Mon, 20 Feb 2006 16:24:32 +0000 > From: Graeme Pound <G.E.POUND@soton.ac.uk> > To: Andre Merzky <andre@merzky.net> > CC: SAGA RG <saga-rg@ggf.org> > Subject: Re: [saga-rg] JobService.submitJob() query > > Andre, > > >> 'SAGA_HostList' is on the job, and read only - it gives >> information on where the job _is_ running, not where it >> _should_ run. > I do not follow you here. 'SAGA_HostList' is an attribute of > JobDefinition (not JobInfo), therefore it should not be read only. > > Graeme > > > Andre Merzky wrote: >> Uhm, I think you got us there. I don't see any way to >> specify the resource either, so its left completely up to >> the backend to schedule the job. I am not sure if that was >> intended. >> >> Chris, are we missing something? Did we intend to leave >> resource specification out? Can't really be, as we have it >> in run_job as Graeme points out... >> >> 'SAGA_HostList' is on the job, and read only - it gives >> information on where the job _is_ running, not where it >> _should_ run. >> >> Andre. >> >> >> Quoting [Graeme Pound] (Feb 20 2006): >>> Hi, >>> >>> Can anybody clear up this issue for me? I may be missing something in >>> the spec. >>> >>> Thanks, >>> Graeme >>> >>> >>> -3.31 How is the resource manager endpoint for JobService.submitJob() >>> specified? >>> >>> The endpoint is specified by an argument of JobService.runJob(), but is >>> not _obvious_ for submitJob(). Is the endpoint a property of the >>> instance of JobService (with runJob() a 'static' method of the class), >>> or is the endpoint specified as an attribute of JobDefinition? >>> >>> I assume that the contents of the JobDefinition attribute >>> 'SAGA_HostList' is intended to be passed to the resource manager >>> (rather than specify the resource manager itself); for example mapping >>> to the 'CandidateHosts' element of a JSDL document. Is an additional >>> attribute within JobDefinition required to specify the endpoint?
Ah, I see, right. And yes, I think that should be left to the language binding. Andre. Quoting [Graeme Pound] (Mar 07 2006):
Andre Merzky wrote:
Hi Chris, Graeme,
if we go for 2, we should actually remove the RM contact from run_job, as that would conflict with the RM contact given on the job_service creation.
Cheers, Andre.
Andre,
This provoked my question about job_service.run_job() being a 'static' Java method for which job_service need not be instantiated.
However this may be beyond the scope of the language independent SAGA API.
Graeme
Quoting [Christopher Smith] (Mar 06 2006):
I'll finally provide some context first ....
As Andre mentioned in a previous email, we had left the choice of resource manager up to the implementation of the library, and didn't expose it up into the API layer. Feedback indicates that this was a mistake. :-)
I can go for either 1 or 2, with a preference for 2. The reason is that to me the JobService represents the service endpoint to a job scheduler or resource manager. That said, I'm not religious about it.
-- Chris
On 22/2/06 01:27, "Graeme Pound" <G.E.POUND@soton.ac.uk> wrote:
Andre,
I may have confused the issue by mentioning the JobDefinition attribute "SAGA_HostList". This appears to have a valid role mapping to the "CandidateHosts" element of a JSDL document (and also in LSF?). The "SAGA_HostList" attribute should not be used to specify the resource manager.
The problem rather is how is the resource manager (in the form of a URL or machine name) specified when the user invokes submitJob(). There are three possible solutions: 1) Add an argument "host" to the submitJob() method (similar to runJob()) to specify the resource manager 2) Specify the resource manager in the JobService constructor 3) Add a new required attribute to the JobDefinition class (e.g. "SAGA_ResouceManager")
I do not like solution 3 since job definition is conceptually distinct from the choice of resource manager, for example you may wish to define a single JobDefinition and submit that job to several different resource managers.
I do not have a strong preference between solutions 1 and 2. (Although constructors pose a small problem for the Java bindings)
Graeme
Andre Merzky wrote:
Hi Graeme,
I am not much of an expert in resource thingies, so what I say should be taken with a grain of salt. Or rather with a spoon I guess...
I assumed that run_job ("myhost", ...) would run the job on myhost. But what you say (and what the spec implies I think, after reading again) is that myhost is specifying the resource manager contact, not the target host. Right?
Well, then we have the ability to specify a resource manager contact for run_job, and a target resource on job_submit. But no combination seems possible.
Frankly, I think that does not make sense. For one the semantics between job_run and job_submit should not be that different (job_run was supposed to be a shortcut for job_submit). Secondly, there are clearly use cases for both versions (resource manager contact and target resource).
One solution would be to specify the resource manager contact in the constructor of the job_service, as that is supposed to represent the resource and job manager anyway I guess. The host in run_job would then specify the target resource ("" for 'any resource' I guess).
But again, I am not on firm ground in resource management, and rather would not like to mess w/o feedback of someone more knowledgeable. What is your opinion?
Thanks, Andre.
Quoting [Graeme Pound] (Feb 21 2006):
Andre,
I do not think that the attribute 'SAGA_HostList' of JobDefinition is appropriate either. The description of 'SAGA_HostList' in the strawman API reads:
SAGA_HostList - A list of host names, or host group names, which can be considered by the resource manager as candidate hosts for the job. Whether or not the job actually ends up running on one of the hosts in the list, is solely at the discretion of the resource manager. Vector of strings. (JSDL, LSF)
This attribute should be used to pass information to the resource manager, NOT to specify the resource manager. For example it maps to the "CandidateHosts" element of a JSDL document.
The first argument of runJob() is the "host name or IP address of the endpoint which will accept and run the job". This argument is not defined for submitJob(), nor is it defined as an attribute of JobDefinition.
Graeme Andre Merzky wrote: >Oops, you are right! My wrong - mixed it up with >ExecutionHosts. Well, then I really was off target: >SAGA_HostList is then indeed what you should use to specify >the target resource. > >The run_job would, in its simpliest implementation, create a >job description with SAGA_HostList set to the specified >endpoint, and do a submit_job on that description. > >Sorry for creating confusion... > >Andre. > > >Quoting [Graeme Pound] (Feb 20 2006): >>Date: Mon, 20 Feb 2006 16:24:32 +0000 >>From: Graeme Pound <G.E.POUND@soton.ac.uk> >>To: Andre Merzky <andre@merzky.net> >>CC: SAGA RG <saga-rg@ggf.org> >>Subject: Re: [saga-rg] JobService.submitJob() query >> >>Andre, >> >> >>>'SAGA_HostList' is on the job, and read only - it gives >>>information on where the job _is_ running, not where it >>>_should_ run. >>I do not follow you here. 'SAGA_HostList' is an attribute of >>JobDefinition (not JobInfo), therefore it should not be read only. >> >>Graeme >> >> >>Andre Merzky wrote: >>>Uhm, I think you got us there. I don't see any way to >>>specify the resource either, so its left completely up to >>>the backend to schedule the job. I am not sure if that was >>>intended. >>> >>>Chris, are we missing something? Did we intend to leave >>>resource specification out? Can't really be, as we have it >>>in run_job as Graeme points out... >>> >>>'SAGA_HostList' is on the job, and read only - it gives >>>information on where the job _is_ running, not where it >>>_should_ run. >>> >>>Andre. >>> >>> >>>Quoting [Graeme Pound] (Feb 20 2006): >>>>Hi, >>>> >>>>Can anybody clear up this issue for me? I may be missing something >>>>in >>>>the spec. >>>> >>>>Thanks, >>>>Graeme >>>> >>>> >>>>-3.31 How is the resource manager endpoint for >>>>JobService.submitJob() >>>>specified? >>>> >>>>The endpoint is specified by an argument of JobService.runJob(), >>>>but is >>>>not _obvious_ for submitJob(). Is the endpoint a property of the >>>>instance of JobService (with runJob() a 'static' method of the >>>>class), >>>>or is the endpoint specified as an attribute of JobDefinition? >>>> >>>>I assume that the contents of the JobDefinition attribute >>>>'SAGA_HostList' is intended to be passed to the resource manager >>>>(rather than specify the resource manager itself); for example >>>>mapping >>>>to the 'CandidateHosts' element of a JSDL document. Is an >>>>additional >>>>attribute within JobDefinition required to specify the endpoint? >
-- "So much time, so little to do..." -- Garfield
participants (4)
-
Andre Merzky -
Christopher Smith -
Graeme Pound -
Steven Newhouse