Hi, with the latest decision from OGF28, we now have the queueMaxSlotsAllowed attribute in the MonitoringSession interface. Under this impression, I meanwhile wonder about the relation between minSlots / maxSlots and the (also newly added) queueName attribute in the job template. Can you specify minSlots / maxSlots without configuring a queueName to use ? And if yes, don't we need an according maxSlotsAllowed on DRM system level then ? Best, Peter.
In SGE minSlots and maxSlots are in tight relation to the parallel environment which is reflected in the new standard with an configuration name. This inter-attribute dependency does not look like a very clean approach, which I mentioned also in the meeting. Regarding SGE minSlots,maxSlots and the queue name have not a mandatory relationship which means that selecting a special queue is not necessary in order to use the slots attribute. Regards Daniel On 03/22/10 13:53, Peter Tröger wrote:
Hi,
with the latest decision from OGF28, we now have the queueMaxSlotsAllowed attribute in the MonitoringSession interface.
Under this impression, I meanwhile wonder about the relation between minSlots / maxSlots and the (also newly added) queueName attribute in the job template. Can you specify minSlots / maxSlots without configuring a queueName to use ? And if yes, don't we need an according maxSlotsAllowed on DRM system level then ?
Best, Peter.
------------------------------------------------------------------------
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
Beyond being related to the parallel environments, slot availability is also governed by resource quota sets. Note that slots in SGE are both a queue-level and host-level concept. A single host might have two queues, each with 4 slots, but the host itself offers only 6. That means two queue slots will always be empty, but which queue has them depends on the order the jobs are assigned. Hence my previously stated reservations about including a queue concept. Daniel On 03/22/10 06:05, Daniel Gruber wrote:
In SGE minSlots and maxSlots are in tight relation to the parallel environment which is reflected in the new standard with an configuration name. This inter-attribute dependency does not look like a very clean approach, which I mentioned also in the meeting. Regarding SGE minSlots,maxSlots and the queue name have not a mandatory relationship which means that selecting a special queue is not necessary in order to use the slots attribute.
Regards
Daniel
On 03/22/10 13:53, Peter Tröger wrote:
Hi,
with the latest decision from OGF28, we now have the queueMaxSlotsAllowed attribute in the MonitoringSession interface.
Under this impression, I meanwhile wonder about the relation between minSlots / maxSlots and the (also newly added) queueName attribute in the job template. Can you specify minSlots / maxSlots without configuring a queueName to use ? And if yes, don't we need an according maxSlotsAllowed on DRM system level then ?
Best, Peter.
------------------------------------------------------------------------
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
As I said in the email I just wrote, I'm willing to be convinced of the value of adding queues to the job submission side of things. I am, however, fundamentally opposed to adding queues to the monitoring side. The various concepts of queues are too different for that to make any sense. There is absolutely no way we will be able to model both LSF and SGE queues in a way that is abstract enough to be consistent and still specific enough to be meaningful and accurate. We'll talk on the next call. :) Daniel On 03/22/10 05:53, Peter Tröger wrote:
Hi,
with the latest decision from OGF28, we now have the queueMaxSlotsAllowed attribute in the MonitoringSession interface.
Under this impression, I meanwhile wonder about the relation between minSlots / maxSlots and the (also newly added) queueName attribute in the job template. Can you specify minSlots / maxSlots without configuring a queueName to use ? And if yes, don't we need an according maxSlotsAllowed on DRM system level then ?
Best, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
As I said in the email I just wrote, I'm willing to be convinced of the value of adding queues to the job submission side of things. I am, however, fundamentally opposed to adding queues to the monitoring side.
I will heavily insist on queue support in DRMAAv2, This is a long demanded feature, which also popped up again in the survey.
The various concepts of queues are too different for that to make any sense. There is absolutely no way we will be able to model both LSF and SGE queues in a way that is abstract enough to be consistent and still specific enough to be meaningful and accurate. We'll talk on the next call. :)
The intention of the current model is that JobTemplate::queueName and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA would promise that all strings that show up in MonitoringSession:: drmsQueueNames are valid input for JobTemplate::queueName. Nothing more. The use case are DRMAA-based portals and command-line applications. The interpretation of what a queue is can be provided by the library implementation - at the end, the user anyway has to reason about the meaning of queue names. We could relax the conditions so that other values are also allowed in JobTemplate::queueName. This would allow MonitoringSession:: drmsQueueNames to return nothing in SGE. This must be anyway possible - Condor has no queue concept at all. I could also agree to remove MonitoringSession:: queueMaxWallclockTime and MonitoringSession:: queueMaxSlotsAllowed, since these two attributes are the ones that demand a particular understanding of what a queue is. Best, Peter.
As long as the MonitoringSession::drmsQueueNames is nothing more than an opaque set of strings that are the valid values for JobTemplate::queueName, I can live with it. I can see where that would be useful for a portal. I thought, however, that we had come to the conclusion previously that portals and user interfaces were not really our target applications. (Anyone remember what feature spawned that conclusion?) I thought that DRMAA was specifically focused on applications integrating with clusters. If so, a list of opaque strings is useless. By the way, you'll also have to give a little thought to reconciling the 1:1 queue/host model with the 1:n and n:m models, as far as identifying them in a list goes. Daniel On 3/23/10 10:27 AM, Peter Tröger wrote:
As I said in the email I just wrote, I'm willing to be convinced of the value of adding queues to the job submission side of things. I am, however, fundamentally opposed to adding queues to the monitoring side.
I will heavily insist on queue support in DRMAAv2, This is a long demanded feature, which also popped up again in the survey.
The various concepts of queues are too different for that to make any sense. There is absolutely no way we will be able to model both LSF and SGE queues in a way that is abstract enough to be consistent and still specific enough to be meaningful and accurate. We'll talk on the next call. :)
The intention of the current model is that JobTemplate::queueName and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA would promise that all strings that show up in MonitoringSession:: drmsQueueNames are valid input for JobTemplate::queueName. Nothing more. The use case are DRMAA-based portals and command-line applications. The interpretation of what a queue is can be provided by the library implementation - at the end, the user anyway has to reason about the meaning of queue names.
We could relax the conditions so that other values are also allowed in JobTemplate::queueName. This would allow MonitoringSession:: drmsQueueNames to return nothing in SGE. This must be anyway possible - Condor has no queue concept at all.
I could also agree to remove MonitoringSession:: queueMaxWallclockTime and MonitoringSession:: queueMaxSlotsAllowed, since these two attributes are the ones that demand a particular understanding of what a queue is.
Best, Peter.
Hi,
As long as the MonitoringSession::drmsQueueNames is nothing more than an opaque set of strings that are the valid values for JobTemplate::queueName, I can live with it. I can see where that would be useful for a portal. I thought, however, that we had come to the conclusion previously that portals and user interfaces were not really our target applications. (Anyone remember what feature spawned that conclusion?) I thought that DRMAA was specifically focused on applications integrating with clusters. If so, a list of opaque strings is useless.
We dropped the portal example, that's true. The most convincing DRMAA applications at the moment are high-level APIs and meta-schedulers on top of / with DRMAA support. I did some field study to get the picture right. LSF, PBS, SGE, LoadLeveler, SAGA, Globus and GridWay can submit jobs to particular queues. In LoadLeveler, queues are called "classes". Condor, JSDL and OGSA-BES seem to have no queue concept - correct me if I am wrong. The retrieval of the list of queue names is only supported in: LSF: bqueues (http://www.vub.ac.be/BFUCC/LSF/man/bqueues.1.html) PBS: qstat -q (http://linux.die.net/man/1/qstat) SGE: qstat -f LoadLeveler: llclass -l (http://www.ccs.ornl.gov/Cheetah/ LL.html#Classes) So if we add the monitoring facility, an empty return value must be still valid.
By the way, you'll also have to give a little thought to reconciling the 1:1 queue/host model with the 1:n and n:m models, as far as identifying them in a list goes.
This is the true counter argument. If DRMAA monitoring gives no additional hints here, invalid combinations of valid machine / queue names in the job template could occur. Let's wait if any defender of queue list monitoring stands up. Otherwise, I propose to keep only the queue name attribute in the job template. /Peter.
Daniel
On 3/23/10 10:27 AM, Peter Tröger wrote:
As I said in the email I just wrote, I'm willing to be convinced of the value of adding queues to the job submission side of things. I am, however, fundamentally opposed to adding queues to the monitoring side.
I will heavily insist on queue support in DRMAAv2, This is a long demanded feature, which also popped up again in the survey.
The various concepts of queues are too different for that to make any sense. There is absolutely no way we will be able to model both LSF and SGE queues in a way that is abstract enough to be consistent and still specific enough to be meaningful and accurate. We'll talk on the next call. :)
The intention of the current model is that JobTemplate::queueName and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA would promise that all strings that show up in MonitoringSession:: drmsQueueNames are valid input for JobTemplate::queueName. Nothing more. The use case are DRMAA-based portals and command-line applications. The interpretation of what a queue is can be provided by the library implementation - at the end, the user anyway has to reason about the meaning of queue names.
We could relax the conditions so that other values are also allowed in JobTemplate::queueName. This would allow MonitoringSession:: drmsQueueNames to return nothing in SGE. This must be anyway possible - Condor has no queue concept at all.
I could also agree to remove MonitoringSession:: queueMaxWallclockTime and MonitoringSession:: queueMaxSlotsAllowed, since these two attributes are the ones that demand a particular understanding of what a queue is.
Best, Peter.
Hi, Quoting [Peter Tr?ger] (Mar 23 2010):
Hi,
As long as the MonitoringSession::drmsQueueNames is nothing more than an opaque set of strings that are the valid values for JobTemplate::queueName, I can live with it. I can see where that would be useful for a portal. I thought, however, that we had come to the conclusion previously that portals and user interfaces were not really our target applications. (Anyone remember what feature spawned that conclusion?) I thought that DRMAA was specifically focused on applications integrating with clusters. If so, a list of opaque strings is useless.
We dropped the portal example, that's true. The most convincing DRMAA applications at the moment are high-level APIs and meta-schedulers on top of / with DRMAA support.
I did some field study to get the picture right. LSF, PBS, SGE, LoadLeveler, SAGA, Globus and GridWay can submit jobs to particular queues. In LoadLeveler, queues are called "classes". Condor, JSDL and OGSA-BES seem to have no queue concept - correct me if I am wrong. The retrieval of the list of queue names is only supported in:
LSF: bqueues (http://www.vub.ac.be/BFUCC/LSF/man/bqueues.1.html) PBS: qstat -q (http://linux.die.net/man/1/qstat) SGE: qstat -f LoadLeveler: llclass -l (http://www.ccs.ornl.gov/Cheetah/ LL.html#Classes)
So if we add the monitoring facility, an empty return value must be still valid.
By the way, you'll also have to give a little thought to reconciling the 1:1 queue/host model with the 1:n and n:m models, as far as identifying them in a list goes.
This is the true counter argument. If DRMAA monitoring gives no additional hints here, invalid combinations of valid machine / queue names in the job template could occur. Let's wait if any defender of queue list monitoring stands up. Otherwise, I propose to keep only the queue name attribute in the job template.
It's not a hard requirement from our side, but I think its utterly useful. In general, our end users are complaining more often than not if they need to manually retrieve resource details like service contact URLs or queue names from some obscure web page, instead of being able to retrieve those information programatically. The manual way is simply too error prone, tedious and static. Cheers, Andre.
/Peter.
Daniel
On 3/23/10 10:27 AM, Peter Tröger wrote:
As I said in the email I just wrote, I'm willing to be convinced of the value of adding queues to the job submission side of things. I am, however, fundamentally opposed to adding queues to the monitoring side.
I will heavily insist on queue support in DRMAAv2, This is a long demanded feature, which also popped up again in the survey.
The various concepts of queues are too different for that to make any sense. There is absolutely no way we will be able to model both LSF and SGE queues in a way that is abstract enough to be consistent and still specific enough to be meaningful and accurate. We'll talk on the next call. :)
The intention of the current model is that JobTemplate::queueName and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA would promise that all strings that show up in MonitoringSession:: drmsQueueNames are valid input for JobTemplate::queueName. Nothing more. The use case are DRMAA-based portals and command-line applications. The interpretation of what a queue is can be provided by the library implementation - at the end, the user anyway has to reason about the meaning of queue names.
We could relax the conditions so that other values are also allowed in JobTemplate::queueName. This would allow MonitoringSession:: drmsQueueNames to return nothing in SGE. This must be anyway possible - Condor has no queue concept at all.
I could also agree to remove MonitoringSession:: queueMaxWallclockTime and MonitoringSession:: queueMaxSlotsAllowed, since these two attributes are the ones that demand a particular understanding of what a queue is.
Best, Peter.
-- Nothing is ever easy.
What does that mean when you say "retrieve them programatically?" What tool are they using to do it? What is the interface? Daniel On 03/24/10 06:03, Andre Merzky wrote:
Hi,
Quoting [Peter Tr?ger] (Mar 23 2010):
Hi,
As long as the MonitoringSession::drmsQueueNames is nothing more than an opaque set of strings that are the valid values for JobTemplate::queueName, I can live with it. I can see where that would be useful for a portal. I thought, however, that we had come to the conclusion previously that portals and user interfaces were not really our target applications. (Anyone remember what feature spawned that conclusion?) I thought that DRMAA was specifically focused on applications integrating with clusters. If so, a list of opaque strings is useless.
We dropped the portal example, that's true. The most convincing DRMAA applications at the moment are high-level APIs and meta-schedulers on top of / with DRMAA support.
I did some field study to get the picture right. LSF, PBS, SGE, LoadLeveler, SAGA, Globus and GridWay can submit jobs to particular queues. In LoadLeveler, queues are called "classes". Condor, JSDL and OGSA-BES seem to have no queue concept - correct me if I am wrong. The retrieval of the list of queue names is only supported in:
LSF: bqueues (http://www.vub.ac.be/BFUCC/LSF/man/bqueues.1.html) PBS: qstat -q (http://linux.die.net/man/1/qstat) SGE: qstat -f LoadLeveler: llclass -l (http://www.ccs.ornl.gov/Cheetah/ LL.html#Classes)
So if we add the monitoring facility, an empty return value must be still valid.
By the way, you'll also have to give a little thought to reconciling the 1:1 queue/host model with the 1:n and n:m models, as far as identifying them in a list goes.
This is the true counter argument. If DRMAA monitoring gives no additional hints here, invalid combinations of valid machine / queue names in the job template could occur. Let's wait if any defender of queue list monitoring stands up. Otherwise, I propose to keep only the queue name attribute in the job template.
It's not a hard requirement from our side, but I think its utterly useful. In general, our end users are complaining more often than not if they need to manually retrieve resource details like service contact URLs or queue names from some obscure web page, instead of being able to retrieve those information programatically. The manual way is simply too error prone, tedious and static.
Cheers, Andre.
/Peter.
Daniel
On 3/23/10 10:27 AM, Peter Tröger wrote:
As I said in the email I just wrote, I'm willing to be convinced of the value of adding queues to the job submission side of things. I am, however, fundamentally opposed to adding queues to the monitoring side.
I will heavily insist on queue support in DRMAAv2, This is a long demanded feature, which also popped up again in the survey.
The various concepts of queues are too different for that to make any sense. There is absolutely no way we will be able to model both LSF and SGE queues in a way that is abstract enough to be consistent and still specific enough to be meaningful and accurate. We'll talk on the next call. :)
The intention of the current model is that JobTemplate::queueName and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA would promise that all strings that show up in MonitoringSession:: drmsQueueNames are valid input for JobTemplate::queueName. Nothing more. The use case are DRMAA-based portals and command-line applications. The interpretation of what a queue is can be provided by the library implementation - at the end, the user anyway has to reason about the meaning of queue names.
We could relax the conditions so that other values are also allowed in JobTemplate::queueName. This would allow MonitoringSession:: drmsQueueNames to return nothing in SGE. This must be anyway possible - Condor has no queue concept at all.
I could also agree to remove MonitoringSession:: queueMaxWallclockTime and MonitoringSession:: queueMaxSlotsAllowed, since these two attributes are the ones that demand a particular understanding of what a queue is.
Best, Peter.
Quoting [Daniel Templeton] (Mar 24 2010):
What does that mean when you say "retrieve them programatically?" What tool are they using to do it? What is the interface?
I mean retrieving via an API like DRMAA or SAGA, instead of retrieving by some other means outside of the program (reading a web page and entering the queue name in the code or some config file). Cheers, Andre.
Daniel
On 03/24/10 06:03, Andre Merzky wrote:
Hi,
Quoting [Peter Tr?ger] (Mar 23 2010):
Hi,
As long as the MonitoringSession::drmsQueueNames is nothing more than an opaque set of strings that are the valid values for JobTemplate::queueName, I can live with it. I can see where that would be useful for a portal. I thought, however, that we had come to the conclusion previously that portals and user interfaces were not really our target applications. (Anyone remember what feature spawned that conclusion?) I thought that DRMAA was specifically focused on applications integrating with clusters. If so, a list of opaque strings is useless.
We dropped the portal example, that's true. The most convincing DRMAA applications at the moment are high-level APIs and meta-schedulers on top of / with DRMAA support.
I did some field study to get the picture right. LSF, PBS, SGE, LoadLeveler, SAGA, Globus and GridWay can submit jobs to particular queues. In LoadLeveler, queues are called "classes". Condor, JSDL and OGSA-BES seem to have no queue concept - correct me if I am wrong. The retrieval of the list of queue names is only supported in:
LSF: bqueues (http://www.vub.ac.be/BFUCC/LSF/man/bqueues.1.html) PBS: qstat -q (http://linux.die.net/man/1/qstat) SGE: qstat -f LoadLeveler: llclass -l (http://www.ccs.ornl.gov/Cheetah/ LL.html#Classes)
So if we add the monitoring facility, an empty return value must be still valid.
By the way, you'll also have to give a little thought to reconciling the 1:1 queue/host model with the 1:n and n:m models, as far as identifying them in a list goes.
This is the true counter argument. If DRMAA monitoring gives no additional hints here, invalid combinations of valid machine / queue names in the job template could occur. Let's wait if any defender of queue list monitoring stands up. Otherwise, I propose to keep only the queue name attribute in the job template.
It's not a hard requirement from our side, but I think its utterly useful. In general, our end users are complaining more often than not if they need to manually retrieve resource details like service contact URLs or queue names from some obscure web page, instead of being able to retrieve those information programatically. The manual way is simply too error prone, tedious and static.
Cheers, Andre.
/Peter.
Daniel
On 3/23/10 10:27 AM, Peter Tröger wrote:
As I said in the email I just wrote, I'm willing to be convinced of the value of adding queues to the job submission side of things. I am, however, fundamentally opposed to adding queues to the monitoring side.
I will heavily insist on queue support in DRMAAv2, This is a long demanded feature, which also popped up again in the survey.
The various concepts of queues are too different for that to make any sense. There is absolutely no way we will be able to model both LSF and SGE queues in a way that is abstract enough to be consistent and still specific enough to be meaningful and accurate. We'll talk on the next call. :)
The intention of the current model is that JobTemplate::queueName and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA would promise that all strings that show up in MonitoringSession:: drmsQueueNames are valid input for JobTemplate::queueName. Nothing more. The use case are DRMAA-based portals and command-line applications. The interpretation of what a queue is can be provided by the library implementation - at the end, the user anyway has to reason about the meaning of queue names.
We could relax the conditions so that other values are also allowed in JobTemplate::queueName. This would allow MonitoringSession:: drmsQueueNames to return nothing in SGE. This must be anyway possible - Condor has no queue concept at all.
I could also agree to remove MonitoringSession:: queueMaxWallclockTime and MonitoringSession:: queueMaxSlotsAllowed, since these two attributes are the ones that demand a particular understanding of what a queue is.
Best, Peter.
-- Nothing is ever easy.
So the program is going to go read the queue names via the API and then present them to the user to pick from? Doesn't the user still need to go reference a web page somewhere to figure out which queue is the right one for what he's doing? Daniel On 03/24/10 10:59, Andre Merzky wrote:
Quoting [Daniel Templeton] (Mar 24 2010):
What does that mean when you say "retrieve them programatically?" What tool are they using to do it? What is the interface?
I mean retrieving via an API like DRMAA or SAGA, instead of retrieving by some other means outside of the program (reading a web page and entering the queue name in the code or some config file).
Cheers, Andre.
Daniel
On 03/24/10 06:03, Andre Merzky wrote:
Hi,
Quoting [Peter Tr?ger] (Mar 23 2010):
Hi,
As long as the MonitoringSession::drmsQueueNames is nothing more than an opaque set of strings that are the valid values for JobTemplate::queueName, I can live with it. I can see where that would be useful for a portal. I thought, however, that we had come to the conclusion previously that portals and user interfaces were not really our target applications. (Anyone remember what feature spawned that conclusion?) I thought that DRMAA was specifically focused on applications integrating with clusters. If so, a list of opaque strings is useless.
We dropped the portal example, that's true. The most convincing DRMAA applications at the moment are high-level APIs and meta-schedulers on top of / with DRMAA support.
I did some field study to get the picture right. LSF, PBS, SGE, LoadLeveler, SAGA, Globus and GridWay can submit jobs to particular queues. In LoadLeveler, queues are called "classes". Condor, JSDL and OGSA-BES seem to have no queue concept - correct me if I am wrong. The retrieval of the list of queue names is only supported in:
LSF: bqueues (http://www.vub.ac.be/BFUCC/LSF/man/bqueues.1.html) PBS: qstat -q (http://linux.die.net/man/1/qstat) SGE: qstat -f LoadLeveler: llclass -l (http://www.ccs.ornl.gov/Cheetah/ LL.html#Classes)
So if we add the monitoring facility, an empty return value must be still valid.
By the way, you'll also have to give a little thought to reconciling the 1:1 queue/host model with the 1:n and n:m models, as far as identifying them in a list goes.
This is the true counter argument. If DRMAA monitoring gives no additional hints here, invalid combinations of valid machine / queue names in the job template could occur. Let's wait if any defender of queue list monitoring stands up. Otherwise, I propose to keep only the queue name attribute in the job template.
It's not a hard requirement from our side, but I think its utterly useful. In general, our end users are complaining more often than not if they need to manually retrieve resource details like service contact URLs or queue names from some obscure web page, instead of being able to retrieve those information programatically. The manual way is simply too error prone, tedious and static.
Cheers, Andre.
/Peter.
Daniel
On 3/23/10 10:27 AM, Peter Tröger wrote:
> As I said in the email I just wrote, I'm willing to be convinced > of the > value of adding queues to the job submission side of things. I am, > however, fundamentally opposed to adding queues to the monitoring > side.
I will heavily insist on queue support in DRMAAv2, This is a long demanded feature, which also popped up again in the survey.
> The various concepts of queues are too different for that to make > any > sense. There is absolutely no way we will be able to model both > LSF and > SGE queues in a way that is abstract enough to be consistent and > still > specific enough to be meaningful and accurate. We'll talk on the > next > call. :)
The intention of the current model is that JobTemplate::queueName and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA would promise that all strings that show up in MonitoringSession:: drmsQueueNames are valid input for JobTemplate::queueName. Nothing more. The use case are DRMAA-based portals and command-line applications. The interpretation of what a queue is can be provided by the library implementation - at the end, the user anyway has to reason about the meaning of queue names.
We could relax the conditions so that other values are also allowed in JobTemplate::queueName. This would allow MonitoringSession:: drmsQueueNames to return nothing in SGE. This must be anyway possible - Condor has no queue concept at all.
I could also agree to remove MonitoringSession:: queueMaxWallclockTime and MonitoringSession:: queueMaxSlotsAllowed, since these two attributes are the ones that demand a particular understanding of what a queue is.
Best, Peter.
Quoting [Daniel Templeton] (Mar 24 2010):
So the program is going to go read the queue names via the API and then present them to the user to pick from? Doesn't the user still need to go reference a web page somewhere to figure out which queue is the right one for what he's doing?
Yes, the user need to have some idea what the queue names mean. Queue names like 'bigjob', 'preempt', 'checkpt', 'workq' are, however, rather intuitive. Thats of course not always the case. BTW: the above names are from our current systems on loni.org. Anyway, as said: that is not a hard requirement, but our users seem to find this useful. Cheers, Andre.
Daniel
On 03/24/10 10:59, Andre Merzky wrote:
Quoting [Daniel Templeton] (Mar 24 2010):
What does that mean when you say "retrieve them programatically?" What tool are they using to do it? What is the interface?
I mean retrieving via an API like DRMAA or SAGA, instead of retrieving by some other means outside of the program (reading a web page and entering the queue name in the code or some config file).
Cheers, Andre.
Daniel
On 03/24/10 06:03, Andre Merzky wrote:
Hi,
Quoting [Peter Tr?ger] (Mar 23 2010):
Hi,
As long as the MonitoringSession::drmsQueueNames is nothing more than an opaque set of strings that are the valid values for JobTemplate::queueName, I can live with it. I can see where that would be useful for a portal. I thought, however, that we had come to the conclusion previously that portals and user interfaces were not really our target applications. (Anyone remember what feature spawned that conclusion?) I thought that DRMAA was specifically focused on applications integrating with clusters. If so, a list of opaque strings is useless.
We dropped the portal example, that's true. The most convincing DRMAA applications at the moment are high-level APIs and meta-schedulers on top of / with DRMAA support.
I did some field study to get the picture right. LSF, PBS, SGE, LoadLeveler, SAGA, Globus and GridWay can submit jobs to particular queues. In LoadLeveler, queues are called "classes". Condor, JSDL and OGSA-BES seem to have no queue concept - correct me if I am wrong. The retrieval of the list of queue names is only supported in:
LSF: bqueues (http://www.vub.ac.be/BFUCC/LSF/man/bqueues.1.html) PBS: qstat -q (http://linux.die.net/man/1/qstat) SGE: qstat -f LoadLeveler: llclass -l (http://www.ccs.ornl.gov/Cheetah/ LL.html#Classes)
So if we add the monitoring facility, an empty return value must be still valid.
By the way, you'll also have to give a little thought to reconciling the 1:1 queue/host model with the 1:n and n:m models, as far as identifying them in a list goes.
This is the true counter argument. If DRMAA monitoring gives no additional hints here, invalid combinations of valid machine / queue names in the job template could occur. Let's wait if any defender of queue list monitoring stands up. Otherwise, I propose to keep only the queue name attribute in the job template.
It's not a hard requirement from our side, but I think its utterly useful. In general, our end users are complaining more often than not if they need to manually retrieve resource details like service contact URLs or queue names from some obscure web page, instead of being able to retrieve those information programatically. The manual way is simply too error prone, tedious and static.
Cheers, Andre.
/Peter.
Daniel
On 3/23/10 10:27 AM, Peter Tröger wrote: >>As I said in the email I just wrote, I'm willing to be convinced >>of the >>value of adding queues to the job submission side of things. I am, >>however, fundamentally opposed to adding queues to the monitoring >>side. > >I will heavily insist on queue support in DRMAAv2, This is a long >demanded feature, which also popped up again in the survey. > >>The various concepts of queues are too different for that to make >>any >>sense. There is absolutely no way we will be able to model both >>LSF and >>SGE queues in a way that is abstract enough to be consistent and >>still >>specific enough to be meaningful and accurate. We'll talk on the >>next >>call. :) > >The intention of the current model is that JobTemplate::queueName >and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA >would promise that all strings that show up in MonitoringSession:: >drmsQueueNames are valid input for JobTemplate::queueName. Nothing >more. >The use case are DRMAA-based portals and command-line applications. >The interpretation of what a queue is can be provided by the >library implementation - at the end, the user anyway has to reason >about the meaning of queue names. > >We could relax the conditions so that other values are also allowed >in JobTemplate::queueName. This would allow MonitoringSession:: >drmsQueueNames to return nothing in SGE. This must be anyway >possible - Condor has no queue concept at all. > >I could also agree to remove MonitoringSession:: >queueMaxWallclockTime and MonitoringSession:: >queueMaxSlotsAllowed, since these two attributes are the ones that >demand a particular understanding of what a queue is. > >Best, >Peter.
-- Nothing is ever easy.
Hi, This is also usefull for blind users, like servers of applications on top of a DRM. The queue may not be the best one, but at least a submission may occur. Cheers. .Yves. Le Wednesday 24 March 2010 19:30:25 Andre Merzky, vous avez écrit :
Quoting [Daniel Templeton] (Mar 24 2010):
So the program is going to go read the queue names via the API and then present them to the user to pick from? Doesn't the user still need to go reference a web page somewhere to figure out which queue is the right one for what he's doing?
Yes, the user need to have some idea what the queue names mean. Queue names like 'bigjob', 'preempt', 'checkpt', 'workq' are, however, rather intuitive. Thats of course not always the case. BTW: the above names are from our current systems on loni.org.
Anyway, as said: that is not a hard requirement, but our users seem to find this useful.
Cheers, Andre.
Daniel
On 03/24/10 10:59, Andre Merzky wrote:
Quoting [Daniel Templeton] (Mar 24 2010):
What does that mean when you say "retrieve them programatically?" What tool are they using to do it? What is the interface?
I mean retrieving via an API like DRMAA or SAGA, instead of retrieving by some other means outside of the program (reading a web page and entering the queue name in the code or some config file).
Cheers, Andre.
Daniel
On 03/24/10 06:03, Andre Merzky wrote:
Hi,
Quoting [Peter Tr?ger] (Mar 23 2010):
Hi,
>As long as the MonitoringSession::drmsQueueNames is nothing more >than an opaque set of strings that are the valid values for >JobTemplate::queueName, I can live with it. I can see where that >would be useful for a portal. I thought, however, that we had come >to the conclusion previously that portals and user interfaces were >not really our target applications. (Anyone remember what feature >spawned that conclusion?) I thought that DRMAA was specifically >focused on applications integrating with clusters. If so, a list of >opaque strings is useless.
We dropped the portal example, that's true. The most convincing DRMAA applications at the moment are high-level APIs and meta-schedulers on top of / with DRMAA support.
I did some field study to get the picture right. LSF, PBS, SGE, LoadLeveler, SAGA, Globus and GridWay can submit jobs to particular queues. In LoadLeveler, queues are called "classes". Condor, JSDL and OGSA-BES seem to have no queue concept - correct me if I am wrong. The retrieval of the list of queue names is only supported in:
LSF: bqueues (http://www.vub.ac.be/BFUCC/LSF/man/bqueues.1.html) PBS: qstat -q (http://linux.die.net/man/1/qstat) SGE: qstat -f LoadLeveler: llclass -l (http://www.ccs.ornl.gov/Cheetah/ LL.html#Classes)
So if we add the monitoring facility, an empty return value must be still valid.
>By the way, you'll also have to give a little thought to reconciling >the 1:1 queue/host model with the 1:n and n:m models, as far as >identifying them in a list goes.
This is the true counter argument. If DRMAA monitoring gives no additional hints here, invalid combinations of valid machine / queue names in the job template could occur. Let's wait if any defender of queue list monitoring stands up. Otherwise, I propose to keep only the queue name attribute in the job template.
It's not a hard requirement from our side, but I think its utterly useful. In general, our end users are complaining more often than not if they need to manually retrieve resource details like service contact URLs or queue names from some obscure web page, instead of being able to retrieve those information programatically. The manual way is simply too error prone, tedious and static.
Cheers, Andre.
/Peter.
>Daniel > >On 3/23/10 10:27 AM, Peter Tröger wrote: >>>As I said in the email I just wrote, I'm willing to be convinced >>>of the >>>value of adding queues to the job submission side of things. I >>> am, however, fundamentally opposed to adding queues to the >>> monitoring side. >> >>I will heavily insist on queue support in DRMAAv2, This is a long >>demanded feature, which also popped up again in the survey. >> >>>The various concepts of queues are too different for that to make >>>any >>>sense. There is absolutely no way we will be able to model both >>>LSF and >>>SGE queues in a way that is abstract enough to be consistent and >>>still >>>specific enough to be meaningful and accurate. We'll talk on the >>>next >>>call. :) >> >>The intention of the current model is that JobTemplate::queueName >>and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA >>would promise that all strings that show up in MonitoringSession:: >>drmsQueueNames are valid input for JobTemplate::queueName. Nothing >>more. >>The use case are DRMAA-based portals and command-line applications. >>The interpretation of what a queue is can be provided by the >>library implementation - at the end, the user anyway has to reason >>about the meaning of queue names. >> >>We could relax the conditions so that other values are also allowed >>in JobTemplate::queueName. This would allow MonitoringSession:: >>drmsQueueNames to return nothing in SGE. This must be anyway >>possible - Condor has no queue concept at all. >> >>I could also agree to remove MonitoringSession:: >>queueMaxWallclockTime and MonitoringSession:: >>queueMaxSlotsAllowed, since these two attributes are the ones that >>demand a particular understanding of what a queue is. >> >>Best, >>Peter.
-- Yves Caniou Associate Professor at Université Lyon 1, Member of the team project INRIA GRAAL in the LIP ENS-Lyon, Délégation CNRS in Japan French Laboratory of Informatics (JFLI), * in Information Technology Center, The University of Tokyo, 2-11-16 Yayoi, Bunkyo-ku, Tokyo 113-8658, Japan tel: +81-3-5841-0540 * in National Institute of Informatics 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan tel: +81-3-4212-2412 http://graal.ens-lyon.fr/~ycaniou/
I asked about this issue in some personal talks at OGF and D-Grid, and people agreed that a queue name listing is something they really want to see. Dan, please note that the description of both attributes (listing and JT part) is already very relaxed. We only promise that if MonitoringSession::drmsQueueNames returns something, it must be an accepted input in JobTemplate::queueName. All other combinations (no list / custom queue name set; list available / custom queue name set; ...) are still allowed. We will finalize the discussion this week in the phone call. Best, Peter. Am 25.03.2010 um 03:47 schrieb Yves Caniou:
Hi,
This is also usefull for blind users, like servers of applications on top of a DRM. The queue may not be the best one, but at least a submission may occur.
Cheers.
.Yves.
Le Wednesday 24 March 2010 19:30:25 Andre Merzky, vous avez écrit :
Quoting [Daniel Templeton] (Mar 24 2010):
So the program is going to go read the queue names via the API and then present them to the user to pick from? Doesn't the user still need to go reference a web page somewhere to figure out which queue is the right one for what he's doing?
Yes, the user need to have some idea what the queue names mean. Queue names like 'bigjob', 'preempt', 'checkpt', 'workq' are, however, rather intuitive. Thats of course not always the case. BTW: the above names are from our current systems on loni.org.
Anyway, as said: that is not a hard requirement, but our users seem to find this useful.
Cheers, Andre.
Daniel
On 03/24/10 10:59, Andre Merzky wrote:
Quoting [Daniel Templeton] (Mar 24 2010):
What does that mean when you say "retrieve them programatically?" What tool are they using to do it? What is the interface?
I mean retrieving via an API like DRMAA or SAGA, instead of retrieving by some other means outside of the program (reading a web page and entering the queue name in the code or some config file).
Cheers, Andre.
Daniel
On 03/24/10 06:03, Andre Merzky wrote:
Hi,
Quoting [Peter Tr?ger] (Mar 23 2010): > Hi, > >> As long as the MonitoringSession::drmsQueueNames is nothing more >> than an opaque set of strings that are the valid values for >> JobTemplate::queueName, I can live with it. I can see where that >> would be useful for a portal. I thought, however, that we had come >> to the conclusion previously that portals and user interfaces were >> not really our target applications. (Anyone remember what feature >> spawned that conclusion?) I thought that DRMAA was specifically >> focused on applications integrating with clusters. If so, a list of >> opaque strings is useless. > > We dropped the portal example, that's true. The most convincing DRMAA > applications at the moment are high-level APIs and meta-schedulers on > top of / with DRMAA support. > > I did some field study to get the picture right. LSF, PBS, SGE, > LoadLeveler, SAGA, Globus and GridWay can submit jobs to particular > queues. In LoadLeveler, queues are called "classes". Condor, JSDL and > OGSA-BES seem to have no queue concept - correct me if I am wrong. > The retrieval of the list of queue names is only supported in: > > LSF: bqueues (http://www.vub.ac.be/BFUCC/LSF/man/bqueues.1.html) > PBS: qstat -q (http://linux.die.net/man/1/qstat) > SGE: qstat -f > LoadLeveler: llclass -l (http://www.ccs.ornl.gov/Cheetah/ > LL.html#Classes) > > So if we add the monitoring facility, an empty return value must be > still valid. > >> By the way, you'll also have to give a little thought to reconciling >> the 1:1 queue/host model with the 1:n and n:m models, as far as >> identifying them in a list goes. > > This is the true counter argument. If DRMAA monitoring gives no > additional hints here, invalid combinations of valid machine / queue > names in the job template could occur. > Let's wait if any defender of queue list monitoring stands up. > Otherwise, I propose to keep only the queue name attribute in the job > template.
It's not a hard requirement from our side, but I think its utterly useful. In general, our end users are complaining more often than not if they need to manually retrieve resource details like service contact URLs or queue names from some obscure web page, instead of being able to retrieve those information programatically. The manual way is simply too error prone, tedious and static.
Cheers, Andre.
> /Peter. > >> Daniel >> >> On 3/23/10 10:27 AM, Peter Tröger wrote: >>>> As I said in the email I just wrote, I'm willing to be convinced >>>> of the >>>> value of adding queues to the job submission side of things. I >>>> am, however, fundamentally opposed to adding queues to the >>>> monitoring side. >>> >>> I will heavily insist on queue support in DRMAAv2, This is a long >>> demanded feature, which also popped up again in the survey. >>> >>>> The various concepts of queues are too different for that to make >>>> any >>>> sense. There is absolutely no way we will be able to model both >>>> LSF and >>>> SGE queues in a way that is abstract enough to be consistent and >>>> still >>>> specific enough to be meaningful and accurate. We'll talk on the >>>> next >>>> call. :) >>> >>> The intention of the current model is that JobTemplate::queueName >>> and MonitoringSession:: drmsQueueNames act as counterparts. DRMAA >>> would promise that all strings that show up in MonitoringSession:: >>> drmsQueueNames are valid input for JobTemplate::queueName. Nothing >>> more. >>> The use case are DRMAA-based portals and command-line applications. >>> The interpretation of what a queue is can be provided by the >>> library implementation - at the end, the user anyway has to reason >>> about the meaning of queue names. >>> >>> We could relax the conditions so that other values are also allowed >>> in JobTemplate::queueName. This would allow MonitoringSession:: >>> drmsQueueNames to return nothing in SGE. This must be anyway >>> possible - Condor has no queue concept at all. >>> >>> I could also agree to remove MonitoringSession:: >>> queueMaxWallclockTime and MonitoringSession:: >>> queueMaxSlotsAllowed, since these two attributes are the ones that >>> demand a particular understanding of what a queue is. >>> >>> Best, >>> Peter.
-- Yves Caniou Associate Professor at Université Lyon 1, Member of the team project INRIA GRAAL in the LIP ENS-Lyon, Délégation CNRS in Japan French Laboratory of Informatics (JFLI), * in Information Technology Center, The University of Tokyo, 2-11-16 Yayoi, Bunkyo-ku, Tokyo 113-8658, Japan tel: +81-3-5841-0540 * in National Institute of Informatics 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan tel: +81-3-4212-2412 http://graal.ens-lyon.fr/~ycaniou/ -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
participants (5)
-
Andre Merzky -
Daniel Gruber -
Daniel Templeton -
Peter Tröger -
Yves Caniou