What information should JobInfo provide ?
Dear all, at OGF-42, we stumbled over the issue that „JobInfo“ does not contain the original job name being used in the template: http://redmine.ogf.org/issues/102 This leads to the situation that you cannot filter for it when doing monitoring. Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don’t need to fetch information that you handed in by yourself as the calling application. For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment. Another case is the „change job template after submission“ magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure. All in all, it seems like that „JobInfo“ needs all of attributes from „JobTemplate“ too, not only the job name. What do you think ? Best regards, Peter.
I think the job template information are easy to add, and make app writing easier (I don't need to keep the old submitted job template around...). Only drawback I see is increase in memory footprint when used to report on many jobs -- but when used as a filtering instruction, this should not matter. My $0,02, Andre. On Wed, Sep 10, 2014 at 7:48 PM, Peter Tröger <peter@troeger.eu> wrote:
Dear all,
at OGF-42, we stumbled over the issue that „JobInfo“ does not contain the original job name being used in the template:
http://redmine.ogf.org/issues/102
This leads to the situation that you cannot filter for it when doing monitoring.
Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don’t need to fetch information that you handed in by yourself as the calling application.
For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment.
Another case is the „change job template after submission“ magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure.
All in all, it seems like that „JobInfo“ needs all of attributes from „JobTemplate“ too, not only the job name.
What do you think ?
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
-- It was a sad and disappointing day when I discovered that my Universal Remote Control did not, in fact, control the Universe. (Not even remotely.)
I think can't contribute much in this discussion. In my case we store the job ID returned by the API, and store the information we supplied + this job ID in a local datastore. The filtering is local, using the data returned periodically from the cluster (a thread running in background updating the tool queue). This way users will be able to filter for jobs (by name, queue, priority, etc) but most of the data is already in the tool. The most important information retrieved from the server for us is the job status. So in my case I think that this issue is not a blocker, but depending on how users want to use the API this can be useful. Maybe it would be useful to have a 'slim' method that returns a lightweight structure, and another one that returns the complete info? My 0.02 cents too :) Bruno
________________________________ From: Andre Merzky <andre@merzky.net> To: Peter Tröger <peter@troeger.eu> Cc: drmaa-wg <drmaa-wg@ogf.org> Sent: Friday, September 12, 2014 5:18 PM Subject: Re: [DRMAA-WG] What information should JobInfo provide ?
I think the job template information are easy to add, and make app writing easier (I don't need to keep the old submitted job template around...). Only drawback I see is increase in memory footprint when used to report on many jobs -- but when used as a filtering instruction, this should not matter.
My $0,02, Andre.
On Wed, Sep 10, 2014 at 7:48 PM, Peter Tröger <peter@troeger.eu> wrote:
Dear all,
at OGF-42, we stumbled over the issue that „JobInfo“ does not contain the original job name being used in the template:
http://redmine.ogf.org/issues/102
This leads to the situation that you cannot filter for it when doing monitoring.
Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don’t need to fetch information that you handed in by yourself as the calling application.
For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment.
Another case is the „change job template after submission“ magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure.
All in all, it seems like that „JobInfo“ needs all of attributes from „JobTemplate“ too, not only the job name.
What do you think ?
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
-- It was a sad and disappointing day when I discovered that my Universal Remote Control did not, in fact, control the Universe. (Not even remotely.)
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
Maybe it would be useful to have a 'slim' method that returns a lightweight structure, and another one that returns the complete info?
If we want to control that (and I am not sure we do), then I would prefer a flag to the call instead of a separate call. </bikeshed> ;) Andre. On Sat, Sep 13, 2014 at 1:39 AM, Bruno P. Kinoshita <brunodepaulak@yahoo.com.br> wrote:
I think can't contribute much in this discussion. In my case we store the job ID returned by the API, and store the information we supplied + this job ID in a local datastore.
The filtering is local, using the data returned periodically from the cluster (a thread running in background updating the tool queue). This way users will be able to filter for jobs (by name, queue, priority, etc) but most of the data is already in the tool. The most important information retrieved from the server for us is the job status.
So in my case I think that this issue is not a blocker, but depending on how users want to use the API this can be useful. Maybe it would be useful to have a 'slim' method that returns a lightweight structure, and another one that returns the complete info?
My 0.02 cents too :) Bruno
________________________________ From: Andre Merzky <andre@merzky.net> To: Peter Tröger <peter@troeger.eu> Cc: drmaa-wg <drmaa-wg@ogf.org> Sent: Friday, September 12, 2014 5:18 PM Subject: Re: [DRMAA-WG] What information should JobInfo provide ?
I think the job template information are easy to add, and make app writing easier (I don't need to keep the old submitted job template around...). Only drawback I see is increase in memory footprint when used to report on many jobs -- but when used as a filtering instruction, this should not matter.
My $0,02, Andre.
On Wed, Sep 10, 2014 at 7:48 PM, Peter Tröger <peter@troeger.eu> wrote:
Dear all,
at OGF-42, we stumbled over the issue that „JobInfo“ does not contain the original job name being used in the template:
http://redmine.ogf.org/issues/102
This leads to the situation that you cannot filter for it when doing monitoring.
Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don’t need to fetch information that you handed in by yourself as the calling application.
For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment.
Another case is the „change job template after submission“ magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure.
All in all, it seems like that „JobInfo“ needs all of attributes from „JobTemplate“ too, not only the job name.
What do you think ?
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
-- It was a sad and disappointing day when I discovered that my Universal Remote Control did not, in fact, control the Universe. (Not even remotely.)
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
-- It was a sad and disappointing day when I discovered that my Universal Remote Control did not, in fact, control the Universe. (Not even remotely.)
I agree to Bruno and Andre that the advantages are worth it. Performance considerations should be done in the language bindings and / or the implementations, the root spec is the wrong place for that. I therefore vote for the simple addition all job templates attributes to the JobInfo structure. An interesting side aspect are implementation-specific job template attributes. I would prefer to leave it to the implementors if they want to support them also in the JobInfo structure. If Daniel agrees to all of this, especially from his implementation viewpoint in Grid Engine 8.2, then I would update the root spec and the C binding accordingly. Best regards, Peter. Am 12.09.2014 um 22:18 schrieb Andre Merzky <andre@merzky.net>:
I think the job template information are easy to add, and make app writing easier (I don't need to keep the old submitted job template around...). Only drawback I see is increase in memory footprint when used to report on many jobs -- but when used as a filtering instruction, this should not matter.
My $0,02, Andre.
On Wed, Sep 10, 2014 at 7:48 PM, Peter Tröger <peter@troeger.eu> wrote:
Dear all,
at OGF-42, we stumbled over the issue that „JobInfo“ does not contain the original job name being used in the template:
http://redmine.ogf.org/issues/102
This leads to the situation that you cannot filter for it when doing monitoring.
Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don’t need to fetch information that you handed in by yourself as the calling application.
For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment.
Another case is the „change job template after submission“ magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure.
All in all, it seems like that „JobInfo“ needs all of attributes from „JobTemplate“ too, not only the job name.
What do you think ?
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
-- It was a sad and disappointing day when I discovered that my Universal Remote Control did not, in fact, control the Universe. (Not even remotely.)
As you indicated, Peter, it is not as easy as it sounds to add all job template attributes to the job info object. As mentioned before I think in most systems you can change the submission attributes after submission (in Grid Engine we have qalter, JSV, job classes, sge_request files, the -@ parameter, path aliasing) therefore we should be conservative and not specify that the submission parameters in job info needs to match 100% the job template. Andre: You don’t need to keep the job template at this point in time anyhow. There is a get job template method specified on the job object itself. When it comes to the Univa Grid Engine implementation, I’m storing the job template in the library when a submission is done. When you leave the application and do a re-connect the behavior is unspecified (meaning there is no job template available anymore). While there is now a way in Univa Grid Engine 8.2 to get the information out of qstat (you can see the qsub request line in qstat and qacct) I would not rely on this to be available on all systems. Even when a 3rd party DRMAA2 implementation relay on parsing this out, this is highly error prone (as new parameters and syntax changes can come in, which makes an implementation unusable). Also, we need to consider that the job can be either submitted from the drmaa2 app (where we have the job template) and in a monitoring session the job can be from any source (where we don’t have the job template). Having here a bunch of attributes which finally are optional and not available makes the interface not attractive. The job info object is extensible hence each implementation can decided what to add and what not as the last way out to get something like a job name into job info. Just one example to make it more clearer: You can submit a job with an certain output path in job template but on the machine where it runs the output path could be translated in something different, probably because of a different mount point. In UGE we have something called path aliasing, which is done by the system transparently to the user. Now which output path would be correct? When the job template output path is right, then the question is if it is catchable by all systems in a general way with job status calls? When the real output path is right then the question is do we get that one for all systems? I don’t think we can specify which one is the right on DRMAA level hence we would need to make all of them optional or implementation specific. Hence the implementation specific approach seems more natural for me. Due to the fact that we have already a method for returning the job template, and in order to have small consistent set of attributes in job info, I would really not go the way to do such a major change. Why I added this *job name* request in the job info object is because the job name (rather then other attributes) is probably the most natural attribute of a job. In Univa Grid Engine you can do a lot of useful things with it: defining job dependencies (or multi-job dependencies), using is for a grouped qstat etc. Short: - I just want to add a job name in job info since it is one of the most basic job info attributes - I do not want to specify whether the job name matches the job name of the job template or not. I would only specify that this is the job name the system considers as the job name - and as a fall back to have it available in all implementations it could be also just the job id. - I do not want to add all job template attributes to the job info object since most of them we don’t need for filtering in real live and due to the fact that they would be optional anyhow. Hence implementation specific extensions is the way to go IMHO. - I can also live with the fact that job name is implementation specific but I really see the job name very basic and really worthy for job filtering (especially when you want to group jobs like in UGE case). Sorry the delayed answer Cheers Daniel Am 14.09.2014 um 20:53 schrieb Peter Tröger <peter@troeger.eu>:
I agree to Bruno and Andre that the advantages are worth it. Performance considerations should be done in the language bindings and / or the implementations, the root spec is the wrong place for that. I therefore vote for the simple addition all job templates attributes to the JobInfo structure.
An interesting side aspect are implementation-specific job template attributes. I would prefer to leave it to the implementors if they want to support them also in the JobInfo structure.
If Daniel agrees to all of this, especially from his implementation viewpoint in Grid Engine 8.2, then I would update the root spec and the C binding accordingly.
Best regards, Peter.
Am 12.09.2014 um 22:18 schrieb Andre Merzky <andre@merzky.net>:
I think the job template information are easy to add, and make app writing easier (I don't need to keep the old submitted job template around...). Only drawback I see is increase in memory footprint when used to report on many jobs -- but when used as a filtering instruction, this should not matter.
My $0,02, Andre.
On Wed, Sep 10, 2014 at 7:48 PM, Peter Tröger <peter@troeger.eu> wrote:
Dear all,
at OGF-42, we stumbled over the issue that „JobInfo“ does not contain the original job name being used in the template:
http://redmine.ogf.org/issues/102
This leads to the situation that you cannot filter for it when doing monitoring.
Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don’t need to fetch information that you handed in by yourself as the calling application.
For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment.
Another case is the „change job template after submission“ magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure.
All in all, it seems like that „JobInfo“ needs all of attributes from „JobTemplate“ too, not only the job name.
What do you think ?
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
-- It was a sad and disappointing day when I discovered that my Universal Remote Control did not, in fact, control the Universe. (Not even remotely.)
Daniel, all good points. Best, Andre. On Mon, Sep 15, 2014 at 8:46 AM, Daniel Gruber <dgruber@univa.com> wrote:
As you indicated, Peter, it is not as easy as it sounds to add all job template attributes to the job info object. As mentioned before I think in most systems you can change the submission attributes after submission (in Grid Engine we have qalter, JSV, job classes, sge_request files, the -@ parameter, path aliasing) therefore we should be conservative and not specify that the submission parameters in job info needs to match 100% the job template.
Andre: You don’t need to keep the job template at this point in time anyhow. There is a get job template method specified on the job object itself.
When it comes to the Univa Grid Engine implementation, I’m storing the job template in the library when a submission is done. When you leave the application and do a re-connect the behavior is unspecified (meaning there is no job template available anymore). While there is now a way in Univa Grid Engine 8.2 to get the information out of qstat (you can see the qsub request line in qstat and qacct) I would not rely on this to be available on all systems. Even when a 3rd party DRMAA2 implementation relay on parsing this out, this is highly error prone (as new parameters and syntax changes can come in, which makes an implementation unusable).
Also, we need to consider that the job can be either submitted from the drmaa2 app (where we have the job template) and in a monitoring session the job can be from any source (where we don’t have the job template). Having here a bunch of attributes which finally are optional and not available makes the interface not attractive. The job info object is extensible hence each implementation can decided what to add and what not as the last way out to get something like a job name into job info.
Just one example to make it more clearer:
You can submit a job with an certain output path in job template but on the machine where it runs the output path could be translated in something different, probably because of a different mount point. In UGE we have something called path aliasing, which is done by the system transparently to the user. Now which output path would be correct? When the job template output path is right, then the question is if it is catchable by all systems in a general way with job status calls? When the real output path is right then the question is do we get that one for all systems? I don’t think we can specify which one is the right on DRMAA level hence we would need to make all of them optional or implementation specific. Hence the implementation specific approach seems more natural for me.
Due to the fact that we have already a method for returning the job template, and in order to have small consistent set of attributes in job info, I would really not go the way to do such a major change.
Why I added this *job name* request in the job info object is because the job name (rather then other attributes) is probably the most natural attribute of a job. In Univa Grid Engine you can do a lot of useful things with it: defining job dependencies (or multi-job dependencies), using is for a grouped qstat etc.
Short: - I just want to add a job name in job info since it is one of the most basic job info attributes - I do not want to specify whether the job name matches the job name of the job template or not. I would only specify that this is the job name the system considers as the job name - and as a fall back to have it available in all implementations it could be also just the job id. - I do not want to add all job template attributes to the job info object since most of them we don’t need for filtering in real live and due to the fact that they would be optional anyhow. Hence implementation specific extensions is the way to go IMHO. - I can also live with the fact that job name is implementation specific but I really see the job name very basic and really worthy for job filtering (especially when you want to group jobs like in UGE case).
Sorry the delayed answer
Cheers
Daniel
Am 14.09.2014 um 20:53 schrieb Peter Tröger <peter@troeger.eu>:
I agree to Bruno and Andre that the advantages are worth it. Performance considerations should be done in the language bindings and / or the implementations, the root spec is the wrong place for that. I therefore vote for the simple addition all job templates attributes to the JobInfo structure.
An interesting side aspect are implementation-specific job template attributes. I would prefer to leave it to the implementors if they want to support them also in the JobInfo structure.
If Daniel agrees to all of this, especially from his implementation viewpoint in Grid Engine 8.2, then I would update the root spec and the C binding accordingly.
Best regards, Peter.
Am 12.09.2014 um 22:18 schrieb Andre Merzky <andre@merzky.net>:
I think the job template information are easy to add, and make app writing easier (I don't need to keep the old submitted job template around...). Only drawback I see is increase in memory footprint when used to report on many jobs -- but when used as a filtering instruction, this should not matter.
My $0,02, Andre.
On Wed, Sep 10, 2014 at 7:48 PM, Peter Tröger <peter@troeger.eu> wrote:
Dear all,
at OGF-42, we stumbled over the issue that „JobInfo“ does not contain the original job name being used in the template:
http://redmine.ogf.org/issues/102
This leads to the situation that you cannot filter for it when doing monitoring.
Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don’t need to fetch information that you handed in by yourself as the calling application.
For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment.
Another case is the „change job template after submission“ magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure.
All in all, it seems like that „JobInfo“ needs all of attributes from „JobTemplate“ too, not only the job name.
What do you think ?
Best regards, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org https://www.ogf.org/mailman/listinfo/drmaa-wg
-- It was a sad and disappointing day when I discovered that my Universal Remote Control did not, in fact, control the Universe. (Not even remotely.)
-- It was a sad and disappointing day when I discovered that my Universal Remote Control did not, in fact, control the Universe. (Not even remotely.)
As you indicated, Peter, it is not as easy as it sounds to add all job template attributes to the job info object. As mentioned before I think in most systems you can change the submission attributes after submission (in Grid Engine we have qalter, JSV, job classes, sge_request files, the -@ parameter, path aliasing) therefore we should be conservative and not specify that the submission parameters in job info needs to match 100% the job template.
This is not the case so far, so everything is fine from this perspective.
different, probably because of a different mount point. In UGE we have something called path aliasing, which is done by the system transparently to the user. Now which output path would be correct?
A similar argumentation would hold for Condor.
When the job template output path is right, then the question is if it is catchable by all systems in a general way with job status calls? When the real output path is right then the question is do we get that one for all systems? I don’t think we can specify which one is the right on DRMAA level hence we would need to make all of them optional or implementation specific. Hence the implementation specific approach seems more natural for me.
I agree to Andre, this sounds reasonable.
Due to the fact that we have already a method for returning the job template, and in order to have small consistent set of attributes in job info, I would really not go the way to do such a major change.
Why I added this *job name* request in the job info object is because the job name (rather then other attributes) is probably the most natural attribute of a job. In Univa Grid Engine you can do a lot of useful things with it: defining job dependencies (or multi-job dependencies), using is for a grouped qstat etc.
- I can also live with the fact that job name is implementation specific but I really see the job name very basic and really worthy for job filtering (especially when you want to group jobs like in UGE case).
Ok, since adding all job template attributes to JobInfo is overkill, I would be the best to have an objective decision criteria if a job template attribute should be in JobInfo or not. The above argumentation (‚useful for one product‘) is not good enough. Does anybody has a smart idea on this ? If not, then I would vote for keep the spec as it is, meaning that JobInfo::jobName support becomes an implementation-specific feature of UGE. For the moment. If we have more than one implementation, than it is time to identify commonly added JobInfo attributes and adjust the „final recommendation“ spec accordingly. Best regards, Peter.
participants (4)
-
Andre Merzky -
Bruno P. Kinoshita -
Daniel Gruber -
Peter Tröger