Treat job templates as value types ?
Dear all, we have an ongoing discussion about the possible removal of "createJobTemplate" and "deleteJobTemplate". The last proposal was to move this functions to the language bindings that need it. This is currently only C - all other languages have their own way of performing instantiation and termination explicitly. After some more thinking, I think I got the true underlying issue. So far, we are treating job templates as instances with state and behavior - objects, in most languages. The only reason for doing this (so far) is the ability to throw errors on attribute access, since we need that for DRMAA's understanding of optional job template attributes. However, from all other perspectives, job templates are just value types. If you got one, you fill it, and then you pass the thing as a whole. In a RPC scenario, you would also expect to send filled job templates as a whole, similar to a filled JSDL document. So if we change that, what would that mean: C: Generic getter / setter functions for job template attributes would go away. Instead, you would create / delete JobTemplate structs directly. runJob() would take a pointer to this struct. Explicit removal is no longer needed, since the stack is cleared automatically. C# : JobTemplate class would become JobTemplate struct. Java / Python: JobTemplate class remains JobTemplate class, since they have no struct concept. Creation and destruction can be managed with OO mechanisms. Another effect would be a change in the semantics of optional attributes. We already demand the syntactical inclusion of optional attribute names in the class definition. In C, the usage of a non-implemented optional attribute names gives a specialized error. With the change, we would have struct members that you are not allowed to fill in with some DRMAA libraries. But this would be detected only at submission time, since struct changes do not involve library code. And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at all then, regardless of the language. Best, Peter.
Hi Peter, all, 2010/3/22 Peter Tröger <peter@troeger.eu>:
Dear all,
we have an ongoing discussion about the possible removal of "createJobTemplate" and "deleteJobTemplate". The last proposal was to move this functions to the language bindings that need it. This is currently only C - all other languages have their own way of performing instantiation and termination explicitly.
After some more thinking, I think I got the true underlying issue. So far, we are treating job templates as instances with state and behavior - objects, in most languages. The only reason for doing this (so far) is the ability to throw errors on attribute access, since we need that for DRMAA's understanding of optional job template attributes. However, from all other perspectives, job templates are just value types. If you got one, you fill it, and then you pass the thing as a whole. In a RPC scenario, you would also expect to send filled job templates as a whole, similar to a filled JSDL document.
So if we change that, what would that mean:
C: Generic getter / setter functions for job template attributes would go away. Instead, you would create / delete JobTemplate structs directly. runJob() would take a pointer to this struct. Explicit removal is no longer needed, since the stack is cleared automatically.
C# : JobTemplate class would become JobTemplate struct.
Java / Python: JobTemplate class remains JobTemplate class, since they have no struct concept. Creation and destruction can be managed with OO mechanisms.
Another effect would be a change in the semantics of optional attributes. We already demand the syntactical inclusion of optional attribute names in the class definition. In C, the usage of a non-implemented optional attribute names gives a specialized error. With the change, we would have struct members that you are not allowed to fill in with some DRMAA libraries. But this would be detected only at submission time, since struct changes do not involve library code.
just only one concern: what if one of the DRM vendor wish to provide some JobTemplate attributes additional to those specified in DRMAA (as i remember SGE was an example)? I expect having different struct definition (different drmaa2.h ) leads to serious problems. I'm not saying no (actually having a self allocated struct in C is more convenient than using getter/setters - e.g. error handling), but this, for me, requires addressing the extension methodology on the IDL level.
And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at all then, regardless of the language.
Best, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
Cheers, -- Mariusz
And that may be the original reason why the SGE C binding uses that pseudo-OO structure. It is more or less a hash map, allowing arbitrary name-value pairs to be stored, facilitating vendor-defined job template attributes. If we can't find a way around that issue, it's a show-stopper. We have to allow for vendor-defined attributes. Daniel On 03/22/10 07:29, Mariusz Mamoński wrote:
Hi Peter, all,
2010/3/22 Peter Tröger<peter@troeger.eu>:
Dear all,
we have an ongoing discussion about the possible removal of "createJobTemplate" and "deleteJobTemplate". The last proposal was to move this functions to the language bindings that need it. This is currently only C - all other languages have their own way of performing instantiation and termination explicitly.
After some more thinking, I think I got the true underlying issue. So far, we are treating job templates as instances with state and behavior - objects, in most languages. The only reason for doing this (so far) is the ability to throw errors on attribute access, since we need that for DRMAA's understanding of optional job template attributes. However, from all other perspectives, job templates are just value types. If you got one, you fill it, and then you pass the thing as a whole. In a RPC scenario, you would also expect to send filled job templates as a whole, similar to a filled JSDL document.
So if we change that, what would that mean:
C: Generic getter / setter functions for job template attributes would go away. Instead, you would create / delete JobTemplate structs directly. runJob() would take a pointer to this struct. Explicit removal is no longer needed, since the stack is cleared automatically.
C# : JobTemplate class would become JobTemplate struct.
Java / Python: JobTemplate class remains JobTemplate class, since they have no struct concept. Creation and destruction can be managed with OO mechanisms.
Another effect would be a change in the semantics of optional attributes. We already demand the syntactical inclusion of optional attribute names in the class definition. In C, the usage of a non-implemented optional attribute names gives a specialized error. With the change, we would have struct members that you are not allowed to fill in with some DRMAA libraries. But this would be detected only at submission time, since struct changes do not involve library code.
just only one concern: what if one of the DRM vendor wish to provide some JobTemplate attributes additional to those specified in DRMAA (as i remember SGE was an example)? I expect having different struct definition (different drmaa2.h ) leads to serious problems. I'm not saying no (actually having a self allocated struct in C is more convenient than using getter/setters - e.g. error handling), but this, for me, requires addressing the extension methodology on the IDL level.
And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at all then, regardless of the language.
Best, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
Cheers,
Ok, it was a bad idea. Thanks for commenting. /Peter. Am 22.03.2010 um 16:14 schrieb Daniel Templeton:
And that may be the original reason why the SGE C binding uses that pseudo-OO structure. It is more or less a hash map, allowing arbitrary name-value pairs to be stored, facilitating vendor-defined job template attributes. If we can't find a way around that issue, it's a show-stopper. We have to allow for vendor-defined attributes.
Daniel
On 03/22/10 07:29, Mariusz Mamoński wrote:
Hi Peter, all,
2010/3/22 Peter Tröger<peter@troeger.eu>:
Dear all,
we have an ongoing discussion about the possible removal of "createJobTemplate" and "deleteJobTemplate". The last proposal was to move this functions to the language bindings that need it. This is currently only C - all other languages have their own way of performing instantiation and termination explicitly.
After some more thinking, I think I got the true underlying issue. So far, we are treating job templates as instances with state and behavior - objects, in most languages. The only reason for doing this (so far) is the ability to throw errors on attribute access, since we need that for DRMAA's understanding of optional job template attributes. However, from all other perspectives, job templates are just value types. If you got one, you fill it, and then you pass the thing as a whole. In a RPC scenario, you would also expect to send filled job templates as a whole, similar to a filled JSDL document.
So if we change that, what would that mean:
C: Generic getter / setter functions for job template attributes would go away. Instead, you would create / delete JobTemplate structs directly. runJob() would take a pointer to this struct. Explicit removal is no longer needed, since the stack is cleared automatically.
C# : JobTemplate class would become JobTemplate struct.
Java / Python: JobTemplate class remains JobTemplate class, since they have no struct concept. Creation and destruction can be managed with OO mechanisms.
Another effect would be a change in the semantics of optional attributes. We already demand the syntactical inclusion of optional attribute names in the class definition. In C, the usage of a non-implemented optional attribute names gives a specialized error. With the change, we would have struct members that you are not allowed to fill in with some DRMAA libraries. But this would be detected only at submission time, since struct changes do not involve library code.
just only one concern: what if one of the DRM vendor wish to provide some JobTemplate attributes additional to those specified in DRMAA (as i remember SGE was an example)? I expect having different struct definition (different drmaa2.h ) leads to serious problems. I'm not saying no (actually having a self allocated struct in C is more convenient than using getter/setters - e.g. error handling), but this, for me, requires addressing the extension methodology on the IDL level.
And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at all then, regardless of the language.
Best, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
Cheers, -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
It's not a bad idea. I would really like to find a way to make it work, because it would make life quite a bit simpler. Is there a way to make it work? Daniel On 03/23/10 13:56, Peter Tröger wrote:
Ok, it was a bad idea. Thanks for commenting.
/Peter.
Am 22.03.2010 um 16:14 schrieb Daniel Templeton:
And that may be the original reason why the SGE C binding uses that pseudo-OO structure. It is more or less a hash map, allowing arbitrary name-value pairs to be stored, facilitating vendor-defined job template attributes. If we can't find a way around that issue, it's a show-stopper. We have to allow for vendor-defined attributes.
Daniel
On 03/22/10 07:29, Mariusz Mamoński wrote:
Hi Peter, all,
2010/3/22 Peter Tröger<peter@troeger.eu>:
Dear all,
we have an ongoing discussion about the possible removal of "createJobTemplate" and "deleteJobTemplate". The last proposal was to move this functions to the language bindings that need it. This is currently only C - all other languages have their own way of performing instantiation and termination explicitly.
After some more thinking, I think I got the true underlying issue. So far, we are treating job templates as instances with state and behavior - objects, in most languages. The only reason for doing this (so far) is the ability to throw errors on attribute access, since we need that for DRMAA's understanding of optional job template attributes. However, from all other perspectives, job templates are just value types. If you got one, you fill it, and then you pass the thing as a whole. In a RPC scenario, you would also expect to send filled job templates as a whole, similar to a filled JSDL document.
So if we change that, what would that mean:
C: Generic getter / setter functions for job template attributes would go away. Instead, you would create / delete JobTemplate structs directly. runJob() would take a pointer to this struct. Explicit removal is no longer needed, since the stack is cleared automatically.
C# : JobTemplate class would become JobTemplate struct.
Java / Python: JobTemplate class remains JobTemplate class, since they have no struct concept. Creation and destruction can be managed with OO mechanisms.
Another effect would be a change in the semantics of optional attributes. We already demand the syntactical inclusion of optional attribute names in the class definition. In C, the usage of a non-implemented optional attribute names gives a specialized error. With the change, we would have struct members that you are not allowed to fill in with some DRMAA libraries. But this would be detected only at submission time, since struct changes do not involve library code.
just only one concern: what if one of the DRM vendor wish to provide some JobTemplate attributes additional to those specified in DRMAA (as i remember SGE was an example)? I expect having different struct definition (different drmaa2.h ) leads to serious problems. I'm not saying no (actually having a self allocated struct in C is more convenient than using getter/setters - e.g. error handling), but this, for me, requires addressing the extension methodology on the IDL level.
And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at all then, regardless of the language.
Best, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
Cheers, -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
my biased point of view: use the saga model: { // create job template on the stack saga::job::description jd; // 3 equivalent ways for predefined attribute jd.set_attribute (saga::job::description_executable, "/bin/date"); jd.set_attribute ("Executable", "/bin/date"); jd.set_executable ("/bin/date"); // vendor-defined attributes: jd.set_attribute ("SGE:Executable-Type", "UniversalBinary"); // job template gets deleted from the stack when leaving the scope } Memory management of the hash table is done internally. The implementation also needs to maintain a list of predefined attributes - but that is anyway the case I think. C language bindings could of course still fall back to the explicit create/delete methods. Again, sorry for being biased... Best, Andre. Quoting [Daniel Templeton] (Mar 23 2010):
It's not a bad idea. I would really like to find a way to make it work, because it would make life quite a bit simpler. Is there a way to make it work?
Daniel
On 03/23/10 13:56, Peter Tröger wrote:
Ok, it was a bad idea. Thanks for commenting.
/Peter.
Am 22.03.2010 um 16:14 schrieb Daniel Templeton:
And that may be the original reason why the SGE C binding uses that pseudo-OO structure. It is more or less a hash map, allowing arbitrary name-value pairs to be stored, facilitating vendor-defined job template attributes. If we can't find a way around that issue, it's a show-stopper. We have to allow for vendor-defined attributes.
Daniel
On 03/22/10 07:29, Mariusz Mamo??ski wrote:
Hi Peter, all,
2010/3/22 Peter Tröger<peter@troeger.eu>:
Dear all,
we have an ongoing discussion about the possible removal of "createJobTemplate" and "deleteJobTemplate". The last proposal was to move this functions to the language bindings that need it. This is currently only C - all other languages have their own way of performing instantiation and termination explicitly.
After some more thinking, I think I got the true underlying issue. So far, we are treating job templates as instances with state and behavior - objects, in most languages. The only reason for doing this (so far) is the ability to throw errors on attribute access, since we need that for DRMAA's understanding of optional job template attributes. However, from all other perspectives, job templates are just value types. If you got one, you fill it, and then you pass the thing as a whole. In a RPC scenario, you would also expect to send filled job templates as a whole, similar to a filled JSDL document.
So if we change that, what would that mean:
C: Generic getter / setter functions for job template attributes would go away. Instead, you would create / delete JobTemplate structs directly. runJob() would take a pointer to this struct. Explicit removal is no longer needed, since the stack is cleared automatically.
C# : JobTemplate class would become JobTemplate struct.
Java / Python: JobTemplate class remains JobTemplate class, since they have no struct concept. Creation and destruction can be managed with OO mechanisms.
Another effect would be a change in the semantics of optional attributes. We already demand the syntactical inclusion of optional attribute names in the class definition. In C, the usage of a non-implemented optional attribute names gives a specialized error. With the change, we would have struct members that you are not allowed to fill in with some DRMAA libraries. But this would be detected only at submission time, since struct changes do not involve library code.
just only one concern: what if one of the DRM vendor wish to provide some JobTemplate attributes additional to those specified in DRMAA (as i remember SGE was an example)? I expect having different struct definition (different drmaa2.h ) leads to serious problems. I'm not saying no (actually having a self allocated struct in C is more convenient than using getter/setters - e.g. error handling), but this, for me, requires addressing the extension methodology on the IDL level.
And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at all then, regardless of the language.
Best, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
Cheers, -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- Nothing is ever easy.
Andre, Don't worry about being biased. I'm unabashedly SGE-biased. :) Unfortunately, though, what you proposed is what we're already discussing. It's the problem, not the solution. We're trying to have a consistent API across OO and non-OO languages. The issue goes away if we just add "except in C" to the description. Andre does bring up a valid point, though, which is that we really could just special-case the non-OO languages, just like we did in v1. Or, like I said before, we could leave the template management to the implementation, scoped to the session, with an overflow value of being allowed to free templates as needed. Daniel On 03/24/10 06:03, Andre Merzky wrote:
my biased point of view: use the saga model:
{ // create job template on the stack saga::job::description jd;
// 3 equivalent ways for predefined attribute jd.set_attribute (saga::job::description_executable, "/bin/date"); jd.set_attribute ("Executable", "/bin/date"); jd.set_executable ("/bin/date");
// vendor-defined attributes: jd.set_attribute ("SGE:Executable-Type", "UniversalBinary");
// job template gets deleted from the stack when leaving the scope }
Memory management of the hash table is done internally. The implementation also needs to maintain a list of predefined attributes - but that is anyway the case I think. C language bindings could of course still fall back to the explicit create/delete methods.
Again, sorry for being biased...
Best, Andre.
Quoting [Daniel Templeton] (Mar 23 2010):
It's not a bad idea. I would really like to find a way to make it work, because it would make life quite a bit simpler. Is there a way to make it work?
Daniel
On 03/23/10 13:56, Peter Tröger wrote:
Ok, it was a bad idea. Thanks for commenting.
/Peter.
Am 22.03.2010 um 16:14 schrieb Daniel Templeton:
And that may be the original reason why the SGE C binding uses that pseudo-OO structure. It is more or less a hash map, allowing arbitrary name-value pairs to be stored, facilitating vendor-defined job template attributes. If we can't find a way around that issue, it's a show-stopper. We have to allow for vendor-defined attributes.
Daniel
On 03/22/10 07:29, Mariusz Mamo??ski wrote:
Hi Peter, all,
2010/3/22 Peter Tröger<peter@troeger.eu>:
Dear all,
we have an ongoing discussion about the possible removal of "createJobTemplate" and "deleteJobTemplate". The last proposal was to move this functions to the language bindings that need it. This is currently only C - all other languages have their own way of performing instantiation and termination explicitly.
After some more thinking, I think I got the true underlying issue. So far, we are treating job templates as instances with state and behavior - objects, in most languages. The only reason for doing this (so far) is the ability to throw errors on attribute access, since we need that for DRMAA's understanding of optional job template attributes. However, from all other perspectives, job templates are just value types. If you got one, you fill it, and then you pass the thing as a whole. In a RPC scenario, you would also expect to send filled job templates as a whole, similar to a filled JSDL document.
So if we change that, what would that mean:
C: Generic getter / setter functions for job template attributes would go away. Instead, you would create / delete JobTemplate structs directly. runJob() would take a pointer to this struct. Explicit removal is no longer needed, since the stack is cleared automatically.
C# : JobTemplate class would become JobTemplate struct.
Java / Python: JobTemplate class remains JobTemplate class, since they have no struct concept. Creation and destruction can be managed with OO mechanisms.
Another effect would be a change in the semantics of optional attributes. We already demand the syntactical inclusion of optional attribute names in the class definition. In C, the usage of a non-implemented optional attribute names gives a specialized error. With the change, we would have struct members that you are not allowed to fill in with some DRMAA libraries. But this would be detected only at submission time, since struct changes do not involve library code.
just only one concern: what if one of the DRM vendor wish to provide some JobTemplate attributes additional to those specified in DRMAA (as i remember SGE was an example)? I expect having different struct definition (different drmaa2.h ) leads to serious problems. I'm not saying no (actually having a self allocated struct in C is more convenient than using getter/setters - e.g. error handling), but this, for me, requires addressing the extension methodology on the IDL level.
And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at all then, regardless of the language.
Best, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
Cheers, -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
Quoting [Daniel Templeton] (Mar 24 2010):
Andre,
Don't worry about being biased. I'm unabashedly SGE-biased. :)
:-)
Unfortunately, though, what you proposed is what we're already discussing. It's the problem, not the solution. We're trying to have a consistent API across OO and non-OO languages. The issue goes away if we just add "except in C" to the description.
Hmm, I thought the problem was:
just only one concern: what if one of the DRM vendor wish to provide some JobTemplate attributes additional to those specified in DRMAA (as i remember SGE was an example)? I expect having different struct definition (different drmaa2.h ) leads to serious problems. I'm not saying no (actually having a self allocated struct in C is more convenient than using getter/setters - e.g. error handling), but this, for me, requires addressing the extension methodology on the IDL level.
The SAGA scheme would still map to C in a very simple way // create job template drmaa_job_template_t jt = drmaa_job_template_create (); // set a predefined DRMAA attribute // these calls would always succeed, basically. drmaa_job_template_set_attribute (jt, DRMAA_JOB_TEMPLATE_EXECUTABLE, "/bin/date"); drmaa_job_template_set_attribute (jt, "Executable", "/bin/date"); drmaa_job_template_set_executable (jt, "/bin/date"); // set a vendor specific attribute // this call would return an error if the attribute is not // supported drmaa_job_template_set_attribute (jt, "SGE:Executable-Type", "UniversalBinary"); // free job template drmaa_job_template_destroy (jt); So, no special rule for C, and the same drmaa2.h for all vendors. Well, the special rule for C is to have a Create and Destrpy method - but you have that for all classes/types, job template is not special in that respect... Cheers, Andre.
Andre does bring up a valid point, though, which is that we really could just special-case the non-OO languages, just like we did in v1. Or, like I said before, we could leave the template management to the implementation, scoped to the session, with an overflow value of being allowed to free templates as needed.
Daniel
On 03/24/10 06:03, Andre Merzky wrote:
my biased point of view: use the saga model:
{ // create job template on the stack saga::job::description jd;
// 3 equivalent ways for predefined attribute jd.set_attribute (saga::job::description_executable, "/bin/date"); jd.set_attribute ("Executable", "/bin/date"); jd.set_executable ("/bin/date");
// vendor-defined attributes: jd.set_attribute ("SGE:Executable-Type", "UniversalBinary");
// job template gets deleted from the stack when leaving the scope }
Memory management of the hash table is done internally. The implementation also needs to maintain a list of predefined attributes - but that is anyway the case I think. C language bindings could of course still fall back to the explicit create/delete methods.
Again, sorry for being biased...
Best, Andre.
Quoting [Daniel Templeton] (Mar 23 2010):
It's not a bad idea. I would really like to find a way to make it work, because it would make life quite a bit simpler. Is there a way to make it work?
Daniel
On 03/23/10 13:56, Peter Tröger wrote:
Ok, it was a bad idea. Thanks for commenting.
/Peter.
Am 22.03.2010 um 16:14 schrieb Daniel Templeton:
And that may be the original reason why the SGE C binding uses that pseudo-OO structure. It is more or less a hash map, allowing arbitrary name-value pairs to be stored, facilitating vendor-defined job template attributes. If we can't find a way around that issue, it's a show-stopper. We have to allow for vendor-defined attributes.
Daniel
On 03/22/10 07:29, Mariusz Mamo??ski wrote:
Hi Peter, all,
2010/3/22 Peter Tröger<peter@troeger.eu>: >Dear all, > >we have an ongoing discussion about the possible removal of >"createJobTemplate" and "deleteJobTemplate". The last proposal was >to move this functions to the language bindings that need it. This >is currently only C - all other languages have their own way of >performing instantiation and termination explicitly. > >After some more thinking, I think I got the true underlying issue. >So far, we are treating job templates as instances with state and >behavior - objects, in most languages. The only reason for doing >this (so far) is the ability to throw errors on attribute access, >since we need that for DRMAA's understanding of optional job >template attributes. However, from all other perspectives, job >templates are just value types. If you got one, you fill it, and >then you pass the thing as a whole. In a RPC scenario, you would >also expect to send filled job templates as a whole, similar to a >filled JSDL document. > >So if we change that, what would that mean: > >C: Generic getter / setter functions for job template attributes >would go away. Instead, you would create / delete JobTemplate >structs directly. runJob() would take a pointer to this struct. >Explicit removal is no longer needed, since the stack is cleared >automatically. > >C# : JobTemplate class would become JobTemplate struct. > >Java / Python: JobTemplate class remains JobTemplate class, since >they have no struct concept. Creation and destruction can be >managed with OO mechanisms. > >Another effect would be a change in the semantics of optional >attributes. We already demand the syntactical inclusion of >optional attribute names in the class definition. In C, the usage >of a non-implemented optional attribute names gives a specialized >error. With the change, we would have struct members that you are >not allowed to fill in with some DRMAA libraries. But this would >be detected only at submission time, since struct changes do not >involve library code. > just only one concern: what if one of the DRM vendor wish to provide some JobTemplate attributes additional to those specified in DRMAA (as i remember SGE was an example)? I expect having different struct definition (different drmaa2.h ) leads to serious problems. I'm not saying no (actually having a self allocated struct in C is more convenient than using getter/setters - e.g. error handling), but this, for me, requires addressing the extension methodology on the IDL level.
>And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at >all then, regardless of the language. > >Best, >Peter. > > > > > >-- > drmaa-wg mailing list > drmaa-wg@ogf.org > http://www.ogf.org/mailman/listinfo/drmaa-wg >
Cheers, -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- Nothing is ever easy.
Hi, I have tried to collect all the problems, suggestions, and also my experience with different APIs and created a scratch page on my wiki: "How the JobTamplate could look if.. ") with several different possible options: http://fury.man.poznan.pl/~mmamonski/wiki/index.php/DRMAAv2/JobTemplate_-_Bi... Please review every of the 4 options and provide your comments! We can always combine some ideas. After writing this summary i asked one dummy question to myself: "What is the benefit of having vendor specific attributes if we already have nativeOptions (nativeSpecification)"? Cheers, On 24 March 2010 14:28, Andre Merzky <andre@merzky.net> wrote:
Quoting [Daniel Templeton] (Mar 24 2010):
Andre,
Don't worry about being biased. I'm unabashedly SGE-biased. :)
:-)
Unfortunately, though, what you proposed is what we're already discussing. It's the problem, not the solution. We're trying to have a consistent API across OO and non-OO languages. The issue goes away if we just add "except in C" to the description.
Hmm, I thought the problem was:
> just only one concern: what if one of the DRM vendor wish to > provide some JobTemplate attributes additional to those > specified in DRMAA (as i remember SGE was an example)? I > expect having different struct definition (different > drmaa2.h ) leads to serious problems. I'm not saying no > (actually having a self allocated struct in C is more > convenient than using getter/setters - e.g. error handling), > but this, for me, requires addressing the extension > methodology on the IDL level.
The SAGA scheme would still map to C in a very simple way
// create job template drmaa_job_template_t jt = drmaa_job_template_create ();
// set a predefined DRMAA attribute // these calls would always succeed, basically. drmaa_job_template_set_attribute (jt, DRMAA_JOB_TEMPLATE_EXECUTABLE, "/bin/date"); drmaa_job_template_set_attribute (jt, "Executable", "/bin/date"); drmaa_job_template_set_executable (jt, "/bin/date");
// set a vendor specific attribute // this call would return an error if the attribute is not // supported drmaa_job_template_set_attribute (jt, "SGE:Executable-Type", "UniversalBinary");
// free job template drmaa_job_template_destroy (jt);
So, no special rule for C, and the same drmaa2.h for all vendors.
Well, the special rule for C is to have a Create and Destrpy method - but you have that for all classes/types, job template is not special in that respect...
Cheers, Andre.
Andre does bring up a valid point, though, which is that we really could just special-case the non-OO languages, just like we did in v1. Or, like I said before, we could leave the template management to the implementation, scoped to the session, with an overflow value of being allowed to free templates as needed.
Daniel
On 03/24/10 06:03, Andre Merzky wrote:
my biased point of view: use the saga model:
{ // create job template on the stack saga::job::description jd;
// 3 equivalent ways for predefined attribute jd.set_attribute (saga::job::description_executable, "/bin/date"); jd.set_attribute ("Executable", "/bin/date"); jd.set_executable ("/bin/date");
// vendor-defined attributes: jd.set_attribute ("SGE:Executable-Type", "UniversalBinary");
// job template gets deleted from the stack when leaving the scope }
Memory management of the hash table is done internally. The implementation also needs to maintain a list of predefined attributes - but that is anyway the case I think. C language bindings could of course still fall back to the explicit create/delete methods.
Again, sorry for being biased...
Best, Andre.
Quoting [Daniel Templeton] (Mar 23 2010):
It's not a bad idea. I would really like to find a way to make it work, because it would make life quite a bit simpler. Is there a way to make it work?
Daniel
On 03/23/10 13:56, Peter Tröger wrote:
Ok, it was a bad idea. Thanks for commenting.
/Peter.
Am 22.03.2010 um 16:14 schrieb Daniel Templeton:
And that may be the original reason why the SGE C binding uses that pseudo-OO structure. It is more or less a hash map, allowing arbitrary name-value pairs to be stored, facilitating vendor-defined job template attributes. If we can't find a way around that issue, it's a show-stopper. We have to allow for vendor-defined attributes.
Daniel
On 03/22/10 07:29, Mariusz Mamo??ski wrote: >Hi Peter, all, > >2010/3/22 Peter Tröger<peter@troeger.eu>: >>Dear all, >> >>we have an ongoing discussion about the possible removal of >>"createJobTemplate" and "deleteJobTemplate". The last proposal was >>to move this functions to the language bindings that need it. This >>is currently only C - all other languages have their own way of >>performing instantiation and termination explicitly. >> >>After some more thinking, I think I got the true underlying issue. >>So far, we are treating job templates as instances with state and >>behavior - objects, in most languages. The only reason for doing >>this (so far) is the ability to throw errors on attribute access, >>since we need that for DRMAA's understanding of optional job >>template attributes. However, from all other perspectives, job >>templates are just value types. If you got one, you fill it, and >>then you pass the thing as a whole. In a RPC scenario, you would >>also expect to send filled job templates as a whole, similar to a >>filled JSDL document. >> >>So if we change that, what would that mean: >> >>C: Generic getter / setter functions for job template attributes >>would go away. Instead, you would create / delete JobTemplate >>structs directly. runJob() would take a pointer to this struct. >>Explicit removal is no longer needed, since the stack is cleared >>automatically. >> >>C# : JobTemplate class would become JobTemplate struct. >> >>Java / Python: JobTemplate class remains JobTemplate class, since >>they have no struct concept. Creation and destruction can be >>managed with OO mechanisms. >> >>Another effect would be a change in the semantics of optional >>attributes. We already demand the syntactical inclusion of >>optional attribute names in the class definition. In C, the usage >>of a non-implemented optional attribute names gives a specialized >>error. With the change, we would have struct members that you are >>not allowed to fill in with some DRMAA libraries. But this would >>be detected only at submission time, since struct changes do not >>involve library code. >> >just only one concern: what if one of the DRM vendor wish to provide >some JobTemplate attributes additional to those specified in DRMAA >(as >i remember SGE was an example)? I expect having different struct >definition (different drmaa2.h ) leads to serious problems. I'm not >saying no (actually having a self allocated struct in C is more >convenient than using getter/setters - e.g. error handling), but >this, >for me, requires addressing the extension methodology on the IDL >level. > >>And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at >>all then, regardless of the language. >> >>Best, >>Peter. >> >> >> >> >> >>-- >> drmaa-wg mailing list >> drmaa-wg@ogf.org >> http://www.ogf.org/mailman/listinfo/drmaa-wg >> > > >Cheers, -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- Nothing is ever easy. -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- Mariusz
The reason we have vendor extensions is that it's much cleaner than the native specification, and it gives the DRMAA implementation the opportunity to understand the meaning of the job template without having to understand how to parse the native specification. In the SGE implementation, the native specification is passed directly to SGE. DRMAA knows nothing about what's in it. (Hence it's not sanity checked.) I could live with the idea of making the job template a value type and adding in C an undefined struct type field (e.g. struct drmaa2_vendor_attributes_t) that vendors are allowed to define themselves, with an empty struct being a valid definition. How would this approach affect the other non-OO languages, like Perl? Without thinking about it too hard, I think Perl would actually behaving like an OO language in this case. Are there other non-OO languages we'd need to worry about? I doubt it, but we need to be careful before closing the door on any implementations. I'm still chewing on whether it's really OK for the job template to allow bogus values, expecting that the run() method will point out the problems. We had a discussion about allowing DRMAA to filter the list of jobs to be returned from a search. If the job template were a value type, it could be used as the filter. Fill in the fields that interest you, and leave the rest empty. You'll get all the jobs that have those values set in those fields. And the easiest answer would be not to allow wildcarding. See the Jini discovery service for a complete implementation of this approach. Daniel On 03/26/10 07:23, Mariusz Mamoński wrote:
Hi,
I have tried to collect all the problems, suggestions, and also my experience with different APIs and created a scratch page on my wiki: "How the JobTamplate could look if.. ") with several different possible options:
http://fury.man.poznan.pl/~mmamonski/wiki/index.php/DRMAAv2/JobTemplate_-_Bi...
Please review every of the 4 options and provide your comments! We can always combine some ideas.
After writing this summary i asked one dummy question to myself: "What is the benefit of having vendor specific attributes if we already have nativeOptions (nativeSpecification)"?
Cheers,
On 24 March 2010 14:28, Andre Merzky<andre@merzky.net> wrote:
Quoting [Daniel Templeton] (Mar 24 2010):
Andre,
Don't worry about being biased. I'm unabashedly SGE-biased. :)
:-)
Unfortunately, though, what you proposed is what we're already discussing. It's the problem, not the solution. We're trying to have a consistent API across OO and non-OO languages. The issue goes away if we just add "except in C" to the description.
Hmm, I thought the problem was:
>> just only one concern: what if one of the DRM vendor wish to >> provide some JobTemplate attributes additional to those >> specified in DRMAA (as i remember SGE was an example)? I >> expect having different struct definition (different >> drmaa2.h ) leads to serious problems. I'm not saying no >> (actually having a self allocated struct in C is more >> convenient than using getter/setters - e.g. error handling), >> but this, for me, requires addressing the extension >> methodology on the IDL level.
The SAGA scheme would still map to C in a very simple way
// create job template drmaa_job_template_t jt = drmaa_job_template_create ();
// set a predefined DRMAA attribute // these calls would always succeed, basically. drmaa_job_template_set_attribute (jt, DRMAA_JOB_TEMPLATE_EXECUTABLE, "/bin/date"); drmaa_job_template_set_attribute (jt, "Executable", "/bin/date"); drmaa_job_template_set_executable (jt, "/bin/date");
// set a vendor specific attribute // this call would return an error if the attribute is not // supported drmaa_job_template_set_attribute (jt, "SGE:Executable-Type", "UniversalBinary");
// free job template drmaa_job_template_destroy (jt);
So, no special rule for C, and the same drmaa2.h for all vendors.
Well, the special rule for C is to have a Create and Destrpy method - but you have that for all classes/types, job template is not special in that respect...
Cheers, Andre.
Andre does bring up a valid point, though, which is that we really could just special-case the non-OO languages, just like we did in v1. Or, like I said before, we could leave the template management to the implementation, scoped to the session, with an overflow value of being allowed to free templates as needed.
Daniel
On 03/24/10 06:03, Andre Merzky wrote:
my biased point of view: use the saga model:
{ // create job template on the stack saga::job::description jd;
// 3 equivalent ways for predefined attribute jd.set_attribute (saga::job::description_executable, "/bin/date"); jd.set_attribute ("Executable", "/bin/date"); jd.set_executable ("/bin/date");
// vendor-defined attributes: jd.set_attribute ("SGE:Executable-Type", "UniversalBinary");
// job template gets deleted from the stack when leaving the scope }
Memory management of the hash table is done internally. The implementation also needs to maintain a list of predefined attributes - but that is anyway the case I think. C language bindings could of course still fall back to the explicit create/delete methods.
Again, sorry for being biased...
Best, Andre.
Quoting [Daniel Templeton] (Mar 23 2010):
It's not a bad idea. I would really like to find a way to make it work, because it would make life quite a bit simpler. Is there a way to make it work?
Daniel
On 03/23/10 13:56, Peter Tröger wrote:
Ok, it was a bad idea. Thanks for commenting.
/Peter.
Am 22.03.2010 um 16:14 schrieb Daniel Templeton:
> And that may be the original reason why the SGE C binding uses that > pseudo-OO structure. It is more or less a hash map, allowing > arbitrary > name-value pairs to be stored, facilitating vendor-defined job > template > attributes. If we can't find a way around that issue, it's a > show-stopper. We have to allow for vendor-defined attributes. > > Daniel > > On 03/22/10 07:29, Mariusz Mamo??ski wrote: >> Hi Peter, all, >> >> 2010/3/22 Peter Tröger<peter@troeger.eu>: >>> Dear all, >>> >>> we have an ongoing discussion about the possible removal of >>> "createJobTemplate" and "deleteJobTemplate". The last proposal was >>> to move this functions to the language bindings that need it. This >>> is currently only C - all other languages have their own way of >>> performing instantiation and termination explicitly. >>> >>> After some more thinking, I think I got the true underlying issue. >>> So far, we are treating job templates as instances with state and >>> behavior - objects, in most languages. The only reason for doing >>> this (so far) is the ability to throw errors on attribute access, >>> since we need that for DRMAA's understanding of optional job >>> template attributes. However, from all other perspectives, job >>> templates are just value types. If you got one, you fill it, and >>> then you pass the thing as a whole. In a RPC scenario, you would >>> also expect to send filled job templates as a whole, similar to a >>> filled JSDL document. >>> >>> So if we change that, what would that mean: >>> >>> C: Generic getter / setter functions for job template attributes >>> would go away. Instead, you would create / delete JobTemplate >>> structs directly. runJob() would take a pointer to this struct. >>> Explicit removal is no longer needed, since the stack is cleared >>> automatically. >>> >>> C# : JobTemplate class would become JobTemplate struct. >>> >>> Java / Python: JobTemplate class remains JobTemplate class, since >>> they have no struct concept. Creation and destruction can be >>> managed with OO mechanisms. >>> >>> Another effect would be a change in the semantics of optional >>> attributes. We already demand the syntactical inclusion of >>> optional attribute names in the class definition. In C, the usage >>> of a non-implemented optional attribute names gives a specialized >>> error. With the change, we would have struct members that you are >>> not allowed to fill in with some DRMAA libraries. But this would >>> be detected only at submission time, since struct changes do not >>> involve library code. >>> >> just only one concern: what if one of the DRM vendor wish to provide >> some JobTemplate attributes additional to those specified in DRMAA >> (as >> i remember SGE was an example)? I expect having different struct >> definition (different drmaa2.h ) leads to serious problems. I'm not >> saying no (actually having a self allocated struct in C is more >> convenient than using getter/setters - e.g. error handling), but >> this, >> for me, requires addressing the extension methodology on the IDL >> level. >> >>> And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at >>> all then, regardless of the language. >>> >>> Best, >>> Peter. >>> >>> >>> >>> >>> >>> -- >>> drmaa-wg mailing list >>> drmaa-wg@ogf.org >>> http://www.ogf.org/mailman/listinfo/drmaa-wg >>> >> >> >> Cheers, > -- > drmaa-wg mailing list > drmaa-wg@ogf.org > http://www.ogf.org/mailman/listinfo/drmaa-wg
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- Nothing is ever easy. -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
The reason why the job template was originally considered an "object" was that the original prototype (the SGE C binding) treated it like one internally. (There's thing pseudo-OO data structure that permeates SGE, and the job template is implemented as one.) It had to be malloced and freed in the prototype, which required explicit create and delete methods. From the Java side, I never argued against it because the use of finalizers in Java is frowned upon because it gums up the garbage collector. The last time we talked about it, I'm pretty sure we decided that we'd leave the create and drop the delete, making the job templates scoped to the session. I guess you can look at that as the compromise solution. If we make the job template a struct in C, there's no way to initialize it with default values. The current SGE C binding doesn't assign default values either, but that's beside the point. It could if it wanted to. The Java binding does. Having the job template as a value type only would definitely simplify the issue of using DRMAA as a remote interface. Daniel On 03/22/10 07:05, Peter Tröger wrote:
Dear all,
we have an ongoing discussion about the possible removal of "createJobTemplate" and "deleteJobTemplate". The last proposal was to move this functions to the language bindings that need it. This is currently only C - all other languages have their own way of performing instantiation and termination explicitly.
After some more thinking, I think I got the true underlying issue. So far, we are treating job templates as instances with state and behavior - objects, in most languages. The only reason for doing this (so far) is the ability to throw errors on attribute access, since we need that for DRMAA's understanding of optional job template attributes. However, from all other perspectives, job templates are just value types. If you got one, you fill it, and then you pass the thing as a whole. In a RPC scenario, you would also expect to send filled job templates as a whole, similar to a filled JSDL document.
So if we change that, what would that mean:
C: Generic getter / setter functions for job template attributes would go away. Instead, you would create / delete JobTemplate structs directly. runJob() would take a pointer to this struct. Explicit removal is no longer needed, since the stack is cleared automatically.
C# : JobTemplate class would become JobTemplate struct.
Java / Python: JobTemplate class remains JobTemplate class, since they have no struct concept. Creation and destruction can be managed with OO mechanisms.
Another effect would be a change in the semantics of optional attributes. We already demand the syntactical inclusion of optional attribute names in the class definition. In C, the usage of a non-implemented optional attribute names gives a specialized error. With the change, we would have struct members that you are not allowed to fill in with some DRMAA libraries. But this would be detected only at submission time, since struct changes do not involve library code.
And "createJobTemplate" resp. "deleteJobTemplate" ? Not needed at all then, regardless of the language.
Best, Peter.
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
participants (4)
-
Andre Merzky -
Daniel Templeton -
Mariusz Mamoński -
Peter Tröger