Hi group(s), a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation? The reasons for doing that are compelling: JSDL is one of the most acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where these are already available. Well, I have however some problems with that approach, which are outlined in the cited email below. Do you guys have any other thoughts, and what solution would you prefer? Cheers, Andre. ----- Forwarded message from 'Andre Merzky' <andre@merzky.net> -----
From: 'Andre Merzky' <andre@merzky.net> To: Shantenu Jha <sjha@cct.lsu.edu> Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu>, 'Thilo Kielmann' <kielmann@cs.vu.nl>, 'Andre Merzky' <andre@merzky.net> Subject: Re: JSDL - SAGA
Hartmut and I discussed that somewhat last week. So he knows I am not wholehartedly for that option. SAGA is supposed to abstract the low level details, not to expose them.
JSDL is going to define a number of extensions now. Some of these extensions are very useful for us, others not. Mostly, they will be more complex than JSDL itself.
Are we going to support the extensions? Which? All/some? How to select? What error do we report on unsupported extensions? Do we mandate that extensions are supported by the backends? Which?
Even w/o extensions: is the job description updated after an JSDL attrib is set? What about those attribs which are not JSDL keys? Assume an implementation which implements the existing SAGA job description keys: MUST it support complete JSDL now? What error whould it report?
These are probably all solvable problems, and I do agree that there are advantages, i.e. the re-use of existing JSDL documents. Anyway, IMHO we should be careful, consider if we really have enough use cases etc. Also, a free function jsdl_to_job_description may do the trick, w/o complicating the job package itself.
Cheers, Andre.
Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach? Andre.
Quoting [Shantenu Jha]
What little I know, I think so to.
Quoting [Hartmut Kaiser]
Agree 100% The easiest way is probably just to add a attribute in the job description taking the whole JSDL.
Quoting [Thilo Kielmann]
Yes!
Quoting [Shantenu Jha]
Shouldn't we ensure that SAGA consumes JSDL w/o any problem/changes?
-- "So much time, so little to do..." -- Garfield
Folks, I think we are having conflicting goals here. (Technical goals, not personal ones ;-) On one hand, we have the "S for simplicity" in SAGA, and we must keep it. On the other hand, we have the necessity to support JSDL, future JSDL extensions, or any other types of job decscriptions that people want to use. (And still might to be invented.) Assume, JSDL++ (whatever it will look like in near future) will become a widely adopted standard. (Or anything else, doesn't matter in the following.) Then, SAGA implementations will have to use JSDL++, and to form JSDL++ from SAGA job descriptions. Simultaneously, users are likely to use JSDL++ themselves, and may wnat to use JSDL++ to express their resource needs. At this point in time, SAGA will sit in the middle, and it may be very clumsy to first translate from JSDL++ to a SAGA job description, and then back to JSDL++ somewhere "down under" in the implementation. For the very purpose of SAGA as a universal and simple grid API, it has to: - be independent of job description standards (mostly simpler than them) - support job description standards My suggestion: SAGA should have one class of job descriptions, and the possibility to create subclasses for more specific job descriptions (like JSDL). Such subclasses could be defined as separate extension documents (just like gridcpr). (Or was it "resource descriptions"???) With this approach, users could still write programs that are independent of the underlying job submission machinery, having a simplified view on jobs and resource attributes etc. etc. At the same time, subclassed job/resource descriptions could be "passed through" transparently from the API to the implementeation, without being converted back and forth, a process that is very likely to loose some important details. Is this a route to go? Thilo On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
From: 'Andre Merzky' <andre@merzky.net> To: SAGA RG <saga-rg@ogf.org> Subject: [SAGA-RG] Fwd (andre@merzky.net): Re: JSDL - SAGA
Hi group(s),
a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation?
The reasons for doing that are compelling: JSDL is one of the most acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where these are already available.
Well, I have however some problems with that approach, which are outlined in the cited email below.
Do you guys have any other thoughts, and what solution would you prefer?
Cheers, Andre.
----- Forwarded message from 'Andre Merzky' <andre@merzky.net> -----
From: 'Andre Merzky' <andre@merzky.net> To: Shantenu Jha <sjha@cct.lsu.edu> Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu>, 'Thilo Kielmann' <kielmann@cs.vu.nl>, 'Andre Merzky' <andre@merzky.net> Subject: Re: JSDL - SAGA
Hartmut and I discussed that somewhat last week. So he knows I am not wholehartedly for that option. SAGA is supposed to abstract the low level details, not to expose them.
JSDL is going to define a number of extensions now. Some of these extensions are very useful for us, others not. Mostly, they will be more complex than JSDL itself.
Are we going to support the extensions? Which? All/some? How to select? What error do we report on unsupported extensions? Do we mandate that extensions are supported by the backends? Which?
Even w/o extensions: is the job description updated after an JSDL attrib is set? What about those attribs which are not JSDL keys? Assume an implementation which implements the existing SAGA job description keys: MUST it support complete JSDL now? What error whould it report?
These are probably all solvable problems, and I do agree that there are advantages, i.e. the re-use of existing JSDL documents. Anyway, IMHO we should be careful, consider if we really have enough use cases etc. Also, a free function jsdl_to_job_description may do the trick, w/o complicating the job package itself.
Cheers, Andre.
Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach?
Andre.
Quoting [Shantenu Jha]
What little I know, I think so to.
Quoting [Hartmut Kaiser]
Agree 100% The easiest way is probably just to add a attribute in the job description taking the whole JSDL.
Quoting [Thilo Kielmann]
Yes!
Quoting [Shantenu Jha]
Shouldn't we ensure that SAGA consumes JSDL w/o any problem/changes?
-- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
-- Thilo Kielmann http://www.cs.vu.nl/~kielmann/
Quoting [Thilo Kielmann] (Feb 16 2007):
Folks,
I think we are having conflicting goals here. (Technical goals, not personal ones ;-)
On one hand, we have the "S for simplicity" in SAGA, and we must keep it.
On the other hand, we have the necessity to support JSDL, future JSDL extensions, or any other types of job decscriptions that people want to use. (And still might to be invented.)
That is one of the points I kind of disagree with: Do we really have the necessity to support native JSDL, or even JSDL++? The _only_ point in our use cases where JSDL is explicitely mentioned is the RealityGrid use case 11. Grid Technologies currently used: ------------------------------------- If you are currently using or developing this scenario, which grid technologies are you using or considering? Currently using: gsissh, globus-url-copy, globus-job-run, GSI, WSDL, Grid Services (OGSI::Lite), SOAP, XML, GridSphere, Access Grid, VizServer, Chromium, gsoap. Currently considering WS-RF, WS-Notification, WS-Security, SRB, JSDL, UNICORE, Sakai. '_consider_' using JSDL, and after listing WS-RF and other WS-Standards. I also think that this is the level where JSDL is aiming at: the JSDL spec specifies the target use cases of JSDL documents to be exchanged between WS clients and WS services - it is, IMHO, not intended to be a human readable or API level language (see paragraph 2,3 of introduction, and figure 1 in the JSDL spec).
Assume, JSDL++ (whatever it will look like in near future) will become a widely adopted standard. (Or anything else, doesn't matter in the following.) Then, SAGA implementations will have to use JSDL++, and to form JSDL++ from SAGA job descriptions.
Yes, that is right. And not a problem.
Simultaneously, users are likely to use JSDL++ themselves, and may wnat to use JSDL++ to express their resource needs.
That is what I doubt: users will in general NOT write XML documents manually. Even simple JSDL docs are not neccessarily human readable (IMHO), and the extensions will be, well, more interesting (from what I have seen about them so far). So that is why I think that...
At this point in time, SAGA will sit in the middle, and it may be very clumsy to first translate from JSDL++ to a SAGA job description, and then back to JSDL++ somewhere "down under" in the implementation.
... this situation will be the exception, not the rule.
For the very purpose of SAGA as a universal and simple grid API, it has to: - be independent of job description standards (mostly simpler than them)
We are! That is how our JD was derived/defined.
- support job description standards
We do! Our JD is easily translatable into JSDL.
My suggestion:
SAGA should have one class of job descriptions, and the possibility to create subclasses for more specific job descriptions (like JSDL). Such subclasses could be defined as separate extension documents (just like gridcpr). (Or was it "resource descriptions"???)
The problem with that is that you'd allow, at some point, also Globus-RSL job descriptions, or Naregi-RSL job descriptions, which bind your application to the specific back ends, and hence break portability. (BTW, I think the number of users which would like to re-use RSL is certainly larger than thos who want to reuse JSDL...)
With this approach, users could still write programs that are independent of the underlying job submission machinery, having a simplified view on jobs and resource attributes etc. etc. At the same time, subclassed job/resource descriptions could be "passed through" transparently from the API to the implementeation, without being converted back and forth, a process that is very likely to loose some important details.
passing things through is a bad idea - that is what we learned with GAT. I hoestly think we should stay away from that, as far as possible. My suggestion: If there emerge any use cases which are not satisfied with the job description we have in SAGA, then the job description in SAGA needs to be fixed. And yes, we should make sure that this fix is translatable into JSDL, and possibly others. Until we have such use cases, stay Simple! (i.e. change nothing). Cheers, Andre.
Is this a route to go?
Thilo
On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
From: 'Andre Merzky' <andre@merzky.net> To: SAGA RG <saga-rg@ogf.org> Subject: [SAGA-RG] Fwd (andre@merzky.net): Re: JSDL - SAGA
Hi group(s),
a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation?
The reasons for doing that are compelling: JSDL is one of the most acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where these are already available.
Well, I have however some problems with that approach, which are outlined in the cited email below.
Do you guys have any other thoughts, and what solution would you prefer?
Cheers, Andre.
----- Forwarded message from 'Andre Merzky' <andre@merzky.net> -----
From: 'Andre Merzky' <andre@merzky.net> To: Shantenu Jha <sjha@cct.lsu.edu> Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu>, 'Thilo Kielmann' <kielmann@cs.vu.nl>, 'Andre Merzky' <andre@merzky.net> Subject: Re: JSDL - SAGA
Hartmut and I discussed that somewhat last week. So he knows I am not wholehartedly for that option. SAGA is supposed to abstract the low level details, not to expose them.
JSDL is going to define a number of extensions now. Some of these extensions are very useful for us, others not. Mostly, they will be more complex than JSDL itself.
Are we going to support the extensions? Which? All/some? How to select? What error do we report on unsupported extensions? Do we mandate that extensions are supported by the backends? Which?
Even w/o extensions: is the job description updated after an JSDL attrib is set? What about those attribs which are not JSDL keys? Assume an implementation which implements the existing SAGA job description keys: MUST it support complete JSDL now? What error whould it report?
These are probably all solvable problems, and I do agree that there are advantages, i.e. the re-use of existing JSDL documents. Anyway, IMHO we should be careful, consider if we really have enough use cases etc. Also, a free function jsdl_to_job_description may do the trick, w/o complicating the job package itself.
Cheers, Andre.
Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach?
Andre.
Quoting [Shantenu Jha]
What little I know, I think so to.
Quoting [Hartmut Kaiser]
Agree 100% The easiest way is probably just to add a attribute in the job description taking the whole JSDL.
Quoting [Thilo Kielmann]
Yes!
Quoting [Shantenu Jha]
Shouldn't we ensure that SAGA consumes JSDL w/o any problem/changes?
-- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg -- "So much time, so little to do..." -- Garfield
Andre Merzky wrote:
Quoting [Thilo Kielmann] (Feb 16 2007):
Let me state the obvious first: I kind of agree with Thilo. We should support whatever wide spread used (read: standardized) job description language future may bring. SAGA is all about standards and we should align SAGA with other standards, especially if these are produced by the same body as SAGA (OGF).
I think we are having conflicting goals here. (Technical goals, not personal ones ;-)
On one hand, we have the "S for simplicity" in SAGA, and we must keep it.
On the other hand, we have the necessity to support JSDL, future JSDL extensions, or any other types of job decscriptions that people want to use. (And still might to be invented.)
That is one of the points I kind of disagree with: Do we really have the necessity to support native JSDL, or even JSDL++?
The _only_ point in our use cases where JSDL is explicitely mentioned is the RealityGrid use case
11. Grid Technologies currently used: -------------------------------------
If you are currently using or developing this scenario, which grid technologies are you using or considering?
Currently using: gsissh, globus-url-copy, globus-job-run, GSI, WSDL, Grid Services (OGSI::Lite), SOAP, XML, GridSphere, Access Grid, VizServer, Chromium, gsoap.
Currently considering WS-RF, WS-Notification, WS-Security, SRB, JSDL, UNICORE, Sakai.
'_consider_' using JSDL, and after listing WS-RF and other WS-Standards.
I also think that this is the level where JSDL is aiming at: the JSDL spec specifies the target use cases of JSDL documents to be exchanged between WS clients and WS services - it is, IMHO, not intended to be a human readable or API level language (see paragraph 2,3 of introduction, and figure 1 in the JSDL spec).
Agreed. JSDL is not meant to be human readable. But who cares?
Simultaneously, users are likely to use JSDL++ themselves, and may wnat to use JSDL++ to express their resource needs.
That is what I doubt: users will in general NOT write XML documents manually. Even simple JSDL docs are not neccessarily human readable (IMHO), and the extensions will be, well, more interesting (from what I have seen about them so far). So that is why I think that...
As I said: who cares? If JSDL++ will be widely adopted we can be reasonably sure we will get a lot of tooling support to generate these XML documents without (direct) human intervention.
At this point in time, SAGA will sit in the middle, and it may be very clumsy to first translate from JSDL++ to a SAGA job description, and then back to JSDL++ somewhere "down under" in the implementation.
... this situation will be the exception, not the rule.
Who are you to know? We don't know yet, what people will do with SAGA and I think this is part of the problem. We have the use cases - yes. But this was only a starting point, only a first impression we got from a couple of groups of interested people. Wide adoption of SAGA will produce a lot more use cases we need to be prepared for and not constraining our selfs against common sense is the only thing we can do now.
For the very purpose of SAGA as a universal and simple grid API, it has to: - be independent of job description standards (mostly simpler than them)
We are! That is how our JD was derived/defined.
- support job description standards
We do! Our JD is easily translatable into JSDL.
But no v.v.
My suggestion:
SAGA should have one class of job descriptions, and the possibility to create subclasses for more specific job descriptions (like JSDL). Such subclasses could be defined as separate extension documents (just like gridcpr). (Or was it "resource descriptions"???)
The problem with that is that you'd allow, at some point, also Globus-RSL job descriptions, or Naregi-RSL job descriptions, which bind your application to the specific back ends, and hence break portability. (BTW, I think the number of users which would like to re-use RSL is certainly larger than thos who want to reuse JSDL...)
No! RSL will not be standardized by OGF as a job description standard.
With this approach, users could still write programs that are independent of the underlying job submission machinery, having a simplified view on jobs and resource attributes etc. etc. At the same time, subclassed job/resource descriptions could be "passed through" transparently from the API to the implementeation, without being converted back and forth, a process that is very likely to loose some important details.
passing things through is a bad idea - that is what we learned with GAT. I hoestly think we should stay away from that, as far as possible.
Passing things through is a bad idea if you do now specify, what you're going to pass through. Otherwise it's just fine and part of the interface. We are passing through our job description to the adaptors as well...
My suggestion:
If there emerge any use cases which are not satisfied with the job description we have in SAGA, then the job description in SAGA needs to be fixed. And yes, we should make sure that this fix is translatable into JSDL, and possibly others. Until we have such use cases, stay Simple! (i.e. change nothing).
Why create two different standards out of the OGF for the same thing? Didn't we integrate the GridRPC into SAGA almost unchanged as well - just repackaged it into a fancier interface? In my opinion the SAGA job description can only provide a nice interface to the main OGF job description format (which is as it turned out JSDL) as a surplus. It should not replace it. Just my 2c. Regards Hartmut
Cheers, Andre.
Is this a route to go?
Thilo
From: 'Andre Merzky' <andre@merzky.net> To: SAGA RG <saga-rg@ogf.org> Subject: [SAGA-RG] Fwd (andre@merzky.net): Re: JSDL - SAGA
Hi group(s),
a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation?
The reasons for doing that are compelling: JSDL is one of
acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where
On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote: the most these are
already available.
Well, I have however some problems with that approach, which are outlined in the cited email below.
Do you guys have any other thoughts, and what solution would you prefer?
Cheers, Andre.
----- Forwarded message from 'Andre Merzky' <andre@merzky.net> -----
From: 'Andre Merzky' <andre@merzky.net> To: Shantenu Jha <sjha@cct.lsu.edu> Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu>, 'Thilo Kielmann' <kielmann@cs.vu.nl>, 'Andre Merzky' <andre@merzky.net> Subject: Re: JSDL - SAGA
Hartmut and I discussed that somewhat last week. So he knows I am not wholehartedly for that option. SAGA is supposed to abstract the low level details, not to expose them.
JSDL is going to define a number of extensions now. Some of these extensions are very useful for us, others not. Mostly, they will be more complex than JSDL itself.
Are we going to support the extensions? Which? All/some? How to select? What error do we report on unsupported extensions? Do we mandate that extensions are supported by the backends? Which?
Even w/o extensions: is the job description updated after an JSDL attrib is set? What about those attribs which are not JSDL keys? Assume an implementation which implements the existing SAGA job description keys: MUST it support complete JSDL now? What error whould it report?
These are probably all solvable problems, and I do agree that there are advantages, i.e. the re-use of existing JSDL documents. Anyway, IMHO we should be careful, consider if we really have enough use cases etc. Also, a free function jsdl_to_job_description may do the trick, w/o complicating the job package itself.
Cheers, Andre.
Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach?
Andre.
Quoting [Shantenu Jha]
What little I know, I think so to.
Quoting [Hartmut Kaiser]
Agree 100% The easiest way is probably just to add a attribute
in the job
description taking the whole JSDL.
Quoting [Thilo Kielmann]
Yes!
Quoting [Shantenu Jha] > > Shouldn't we ensure that SAGA consumes JSDL w/o any > problem/changes? >
-- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg -- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
Another related idea is for SAGA to have one class for job descriptions with a quite loose ability to add "hints", such as in MPI I/O. The hints would not be required and might not be used, but in some cases would be used, depending on what SAGA is sitting on top of. Dan Thilo Kielmann wrote (on 2/16/07 5:54 AM):
Folks,
I think we are having conflicting goals here. (Technical goals, not personal ones ;-)
On one hand, we have the "S for simplicity" in SAGA, and we must keep it.
On the other hand, we have the necessity to support JSDL, future JSDL extensions, or any other types of job decscriptions that people want to use. (And still might to be invented.)
Assume, JSDL++ (whatever it will look like in near future) will become a widely adopted standard. (Or anything else, doesn't matter in the following.) Then, SAGA implementations will have to use JSDL++, and to form JSDL++ from SAGA job descriptions. Simultaneously, users are likely to use JSDL++ themselves, and may wnat to use JSDL++ to express their resource needs. At this point in time, SAGA will sit in the middle, and it may be very clumsy to first translate from JSDL++ to a SAGA job description, and then back to JSDL++ somewhere "down under" in the implementation.
For the very purpose of SAGA as a universal and simple grid API, it has to: - be independent of job description standards (mostly simpler than them) - support job description standards
My suggestion:
SAGA should have one class of job descriptions, and the possibility to create subclasses for more specific job descriptions (like JSDL). Such subclasses could be defined as separate extension documents (just like gridcpr). (Or was it "resource descriptions"???)
With this approach, users could still write programs that are independent of the underlying job submission machinery, having a simplified view on jobs and resource attributes etc. etc. At the same time, subclassed job/resource descriptions could be "passed through" transparently from the API to the implementeation, without being converted back and forth, a process that is very likely to loose some important details.
Is this a route to go?
Thilo
On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
From: 'Andre Merzky' <andre@merzky.net> To: SAGA RG <saga-rg@ogf.org> Subject: [SAGA-RG] Fwd (andre@merzky.net): Re: JSDL - SAGA
Hi group(s),
a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation?
The reasons for doing that are compelling: JSDL is one of the most acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where these are already available.
Well, I have however some problems with that approach, which are outlined in the cited email below.
Do you guys have any other thoughts, and what solution would you prefer?
Cheers, Andre.
----- Forwarded message from 'Andre Merzky' <andre@merzky.net> -----
From: 'Andre Merzky' <andre@merzky.net> To: Shantenu Jha <sjha@cct.lsu.edu> Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu>, 'Thilo Kielmann' <kielmann@cs.vu.nl>, 'Andre Merzky' <andre@merzky.net> Subject: Re: JSDL - SAGA
Hartmut and I discussed that somewhat last week. So he knows I am not wholehartedly for that option. SAGA is supposed to abstract the low level details, not to expose them.
JSDL is going to define a number of extensions now. Some of these extensions are very useful for us, others not. Mostly, they will be more complex than JSDL itself.
Are we going to support the extensions? Which? All/some? How to select? What error do we report on unsupported extensions? Do we mandate that extensions are supported by the backends? Which?
Even w/o extensions: is the job description updated after an JSDL attrib is set? What about those attribs which are not JSDL keys? Assume an implementation which implements the existing SAGA job description keys: MUST it support complete JSDL now? What error whould it report?
These are probably all solvable problems, and I do agree that there are advantages, i.e. the re-use of existing JSDL documents. Anyway, IMHO we should be careful, consider if we really have enough use cases etc. Also, a free function jsdl_to_job_description may do the trick, w/o complicating the job package itself.
Cheers, Andre.
Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach?
Andre.
Quoting [Shantenu Jha]
What little I know, I think so to.
Quoting [Hartmut Kaiser]
Agree 100% The easiest way is probably just to add a attribute in the job description taking the whole JSDL.
Quoting [Thilo Kielmann]
Yes!
Quoting [Shantenu Jha]
Shouldn't we ensure that SAGA consumes JSDL w/o any problem/changes?
-- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
Dear all, I have given me the week-end to get some reflection on the problem. As mentioned it is not a trivial issue. If things goes out of hand, we might as well rename the project CAGA of not go gaga :P The necessity to support JSDL 1.0.x is a necessity, since it is an OFG standard. I do not think JSDL 1.0 is overly complex right now. The Job Description should be made such that passing from one form to another is automatic (JD -> JSDL). The support of a property key for JSDL isn't bad; if used, all other properties can be potentially ignored. The JD would not have to parse it. The current specification is simple enough to not violate the "S" rule of SAGA. The problem we have is with all the up coming extensions. In my case, I clearly violated the SAGA API rules in order to make the system run with NAREGI. I tweaked the problem to solve my case. In the 80/20 rule, it probably means that the SAGA job submission won't be usable. I will have to write code that does not deal with SAGA and goes directly to the underwear. Now I am speaking as a Grid "expert" and not as a Grid newbie (which is the SAGA user base). If you want to use complex Grid programming and do complex things, SAGA might not be the right solution. The proposal made by Dan is not elegant in my opinion because it will force developer to know the underlying technology they want to use; it will also clutter the JD. The proposal from Thilo might work for OO languages because you can hide a lot behind patterns, but it won't mean a specific implementation will support such scheme. The passing through solution seems to be mildly seen by the some GAT people here. I would say no, if the error handling is explicit enough to return a message telling that this JSDL cannot be interpreted by the underlying system. I will not see this as a bad solution either. -- Best regards, Pascal Kleijer ---------------------------------------------------------------- HPC Marketing Promotion Division, NEC Corporation 1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan. Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382 Daniel S. Katz wrote:
Another related idea is for SAGA to have one class for job descriptions with a quite loose ability to add "hints", such as in MPI I/O. The hints would not be required and might not be used, but in some cases would be used, depending on what SAGA is sitting on top of.
Dan
Thilo Kielmann wrote (on 2/16/07 5:54 AM):
Folks,
I think we are having conflicting goals here. (Technical goals, not personal ones ;-)
On one hand, we have the "S for simplicity" in SAGA, and we must keep it.
On the other hand, we have the necessity to support JSDL, future JSDL extensions, or any other types of job decscriptions that people want to use. (And still might to be invented.)
Assume, JSDL++ (whatever it will look like in near future) will become a widely adopted standard. (Or anything else, doesn't matter in the following.) Then, SAGA implementations will have to use JSDL++, and to form JSDL++ from SAGA job descriptions. Simultaneously, users are likely to use JSDL++ themselves, and may wnat to use JSDL++ to express their resource needs. At this point in time, SAGA will sit in the middle, and it may be very clumsy to first translate from JSDL++ to a SAGA job description, and then back to JSDL++ somewhere "down under" in the implementation.
For the very purpose of SAGA as a universal and simple grid API, it has to: - be independent of job description standards (mostly simpler than them) - support job description standards
My suggestion:
SAGA should have one class of job descriptions, and the possibility to create subclasses for more specific job descriptions (like JSDL). Such subclasses could be defined as separate extension documents (just like gridcpr). (Or was it "resource descriptions"???)
With this approach, users could still write programs that are independent of the underlying job submission machinery, having a simplified view on jobs and resource attributes etc. etc. At the same time, subclassed job/resource descriptions could be "passed through" transparently from the API to the implementeation, without being converted back and forth, a process that is very likely to loose some important details.
Is this a route to go?
Thilo
On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
From: 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> To: SAGA RG <saga-rg@ogf.org> <mailto:saga-rg@ogf.org> Subject: [SAGA-RG] Fwd (andre@merzky.net <mailto:andre@merzky.net>): Re: JSDL - SAGA
Hi group(s),
a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation?
The reasons for doing that are compelling: JSDL is one of the most acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where these are already available.
Well, I have however some problems with that approach, which are outlined in the cited email below.
Do you guys have any other thoughts, and what solution would you prefer?
Cheers, Andre.
----- Forwarded message from 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> -----
From: 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> To: Shantenu Jha <sjha@cct.lsu.edu> <mailto:sjha@cct.lsu.edu> Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu> <mailto:hkaiser@cct.lsu.edu>, 'Thilo Kielmann' <kielmann@cs.vu.nl> <mailto:kielmann@cs.vu.nl>, 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> Subject: Re: JSDL - SAGA
Hartmut and I discussed that somewhat last week. So he knows I am not wholehartedly for that option. SAGA is supposed to abstract the low level details, not to expose them.
JSDL is going to define a number of extensions now. Some of these extensions are very useful for us, others not. Mostly, they will be more complex than JSDL itself.
Are we going to support the extensions? Which? All/some? How to select? What error do we report on unsupported extensions? Do we mandate that extensions are supported by the backends? Which?
Even w/o extensions: is the job description updated after an JSDL attrib is set? What about those attribs which are not JSDL keys? Assume an implementation which implements the existing SAGA job description keys: MUST it support complete JSDL now? What error whould it report?
These are probably all solvable problems, and I do agree that there are advantages, i.e. the re-use of existing JSDL documents. Anyway, IMHO we should be careful, consider if we really have enough use cases etc. Also, a free function jsdl_to_job_description may do the trick, w/o complicating the job package itself.
Cheers, Andre.
Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach?
Andre.
Quoting [Shantenu Jha]
What little I know, I think so to.
Quoting [Hartmut Kaiser]
Agree 100% The easiest way is probably just to add a attribute in the job description taking the whole JSDL.
Quoting [Thilo Kielmann]
Yes!
Quoting [Shantenu Jha]
> Shouldn't we ensure that SAGA consumes JSDL w/o any > problem/changes? > > -- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org <mailto:saga-rg@ogf.org> http://www.ogf.org/mailman/listinfo/saga-rg
------------------------------------------------------------------------
-- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
Hi all, After thinking about this issue a bit more I would like to make a even more 'radical' suggestion. Let's recap: - JSDL is an OGF Standard now and there is no way of ignoring it in SAGA - the current SAGA JD is more constrained in terms of scope if compared to JSDL - there will be a lot of middleware supporting JSDL in the future - there wll be a lot of tooling support to create JSDL in the future A viable solution to this in SAGA could be (duck): - redo the SAGA job_service and job interfaces so that these work solely based on JSDL strings - supply a SAGA JD component providing a _simple_ interface for JSDL construction: job_description::construct(string jsdl); job_description::as_jsdl(); job_service::create_job(string jsdl); job::migrate(..., string jsdl, ...); where we clearly could stick to JSDL V1 for this conversion and the allowed attribute set on our job descriptions. Regards Hartmut
-----Original Message----- From: saga-rg-bounces@ogf.org [mailto:saga-rg-bounces@ogf.org] On Behalf Of Pascal Kleijer Sent: Sunday, February 18, 2007 8:51 PM To: SAGA RG Subject: Re: [SAGA-RG] Fwd (andre@merzky.net): Re: JSDL - SAGA
Dear all,
I have given me the week-end to get some reflection on the problem. As mentioned it is not a trivial issue. If things goes out of hand, we might as well rename the project CAGA of not go gaga :P
The necessity to support JSDL 1.0.x is a necessity, since it is an OFG standard. I do not think JSDL 1.0 is overly complex right now. The Job Description should be made such that passing from one form to another is automatic (JD -> JSDL). The support of a property key for JSDL isn't bad; if used, all other properties can be potentially ignored. The JD would not have to parse it. The current specification is simple enough to not violate the "S" rule of SAGA.
The problem we have is with all the up coming extensions. In my case, I clearly violated the SAGA API rules in order to make the system run with NAREGI. I tweaked the problem to solve my case. In the 80/20 rule, it probably means that the SAGA job submission won't be usable. I will have to write code that does not deal with SAGA and goes directly to the underwear. Now I am speaking as a Grid "expert" and not as a Grid newbie (which is the SAGA user base). If you want to use complex Grid programming and do complex things, SAGA might not be the right solution.
The proposal made by Dan is not elegant in my opinion because it will force developer to know the underlying technology they want to use; it will also clutter the JD. The proposal from Thilo might work for OO languages because you can hide a lot behind patterns, but it won't mean a specific implementation will support such scheme.
The passing through solution seems to be mildly seen by the some GAT people here. I would say no, if the error handling is explicit enough to return a message telling that this JSDL cannot be interpreted by the underlying system. I will not see this as a bad solution either.
--
Best regards, Pascal Kleijer
---------------------------------------------------------------- HPC Marketing Promotion Division, NEC Corporation 1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan. Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382
Another related idea is for SAGA to have one class for job descriptions with a quite loose ability to add "hints", such as in MPI I/O. The hints would not be required and might not be used, but in some cases would be used, depending on what SAGA is sitting on top of.
Dan
Thilo Kielmann wrote (on 2/16/07 5:54 AM):
Folks,
I think we are having conflicting goals here. (Technical goals, not personal ones ;-)
On one hand, we have the "S for simplicity" in SAGA, and we must keep it.
On the other hand, we have the necessity to support JSDL, future JSDL extensions, or any other types of job decscriptions that
(And still might to be invented.)
Assume, JSDL++ (whatever it will look like in near future) will become a widely adopted standard. (Or anything else, doesn't matter in the following.) Then, SAGA implementations will have to use JSDL++, and to form JSDL++ from SAGA job descriptions. Simultaneously, users are likely to use JSDL++ themselves, and may wnat to use JSDL++ to express their resource needs. At this point in time, SAGA will sit in the middle, and it may be very clumsy to first translate from JSDL++ to a SAGA job description, and then back to JSDL++ somewhere "down under" in the implementation.
For the very purpose of SAGA as a universal and simple grid API, it has to: - be independent of job description standards (mostly simpler than them) - support job description standards
My suggestion:
SAGA should have one class of job descriptions, and the
to create subclasses for more specific job descriptions (like JSDL). Such subclasses could be defined as separate extension documents (just like gridcpr). (Or was it "resource descriptions"???)
With this approach, users could still write programs that are independent of the underlying job submission machinery, having a simplified view on jobs and resource attributes etc. etc. At the same time, subclassed job/resource descriptions could be "passed through" transparently from the API to the implementeation, without being converted back and forth, a process that is very likely to loose some important details.
Is this a route to go?
Thilo
On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
From: 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> To: SAGA RG <saga-rg@ogf.org> <mailto:saga-rg@ogf.org> Subject: [SAGA-RG] Fwd (andre@merzky.net <mailto:andre@merzky.net>): Re: JSDL - SAGA
Hi group(s),
a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation?
The reasons for doing that are compelling: JSDL is one of
acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where
already available.
Well, I have however some problems with that approach, which are outlined in the cited email below.
Do you guys have any other thoughts, and what solution would you prefer?
Cheers, Andre.
----- Forwarded message from 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> -----
From: 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> To: Shantenu Jha <sjha@cct.lsu.edu> <mailto:sjha@cct.lsu.edu> Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu> <mailto:hkaiser@cct.lsu.edu>, 'Thilo Kielmann' <kielmann@cs.vu.nl> <mailto:kielmann@cs.vu.nl>, 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> Subject: Re: JSDL - SAGA
Hartmut and I discussed that somewhat last week. So he knows I am not wholehartedly for that option. SAGA is supposed to abstract the low level details, not to expose them.
JSDL is going to define a number of extensions now. Some of these extensions are very useful for us, others not. Mostly, they will be more complex than JSDL itself.
Are we going to support the extensions? Which? All/some? How to select? What error do we report on unsupported extensions? Do we mandate that extensions are supported by the backends? Which?
Even w/o extensions: is the job description updated after an JSDL attrib is set? What about those attribs which are not JSDL keys? Assume an implementation which implements the existing SAGA job description keys: MUST it support complete JSDL now? What error whould it report?
These are probably all solvable problems, and I do agree
Daniel S. Katz wrote: people want to use. possibility the most these are that there
are advantages, i.e. the re-use of existing JSDL documents. Anyway, IMHO we should be careful, consider if we really have enough use cases etc. Also, a free function jsdl_to_job_description may do the trick, w/o complicating the job package itself.
Cheers, Andre.
Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach?
Andre.
Quoting [Shantenu Jha]
What little I know, I think so to.
Quoting [Hartmut Kaiser]
> Agree 100% > The easiest way is probably just to add a attribute in the job > description taking the whole JSDL. > > Quoting [Thilo Kielmann]
> Yes! > > > Quoting [Shantenu Jha] > >> Shouldn't we ensure that SAGA consumes JSDL w/o any >> problem/changes? >> >> -- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org <mailto:saga-rg@ogf.org> http://www.ogf.org/mailman/listinfo/saga-rg
----------------------------------------------------------------------
--
-- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
Ha, actually, that is what I expected this discussion would lead to :-P You write:
- JSDL is an OGF Standard now and there is no way of ignoring it in SAGA
and Pascal:
The necessity to support JSDL 1.0.x is a necessity, since it is an OFG standard.
I think we need to step back a little. Yes, JSDL is an OGF standard - but so is OGSA, OGSA Security, OGSA-DAIS. Yes, we should support JSDL, and in particular be implementable on top of JSDL. We also should support OGSA, and in particular be implementable on top of OGSA. Same with DAIS. That does NOT mean that we need an API to write JSDL documents, nor an API to write OGSA Base NOtification documents, nor an API to write DAIS Collection Access XML docs. SAGA is NOT a general purpose API for existing OGF standards. It is _designed_ to be incomplete. Quoting [Hartmut Kaiser] (Feb 19 2007):
Hi all,
After thinking about this issue a bit more I would like to make a even more 'radical' suggestion.
Let's recap: - JSDL is an OGF Standard now and there is no way of ignoring it in SAGA - the current SAGA JD is more constrained in terms of scope if compared to JSDL - there will be a lot of middleware supporting JSDL in the future - there wll be a lot of tooling support to create JSDL in the future
A viable solution to this in SAGA could be (duck):
- redo the SAGA job_service and job interfaces so that these work solely based on JSDL strings
Further, the saga job description has a (small) number of keys which are not supported by JSDL, and will not be in the near future. Those are scheduling related: Queue, JobStartTime etc.
- supply a SAGA JD component providing a _simple_ interface for JSDL construction:
The JSDL schema has repeatable, hierarchical elements, with 0..n occurences of sections etc etc. It can include XML snippets from arbitrary name spaces. Just one example from many in the JSDL spec: "7.1 Attribute Extension Every JSDL element allows for additional attributes, as many as needed, provided these attributes have a namespace other than the normative namespaces defined by JSDL." There is a good API to write XML documents (DOM), and there are many tools to do so. Why should SAGA re-do that? None of these APIs/tools is simple - they cannot be, by design. Please check the JSDL spec: JSDL is simple for middleware level - but it is too complex (and I mean really) on the level SAGA is targeting at. So, providing an API to create JSDL is, IMHO, out of scope.
job_description::construct(string jsdl); job_description::as_jsdl();
job_service::create_job(string jsdl); job::migrate(..., string jsdl, ...);
where we clearly could stick to JSDL V1 for this conversion and the allowed attribute set on our job descriptions.
I honestly don't see the reason why we should deviate from our design principles (simple API, orient on use cases, don't do more than users ask for) just because JSDL exists. Its nice it exists, we support it (it can and should be used by SAGA _implementations_). But it has no more appeal on API level than has OGSA Base Notification, for example. So, my proposal is: - stick to the JD we have in SAGA (it works nicely with JSDL) - support upcoming JSDL extensions in due time, either by new keys on the job description (JSDL scheduling), or programmatically (i.e. JSDL parameter sweeps) - if there are use cases asking for it (only then), provide a jsdl_to_jd converter Well, I realise I am repeating myself - not a good sign. We probably should give the topic anoter week of rants, and then do a straw poll on the mailing list, or continue to fight on a phone call... :) Cheers, Andre.
Regards Hartmut
-----Original Message----- From: saga-rg-bounces@ogf.org [mailto:saga-rg-bounces@ogf.org] On Behalf Of Pascal Kleijer Sent: Sunday, February 18, 2007 8:51 PM To: SAGA RG Subject: Re: [SAGA-RG] Fwd (andre@merzky.net): Re: JSDL - SAGA
Dear all,
I have given me the week-end to get some reflection on the problem. As mentioned it is not a trivial issue. If things goes out of hand, we might as well rename the project CAGA of not go gaga :P
The necessity to support JSDL 1.0.x is a necessity, since it is an OFG standard. I do not think JSDL 1.0 is overly complex right now. The Job Description should be made such that passing from one form to another is automatic (JD -> JSDL). The support of a property key for JSDL isn't bad; if used, all other properties can be potentially ignored. The JD would not have to parse it. The current specification is simple enough to not violate the "S" rule of SAGA.
The problem we have is with all the up coming extensions. In my case, I clearly violated the SAGA API rules in order to make the system run with NAREGI. I tweaked the problem to solve my case. In the 80/20 rule, it probably means that the SAGA job submission won't be usable. I will have to write code that does not deal with SAGA and goes directly to the underwear. Now I am speaking as a Grid "expert" and not as a Grid newbie (which is the SAGA user base). If you want to use complex Grid programming and do complex things, SAGA might not be the right solution.
The proposal made by Dan is not elegant in my opinion because it will force developer to know the underlying technology they want to use; it will also clutter the JD. The proposal from Thilo might work for OO languages because you can hide a lot behind patterns, but it won't mean a specific implementation will support such scheme.
The passing through solution seems to be mildly seen by the some GAT people here. I would say no, if the error handling is explicit enough to return a message telling that this JSDL cannot be interpreted by the underlying system. I will not see this as a bad solution either.
--
Best regards, Pascal Kleijer
---------------------------------------------------------------- HPC Marketing Promotion Division, NEC Corporation 1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan. Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382
Another related idea is for SAGA to have one class for job descriptions with a quite loose ability to add "hints", such as in MPI I/O. The hints would not be required and might not be used, but in some cases would be used, depending on what SAGA is sitting on top of.
Dan
Thilo Kielmann wrote (on 2/16/07 5:54 AM):
Folks,
I think we are having conflicting goals here. (Technical goals, not personal ones ;-)
On one hand, we have the "S for simplicity" in SAGA, and we must keep it.
On the other hand, we have the necessity to support JSDL, future JSDL extensions, or any other types of job decscriptions that
(And still might to be invented.)
Assume, JSDL++ (whatever it will look like in near future) will become a widely adopted standard. (Or anything else, doesn't matter in the following.) Then, SAGA implementations will have to use JSDL++, and to form JSDL++ from SAGA job descriptions. Simultaneously, users are likely to use JSDL++ themselves, and may wnat to use JSDL++ to express their resource needs. At this point in time, SAGA will sit in the middle, and it may be very clumsy to first translate from JSDL++ to a SAGA job description, and then back to JSDL++ somewhere "down under" in the implementation.
For the very purpose of SAGA as a universal and simple grid API, it has to: - be independent of job description standards (mostly simpler than them) - support job description standards
My suggestion:
SAGA should have one class of job descriptions, and the
to create subclasses for more specific job descriptions (like JSDL). Such subclasses could be defined as separate extension documents (just like gridcpr). (Or was it "resource descriptions"???)
With this approach, users could still write programs that are independent of the underlying job submission machinery, having a simplified view on jobs and resource attributes etc. etc. At the same time, subclassed job/resource descriptions could be "passed through" transparently from the API to the implementeation, without being converted back and forth, a process that is very likely to loose some important details.
Is this a route to go?
Thilo
On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
From: 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> To: SAGA RG <saga-rg@ogf.org> <mailto:saga-rg@ogf.org> Subject: [SAGA-RG] Fwd (andre@merzky.net <mailto:andre@merzky.net>): Re: JSDL - SAGA
Hi group(s),
a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation?
The reasons for doing that are compelling: JSDL is one of
acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where
already available.
Well, I have however some problems with that approach, which are outlined in the cited email below.
Do you guys have any other thoughts, and what solution would you prefer?
Cheers, Andre.
----- Forwarded message from 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> -----
From: 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> To: Shantenu Jha <sjha@cct.lsu.edu> <mailto:sjha@cct.lsu.edu> Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu> <mailto:hkaiser@cct.lsu.edu>, 'Thilo Kielmann' <kielmann@cs.vu.nl> <mailto:kielmann@cs.vu.nl>, 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> Subject: Re: JSDL - SAGA
Hartmut and I discussed that somewhat last week. So he knows I am not wholehartedly for that option. SAGA is supposed to abstract the low level details, not to expose them.
JSDL is going to define a number of extensions now. Some of these extensions are very useful for us, others not. Mostly, they will be more complex than JSDL itself.
Are we going to support the extensions? Which? All/some? How to select? What error do we report on unsupported extensions? Do we mandate that extensions are supported by the backends? Which?
Even w/o extensions: is the job description updated after an JSDL attrib is set? What about those attribs which are not JSDL keys? Assume an implementation which implements the existing SAGA job description keys: MUST it support complete JSDL now? What error whould it report?
These are probably all solvable problems, and I do agree
Daniel S. Katz wrote: people want to use. possibility the most these are that there
are advantages, i.e. the re-use of existing JSDL documents. Anyway, IMHO we should be careful, consider if we really have enough use cases etc. Also, a free function jsdl_to_job_description may do the trick, w/o complicating the job package itself.
Cheers, Andre.
Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach?
Andre.
Quoting [Shantenu Jha]
> What little I know, I think so to. > > > Quoting [Hartmut Kaiser] > >> Agree 100% >> The easiest way is probably just to add a attribute in the job >> description taking the whole JSDL. >> >> > Quoting [Thilo Kielmann] > >> Yes! >> >> >> Quoting [Shantenu Jha] >> >>> Shouldn't we ensure that SAGA consumes JSDL w/o any >>> problem/changes? >>> >>> -- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org <mailto:saga-rg@ogf.org> http://www.ogf.org/mailman/listinfo/saga-rg
----------------------------------------------------------------------
--
-- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg -- "So much time, so little to do..." -- Garfield
You write:
- JSDL is an OGF Standard now and there is no way of ignoring it in SAGA
My 2p. SAGA will probably use JSDL internally - but that will never be seen. Adding an operation to accept accept a JSDL document describing the job to be run would probably be useful. I'm sure if you redid the use cases now that JSDL exists and is being adopted this would emerge as a requirement. Review of the job attributes to see if these need to be adjusted/extended in light of the JSDL specifications. Steven -- ---------------------------------------------------------------- Dr Steven Newhouse Mob:+44(0)7920489420 Tel:+44(0)23 80598789 Director, Open Middleware Infrastructure Institute-UK (OMII-UK) c/o Suite 6005, Faraday Building (B21), Highfield Campus, University of Southampton, Highfield, Southampton, SO17 1BJ, UK
Hi all, I first sought about a solution like Hartmut, but SAGA is targeting Grid newbies, so JSDL might be out of the 80/20 rule scope. I am quit happy with the current set the JD provides; now would I use a Job Submission Framework (JSF) for NAREGI, the current SAGA won't work perfectly. NAREGI has extended the scope to support MPI jobs and I need to support that. I can always stick to my SAGA-style implementation for the moment. To allow power users to still make use of the SAGA adding a simple JSDL property that over rules all others and is passed directly through to the back-end would be fine. The API documentation should clearly tell that if the JSDL tag is used it supersedes all other settings. -- Best regards, Pascal Kleijer ---------------------------------------------------------------- HPC Marketing Promotion Division, NEC Corporation 1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan. Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382 Andre Merzky wrote:
Ha, actually, that is what I expected this discussion would lead to :-P
You write:
- JSDL is an OGF Standard now and there is no way of ignoring it in SAGA
and Pascal:
The necessity to support JSDL 1.0.x is a necessity, since it is an OFG standard.
I think we need to step back a little. Yes, JSDL is an OGF standard - but so is OGSA, OGSA Security, OGSA-DAIS.
Yes, we should support JSDL, and in particular be implementable on top of JSDL. We also should support OGSA, and in particular be implementable on top of OGSA. Same with DAIS.
That does NOT mean that we need an API to write JSDL documents, nor an API to write OGSA Base NOtification documents, nor an API to write DAIS Collection Access XML docs.
SAGA is NOT a general purpose API for existing OGF standards. It is _designed_ to be incomplete.
Quoting [Hartmut Kaiser] (Feb 19 2007):
Hi all,
After thinking about this issue a bit more I would like to make a even more 'radical' suggestion.
Let's recap: - JSDL is an OGF Standard now and there is no way of ignoring it in SAGA - the current SAGA JD is more constrained in terms of scope if compared to JSDL - there will be a lot of middleware supporting JSDL in the future - there wll be a lot of tooling support to create JSDL in the future
A viable solution to this in SAGA could be (duck):
- redo the SAGA job_service and job interfaces so that these work solely based on JSDL strings
Further, the saga job description has a (small) number of keys which are not supported by JSDL, and will not be in the near future. Those are scheduling related: Queue, JobStartTime etc.
- supply a SAGA JD component providing a _simple_ interface for JSDL construction:
The JSDL schema has repeatable, hierarchical elements, with 0..n occurences of sections etc etc. It can include XML snippets from arbitrary name spaces. Just one example from many in the JSDL spec:
"7.1 Attribute Extension
Every JSDL element allows for additional attributes, as many as needed, provided these attributes have a namespace other than the normative namespaces defined by JSDL."
There is a good API to write XML documents (DOM), and there are many tools to do so. Why should SAGA re-do that? None of these APIs/tools is simple - they cannot be, by design.
Please check the JSDL spec: JSDL is simple for middleware level - but it is too complex (and I mean really) on the level SAGA is targeting at.
So, providing an API to create JSDL is, IMHO, out of scope.
job_description::construct(string jsdl); job_description::as_jsdl();
job_service::create_job(string jsdl); job::migrate(..., string jsdl, ...);
where we clearly could stick to JSDL V1 for this conversion and the allowed attribute set on our job descriptions.
I honestly don't see the reason why we should deviate from our design principles (simple API, orient on use cases, don't do more than users ask for) just because JSDL exists.
Its nice it exists, we support it (it can and should be used by SAGA _implementations_). But it has no more appeal on API level than has OGSA Base Notification, for example.
So, my proposal is:
- stick to the JD we have in SAGA (it works nicely with JSDL)
- support upcoming JSDL extensions in due time, either by new keys on the job description (JSDL scheduling), or programmatically (i.e. JSDL parameter sweeps)
- if there are use cases asking for it (only then), provide a jsdl_to_jd converter
Well, I realise I am repeating myself - not a good sign. We probably should give the topic anoter week of rants, and then do a straw poll on the mailing list, or continue to fight on a phone call... :)
Cheers, Andre.
Regards Hartmut
-----Original Message----- From: saga-rg-bounces@ogf.org [mailto:saga-rg-bounces@ogf.org] On Behalf Of Pascal Kleijer Sent: Sunday, February 18, 2007 8:51 PM To: SAGA RG Subject: Re: [SAGA-RG] Fwd (andre@merzky.net): Re: JSDL - SAGA
Dear all,
I have given me the week-end to get some reflection on the problem. As mentioned it is not a trivial issue. If things goes out of hand, we might as well rename the project CAGA of not go gaga :P
The necessity to support JSDL 1.0.x is a necessity, since it is an OFG standard. I do not think JSDL 1.0 is overly complex right now. The Job Description should be made such that passing from one form to another is automatic (JD -> JSDL). The support of a property key for JSDL isn't bad; if used, all other properties can be potentially ignored. The JD would not have to parse it. The current specification is simple enough to not violate the "S" rule of SAGA.
The problem we have is with all the up coming extensions. In my case, I clearly violated the SAGA API rules in order to make the system run with NAREGI. I tweaked the problem to solve my case. In the 80/20 rule, it probably means that the SAGA job submission won't be usable. I will have to write code that does not deal with SAGA and goes directly to the underwear. Now I am speaking as a Grid "expert" and not as a Grid newbie (which is the SAGA user base). If you want to use complex Grid programming and do complex things, SAGA might not be the right solution.
The proposal made by Dan is not elegant in my opinion because it will force developer to know the underlying technology they want to use; it will also clutter the JD. The proposal from Thilo might work for OO languages because you can hide a lot behind patterns, but it won't mean a specific implementation will support such scheme.
The passing through solution seems to be mildly seen by the some GAT people here. I would say no, if the error handling is explicit enough to return a message telling that this JSDL cannot be interpreted by the underlying system. I will not see this as a bad solution either.
--
Best regards, Pascal Kleijer
---------------------------------------------------------------- HPC Marketing Promotion Division, NEC Corporation 1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan. Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382
Another related idea is for SAGA to have one class for job descriptions with a quite loose ability to add "hints", such as in MPI I/O. The hints would not be required and might not be used, but in some cases would be used, depending on what SAGA is sitting on top of. Dan
Thilo Kielmann wrote (on 2/16/07 5:54 AM):
Folks,
I think we are having conflicting goals here. (Technical goals, not personal ones ;-)
On one hand, we have the "S for simplicity" in SAGA, and we must keep it. On the other hand, we have the necessity to support JSDL, future JSDL extensions, or any other types of job decscriptions that
(And still might to be invented.)
Assume, JSDL++ (whatever it will look like in near future) will become a widely adopted standard. (Or anything else, doesn't matter in the following.) Then, SAGA implementations will have to use JSDL++, and to form JSDL++ from SAGA job descriptions. Simultaneously, users are likely to use JSDL++ themselves, and may wnat to use JSDL++ to express their resource needs. At this point in time, SAGA will sit in the middle, and it may be very clumsy to first translate from JSDL++ to a SAGA job description, and then back to JSDL++ somewhere "down under" in the implementation.
For the very purpose of SAGA as a universal and simple grid API, it has to: - be independent of job description standards (mostly simpler than them) - support job description standards
My suggestion:
SAGA should have one class of job descriptions, and the
to create subclasses for more specific job descriptions (like JSDL). Such subclasses could be defined as separate extension documents (just like gridcpr). (Or was it "resource descriptions"???)
With this approach, users could still write programs that are independent of the underlying job submission machinery, having a simplified view on jobs and resource attributes etc. etc. At the same time, subclassed job/resource descriptions could be "passed through" transparently from the API to the implementeation, without being converted back and forth, a process that is very likely to loose some important details.
Is this a route to go?
Thilo
On Tue, Feb 13, 2007 at 06:28:58PM +0100, 'Andre Merzky' wrote:
From: 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> To: SAGA RG <saga-rg@ogf.org> <mailto:saga-rg@ogf.org> Subject: [SAGA-RG] Fwd (andre@merzky.net <mailto:andre@merzky.net>): Re: JSDL - SAGA
Hi group(s),
a couple of us had a recent discussion (f2f and email) about JSDL and SAGA. The question is: should we support JSDL fully on API level? E.g., should we allow the application programmer to specify/use JSDL documents for job creation?
The reasons for doing that are compelling: JSDL is one of
acknowledged standards in OGF, and the number of backends supporting JSDL is rapidly increasing it seems. Supporting JSDL directly would allow to interface with other tools using JSDL, and would allow to reuse JSDL documents where
already available.
Well, I have however some problems with that approach, which are outlined in the cited email below.
Do you guys have any other thoughts, and what solution would you prefer?
Cheers, Andre.
----- Forwarded message from 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> -----
> From: 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> > To: Shantenu Jha <sjha@cct.lsu.edu> <mailto:sjha@cct.lsu.edu> > Cc: Hartmut Kaiser <hkaiser@cct.lsu.edu> <mailto:hkaiser@cct.lsu.edu>, > 'Thilo Kielmann' <kielmann@cs.vu.nl> <mailto:kielmann@cs.vu.nl>, > 'Andre Merzky' <andre@merzky.net> <mailto:andre@merzky.net> > Subject: Re: JSDL - SAGA > > Hartmut and I discussed that somewhat last week. So he knows I am > not wholehartedly for that option. SAGA is supposed to abstract > the low level details, not to expose them. > > JSDL is going to define a number of extensions now. Some of these > extensions are very useful for us, others not. > Mostly, they will be more complex than JSDL itself. > > Are we going to support the extensions? Which? All/some? > How to select? What error do we report on unsupported extensions? > Do we mandate that extensions are supported by the backends? > Which? > > Even w/o extensions: is the job description updated after an JSDL > attrib is set? What about those attribs which are not JSDL keys? > Assume an implementation which implements the existing SAGA job > description keys: MUST it support complete JSDL now? What error > whould it report? > > These are probably all solvable problems, and I do agree
Daniel S. Katz wrote: people want to use. possibility the most these are that there
> are advantages, i.e. the re-use of existing JSDL documents. > Anyway, IMHO we should be careful, consider if we really have > enough use cases etc. Also, a free function > jsdl_to_job_description may do the trick, w/o complicating the job > package itself. > > Cheers, Andre. > > Another point I'd like to raise is: if HPC profile bekomes a widely accepted OGF standard, do we support it directly, too? Or OGSA-Workflow? Where to stop, and where is the 'S' in that approach?
Andre.
> Quoting [Shantenu Jha] > >> What little I know, I think so to. >> >> >> Quoting [Hartmut Kaiser] >> >>> Agree 100% >>> The easiest way is probably just to add a attribute in the job >>> description taking the whole JSDL. >>> >>> >> Quoting [Thilo Kielmann] >> >>> Yes! >>> >>> >>> Quoting [Shantenu Jha] >>> >>>> Shouldn't we ensure that SAGA consumes JSDL w/o any >>>> problem/changes? >>>> >>>> -- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org <mailto:saga-rg@ogf.org> http://www.ogf.org/mailman/listinfo/saga-rg
----------------------------------------------------------------------
--
-- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
All, On Tue, Feb 20, 2007 at 09:13:20AM +0900, Pascal Kleijer wrote:
I first sought about a solution like Hartmut, but SAGA is targeting Grid newbies, so JSDL might be out of the 80/20 rule scope.
may I slightly disagree? SAGA is targetting those application developers who want (or have) to focus on their own application domain. As Andre said, they are scientists, but not computer scientists. Newbies are part of this population, but more experienced people, too. For those people we must at least be open enough to let them use "real stuff" in the framework of SAGA. This is really important. I currently watch what is happening to the (Java) GAT: users liked it, and over time, they still like, but need "more". The current GAT approach is to keep adding features. With SAGA, however, we cannot do this once a standard has been nailed down. so we have to make SAGA open in a way that combines its main asset, the simplicity, with the ability to also support people who need to get more power at their hands. As I wrote last friday, these are conflicting goals. I am still convinced we should use SAGA's job descriptions and allow them to be subclassed, e.g. to be come JSDL job descriptions. (Non-OO languages will have to use type casting to deal with, but this is certainly not the only place where our OO design will look a bit non-natural in non-OO languages.) Also, Andre has a point w.r.t. OGSA. At OGF19 we had a discussion with Andrew Grimshaw after his talk about naming. This is another issue where SAGA needs to be "open": we must keep our simple approach to naming entities, but we must be able to support OGSA WS-Naming for those people who care. If we fail to provide this openness, then we fail substantially in providing an API for OGF-designed middleware. The idea we had at OGF19 was to have the SAGA spec on one hand, and a "profile" on the other, in which we describe how SAGA can/should be used in combination with OGSA-related middleware. Getting back to the JSDL issue: - the SAGA interface itself should stay clean from JSDL-specifics, unless they are so good that we like them to be used within "core SAGA" - we have to define a "hook" for JSDL-like job descriptions to be used along with SAGA (and not just in the backend implementation) Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/
participants (7)
-
'Andre Merzky' -
Andre Merzky
-
Daniel S. Katz -
Hartmut Kaiser -
Pascal Kleijer -
Steven Newhouse -
Thilo Kielmann