Re: [SAGA-RG] SAGA Resource Package Comments
Hi Steve, sorry, I thought I had answered this -- wishful thinking it seems.... But I also put it back on the list, hope that's ok with you... Your comments reflect, to some extent, the reservations Ole has: not focused enough, no clean separation of concerns. I understand what you mean, but I think we are missing the opportunity to make the *usage* of dynamically allocated resources simpler. About pilots: sure, if you don't have control over the pilots, a resource handle is not very useful (beyond inspection). But the same holds for all other resource types, too: if you are using a VM which was not created by you, you likely can't tear it down either, and the resource handle is near useless. In those cases, the resources look like the old job service. The resource API will only be really useful for those cases where you have control over the resources... Either way, thanks for the comments. This will likely mean that we'll trim down the resource API to simply perform - describe resource - acquire resource - release resurce - get job/data service for resource Best, Andre. On Wed, Jan 16, 2013 at 2:21 AM, Steve Fisher <dr.s.m.fisher@gmail.com> wrote:
I am surprised to have received no reaction to this!
Steve
On 2 January 2013 09:29, Steve Fisher <dr.s.m.fisher@gmail.com> wrote:
Hi,
I am just back after Christmas. After all this time my response is rather short.
I do not like the resource API as you have defined it. The problems are caused by trying to extend the job service. The task of allocating resources is just another service. The resource after allocation may itself be a job service but it will look just like any other job service and not know whether it has been dynamically created by a cloud, be on a virtual machine or a physical machine.
I would leave the job stuff alone and think about an API which is open-nebula or OCCI inspired. I find your proposal unnecessarily complex and it is mixing concerns.
I am not sure where pilot jobs come in. For EGI/EGEE a pilot job starts and the job cannot be manipulated by the user so it requires no API. The pilot job just looks after the running of a sequence of other jobs. In this model the pilot job is owned by the experiment and looks after running others users jobs. I have no experience of the way that pilot jobs have evolved. It is possible that if we want to accommodate them in SAGA that another extension might be needed, however I would need to understand better the way in which the pilot job notion has moven on and perhaps diversified.
Steve
On 18 December 2012 15:19, Andre Merzky <andre@merzky.net> wrote:
Sorry, I should have attached both... - here is the package proper...
Andre.
On Tue, Dec 18, 2012 at 2:31 PM, Steve Fisher <dr.s.m.fisher@gmail.com> wrote:
So the discussion just relates to the Python bindings for resources. I thought it was a more general resource package matter as an update to what was presented at OGF 29?
Steve
On 18 December 2012 13:00, Andre Merzky <andre@merzky.net> wrote:
Welcome to the future! :-) document is attached.
Best, Andre.
PS.: the create() methods need to get moved outside of the classes, I'll fix that....
On Tue, Dec 18, 2012 at 1:54 PM, Steve Fisher <dr.s.m.fisher@gmail.com> wrote:
"tomorrow" has finally arrived. Could you send me a copy of the document please.
Steve
On 13 December 2012 01:09, Steve Fisher <dr.s.m.fisher@gmail.com> wrote: > > OK. I will take a look tomorrow > Steve > > On 12 Dec 2012 21:35, "Ole Weidner" <ole.weidner@rutgers.edu> > wrote: >> >> Hi Steve, >> >> would you be willing to provide us with some input with regards to >> the >> proposed saga resource package? Unfortunately, Andre and I >> currently >> are in >> somewhat of a deadlock situation -- see our discussion below ;-) >> We >> haven't >> been very successful at getting any external input / opinion so >> far. >> >> We are planning to implement a first prototype of the resource >> package >> for saga-python in the next two or three days, so it would be good >> to >> have >> at least some consensus of how it should look like. >> >> So if you would like to share your opinion with us, that would be >> much >> appreciated! >> >> Many thanks! >> Ole >> >> On Dec 8, 2012, at 04:07 , Andre Merzky <andre@merzky.net> wrote: >> >> > Hi Ole, >> > >> > >> > On Tue, Nov 13, 2012 at 4:54 PM, Ole Weidner >> > <ole.weidner@rutgers.edu> >> > wrote: >> >> Hi Andre, >> >> >> >> here's some quick comments w.r.t. the resource package. It's >> >> more >> >> for >> >> the records - we can talk about this in detail when we discuss >> >> the >> >> resource >> >> package on the 28th (journal club). >> > >> > Sorry this took so long, I am trying to catch up with things >> > now... >> > >> > >> >> I fundamentally disagree that 'Compute' inherits from >> >> 'saga.job.Service' and that 'Storage' inherits from >> >> saga.filesystem.Directory' because this: >> >> >> >> 1. breaks SAGA's horizontally independent package model: it >> >> would >> >> mean >> >> that I can't implement a resource-package only implementation >> >> of >> >> SAGA. I >> >> would have to implement the Job and the Filesystem package as >> >> well! >> > >> > A compute resource is useless if you can't submit jobs, a >> > storage >> > resource is useless if you can't store files. That does not >> > change >> > if >> > you replace inheritance with get_job_service / get_filesystem -- >> > in >> > both cases, you will need to implement the job package / file >> > package, >> > too. >> > >> > While we indeed try to avoid too many cross dependencies between >> > functional API packages, we do have them in some places, most >> > notably >> > for the namespace derivates. >> > >> > FWIW, another reason why compute resource inherits from >> > job.service >> > is >> > that we intented to fix some shortcomings of the job service, in >> > particular wanted to add the ability to directly submit JSDL. >> > Inheritance provides a very simple means to do so. I agree >> > though >> > that this should not be the foremost concern for API design - >> > but >> > anyway. >> > >> > Another point though I want to make: I don't like the idea to >> > have a >> > job service, which is not stateful, depending on the state of a >> > compute resource (and same for filesystem / storage resource) -- >> > on >> > API level, there are no means to infer if the job service is >> > valid >> > for >> > job submission at any point in time (you can't get a resource >> > handle >> > from a job service instance) - so it always boils down to >> > try/error. >> > We so far managed to avoid those implicit state dependencies, >> > and I >> > would like to keep it this way. [Yes, a decoupling maps better >> > to >> > the >> > Pilot API, but I would rather like to fix that in the Pilot API >> > ;-)] >> > >> > Don't get me wrong: I understand that inheritance is a pretty >> > strong >> > coupling, and it does not necessarily reflect how DCIs are >> > architected >> > internally -- from the end user perspective though, I find this >> > rendering simple, intuitive, and easy to use... >> > >> > >> >> 2. it mixes separate concerns: resource management and job >> >> submission! >> > >> > I kind of agree, but think that this is set off by ease of use: >> > get >> > a >> > compute resource, submit jobs to it - bang. This is, by far, >> > the >> > dominant use case, so I would like to see this rendered >> > exceedingly >> > simple. >> > >> > >> >> 3. I don't think that 'Directory' necessarily provides the >> >> right >> >> abstraction for all 'Storage' types. Certainly for most, but >> >> not >> >> for all. >> >> It's unnecessarily confining. >> > >> > Yes, that is a limitation -- but unless we have a compelling use >> > case >> > for other storage abstractions, and those use cases do not imply >> > an >> > overly complicated approach to storage resources, that is the >> > best >> > abstraction we have, right? Even if the backend storage >> > resources >> > have a limited / constraint namespace (think Amazon S3), the >> > filesystem abstraction still holds up nicely IMHO. Also, I am >> > not >> > concerned about provisioning of databases etc. -- we don't have >> > decent >> > (or any) abstractions for those in SAGA, nor do we have use >> > cases >> > that >> > I know of -- so that would be out of scope for now. >> > >> > >> >> Furthermore, >> >> >> >> - class manager -> Manager >> > >> > fixed, thanks. >> > >> > >> >> - what does manager.describe_resource() do? why can't it be >> >> manager.resources[x].get_description() >> > >> > Hmm, probably right - but while that works nicely in python, you >> > would >> > have >> > >> > manager.get_resource (id).get_description () >> > >> > and chaining is something we do not promote in the API so far. >> > Thus, >> > I would like to keep the method in the API, but I agree that >> > your >> > version is (in Python) the more intuitive one. >> > >> > >> >> - speaking of resources[x] - there's no 'non-property' version, >> >> i.e., >> >> get_resource() >> > >> > thanks, I'll fix that. >> > >> > >> >> - I would prefer explicit list/get_compute(), >> >> list/get_storage(), >> >> list/get_network() and so on, so that we don't have to do type >> >> checking all >> >> over the place. >> > >> > The list / get calls have a 'type' parameter, so you can filter >> > for >> > specific resource types: >> > >> > compute_resources = manager.list_resources >> > (saga.resource.Compute) >> > storage_resources = manager.list_resources >> > (saga.resource.Storage) >> > >> > The default is 'Any' though, which obviously gives you all >> > types. >> > >> > >> >> - why are there two Pool.add() methods? Why do we want to be >> >> able >> >> to >> >> add resources as strings? >> > >> > Alas, we have that in a few places in the API. There was a very >> > long >> > discussion, a long time ago, where people argued that only using >> > handles would have too much of a performance impact (you'd >> > always >> > need >> > to create handles, which is *at least* one round-trip), and that >> > only >> > using IDs would be too unwieldy to handle in many cases. While >> > I >> > agree with the first point, I do not think that the second one >> > is >> > very >> > valid. That is one item I would like to clean up across SAGA in >> > an >> > eventual API revision (if that ever happens). So, for now that >> > is >> > in >> > the resource API as well, for consistency, but I personally do >> > not >> > care much about it. If we limit that, then I would be in favor >> > of >> > the >> > id version. >> > >> > >> > Cheers, Andre. >> > >> > >> >> Cheers! >> >> Ole >> > >> > >> > -- >> > Nothing is really difficult... >>
-- Nothing is really difficult...
-- Nothing is really difficult...
-- There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton
All, On Jan 16, 2013, at 09:14 , Andre Merzky <andre@merzky.net> wrote:
Hi Steve,
sorry, I thought I had answered this -- wishful thinking it seems.... But I also put it back on the list, hope that's ok with you...
Your comments reflect, to some extent, the reservations Ole has: not focused enough, no clean separation of concerns.
Correct. Steve's and my concerns are pretty much the same.
I understand what you mean, but I think we are missing the opportunity to make the *usage* of dynamically allocated resources simpler.
Not sure if this can or should really be accomplished in one single SAGA package.
About pilots: sure, if you don't have control over the pilots, a resource handle is not very useful (beyond inspection). But the same holds for all other resource types, too: if you are using a VM which was not created by you, you likely can't tear it down either, and the resource handle is near useless. In those cases, the resources look like the old job service. The resource API will only be really useful for those cases where you have control over the resources...
Either way, thanks for the comments. This will likely mean that we'll trim down the resource API to simply perform
- describe resource - acquire resource - release resurce - get job/data service for resource
+1
Best, Andre.
Cheers Ole
On Wed, Jan 16, 2013 at 2:21 AM, Steve Fisher <dr.s.m.fisher@gmail.com> wrote:
I am surprised to have received no reaction to this!
Steve
On 2 January 2013 09:29, Steve Fisher <dr.s.m.fisher@gmail.com> wrote:
Hi,
I am just back after Christmas. After all this time my response is rather short.
I do not like the resource API as you have defined it. The problems are caused by trying to extend the job service. The task of allocating resources is just another service. The resource after allocation may itself be a job service but it will look just like any other job service and not know whether it has been dynamically created by a cloud, be on a virtual machine or a physical machine.
I would leave the job stuff alone and think about an API which is open-nebula or OCCI inspired. I find your proposal unnecessarily complex and it is mixing concerns.
I am not sure where pilot jobs come in. For EGI/EGEE a pilot job starts and the job cannot be manipulated by the user so it requires no API. The pilot job just looks after the running of a sequence of other jobs. In this model the pilot job is owned by the experiment and looks after running others users jobs. I have no experience of the way that pilot jobs have evolved. It is possible that if we want to accommodate them in SAGA that another extension might be needed, however I would need to understand better the way in which the pilot job notion has moven on and perhaps diversified.
Steve
On 18 December 2012 15:19, Andre Merzky <andre@merzky.net> wrote:
Sorry, I should have attached both... - here is the package proper...
Andre.
On Tue, Dec 18, 2012 at 2:31 PM, Steve Fisher <dr.s.m.fisher@gmail.com> wrote:
So the discussion just relates to the Python bindings for resources. I thought it was a more general resource package matter as an update to what was presented at OGF 29?
Steve
On 18 December 2012 13:00, Andre Merzky <andre@merzky.net> wrote:
Welcome to the future! :-) document is attached.
Best, Andre.
PS.: the create() methods need to get moved outside of the classes, I'll fix that....
On Tue, Dec 18, 2012 at 1:54 PM, Steve Fisher <dr.s.m.fisher@gmail.com> wrote: > "tomorrow" has finally arrived. Could you send me a copy of the > document > please. > > Steve > > > On 13 December 2012 01:09, Steve Fisher <dr.s.m.fisher@gmail.com> > wrote: >> >> OK. I will take a look tomorrow >> Steve >> >> On 12 Dec 2012 21:35, "Ole Weidner" <ole.weidner@rutgers.edu> >> wrote: >>> >>> Hi Steve, >>> >>> would you be willing to provide us with some input with regards to >>> the >>> proposed saga resource package? Unfortunately, Andre and I >>> currently >>> are in >>> somewhat of a deadlock situation -- see our discussion below ;-) >>> We >>> haven't >>> been very successful at getting any external input / opinion so >>> far. >>> >>> We are planning to implement a first prototype of the resource >>> package >>> for saga-python in the next two or three days, so it would be good >>> to >>> have >>> at least some consensus of how it should look like. >>> >>> So if you would like to share your opinion with us, that would be >>> much >>> appreciated! >>> >>> Many thanks! >>> Ole >>> >>> On Dec 8, 2012, at 04:07 , Andre Merzky <andre@merzky.net> wrote: >>> >>>> Hi Ole, >>>> >>>> >>>> On Tue, Nov 13, 2012 at 4:54 PM, Ole Weidner >>>> <ole.weidner@rutgers.edu> >>>> wrote: >>>>> Hi Andre, >>>>> >>>>> here's some quick comments w.r.t. the resource package. It's >>>>> more >>>>> for >>>>> the records - we can talk about this in detail when we discuss >>>>> the >>>>> resource >>>>> package on the 28th (journal club). >>>> >>>> Sorry this took so long, I am trying to catch up with things >>>> now... >>>> >>>> >>>>> I fundamentally disagree that 'Compute' inherits from >>>>> 'saga.job.Service' and that 'Storage' inherits from >>>>> saga.filesystem.Directory' because this: >>>>> >>>>> 1. breaks SAGA's horizontally independent package model: it >>>>> would >>>>> mean >>>>> that I can't implement a resource-package only implementation >>>>> of >>>>> SAGA. I >>>>> would have to implement the Job and the Filesystem package as >>>>> well! >>>> >>>> A compute resource is useless if you can't submit jobs, a >>>> storage >>>> resource is useless if you can't store files. That does not >>>> change >>>> if >>>> you replace inheritance with get_job_service / get_filesystem -- >>>> in >>>> both cases, you will need to implement the job package / file >>>> package, >>>> too. >>>> >>>> While we indeed try to avoid too many cross dependencies between >>>> functional API packages, we do have them in some places, most >>>> notably >>>> for the namespace derivates. >>>> >>>> FWIW, another reason why compute resource inherits from >>>> job.service >>>> is >>>> that we intented to fix some shortcomings of the job service, in >>>> particular wanted to add the ability to directly submit JSDL. >>>> Inheritance provides a very simple means to do so. I agree >>>> though >>>> that this should not be the foremost concern for API design - >>>> but >>>> anyway. >>>> >>>> Another point though I want to make: I don't like the idea to >>>> have a >>>> job service, which is not stateful, depending on the state of a >>>> compute resource (and same for filesystem / storage resource) -- >>>> on >>>> API level, there are no means to infer if the job service is >>>> valid >>>> for >>>> job submission at any point in time (you can't get a resource >>>> handle >>>> from a job service instance) - so it always boils down to >>>> try/error. >>>> We so far managed to avoid those implicit state dependencies, >>>> and I >>>> would like to keep it this way. [Yes, a decoupling maps better >>>> to >>>> the >>>> Pilot API, but I would rather like to fix that in the Pilot API >>>> ;-)] >>>> >>>> Don't get me wrong: I understand that inheritance is a pretty >>>> strong >>>> coupling, and it does not necessarily reflect how DCIs are >>>> architected >>>> internally -- from the end user perspective though, I find this >>>> rendering simple, intuitive, and easy to use... >>>> >>>> >>>>> 2. it mixes separate concerns: resource management and job >>>>> submission! >>>> >>>> I kind of agree, but think that this is set off by ease of use: >>>> get >>>> a >>>> compute resource, submit jobs to it - bang. This is, by far, >>>> the >>>> dominant use case, so I would like to see this rendered >>>> exceedingly >>>> simple. >>>> >>>> >>>>> 3. I don't think that 'Directory' necessarily provides the >>>>> right >>>>> abstraction for all 'Storage' types. Certainly for most, but >>>>> not >>>>> for all. >>>>> It's unnecessarily confining. >>>> >>>> Yes, that is a limitation -- but unless we have a compelling use >>>> case >>>> for other storage abstractions, and those use cases do not imply >>>> an >>>> overly complicated approach to storage resources, that is the >>>> best >>>> abstraction we have, right? Even if the backend storage >>>> resources >>>> have a limited / constraint namespace (think Amazon S3), the >>>> filesystem abstraction still holds up nicely IMHO. Also, I am >>>> not >>>> concerned about provisioning of databases etc. -- we don't have >>>> decent >>>> (or any) abstractions for those in SAGA, nor do we have use >>>> cases >>>> that >>>> I know of -- so that would be out of scope for now. >>>> >>>> >>>>> Furthermore, >>>>> >>>>> - class manager -> Manager >>>> >>>> fixed, thanks. >>>> >>>> >>>>> - what does manager.describe_resource() do? why can't it be >>>>> manager.resources[x].get_description() >>>> >>>> Hmm, probably right - but while that works nicely in python, you >>>> would >>>> have >>>> >>>> manager.get_resource (id).get_description () >>>> >>>> and chaining is something we do not promote in the API so far. >>>> Thus, >>>> I would like to keep the method in the API, but I agree that >>>> your >>>> version is (in Python) the more intuitive one. >>>> >>>> >>>>> - speaking of resources[x] - there's no 'non-property' version, >>>>> i.e., >>>>> get_resource() >>>> >>>> thanks, I'll fix that. >>>> >>>> >>>>> - I would prefer explicit list/get_compute(), >>>>> list/get_storage(), >>>>> list/get_network() and so on, so that we don't have to do type >>>>> checking all >>>>> over the place. >>>> >>>> The list / get calls have a 'type' parameter, so you can filter >>>> for >>>> specific resource types: >>>> >>>> compute_resources = manager.list_resources >>>> (saga.resource.Compute) >>>> storage_resources = manager.list_resources >>>> (saga.resource.Storage) >>>> >>>> The default is 'Any' though, which obviously gives you all >>>> types. >>>> >>>> >>>>> - why are there two Pool.add() methods? Why do we want to be >>>>> able >>>>> to >>>>> add resources as strings? >>>> >>>> Alas, we have that in a few places in the API. There was a very >>>> long >>>> discussion, a long time ago, where people argued that only using >>>> handles would have too much of a performance impact (you'd >>>> always >>>> need >>>> to create handles, which is *at least* one round-trip), and that >>>> only >>>> using IDs would be too unwieldy to handle in many cases. While >>>> I >>>> agree with the first point, I do not think that the second one >>>> is >>>> very >>>> valid. That is one item I would like to clean up across SAGA in >>>> an >>>> eventual API revision (if that ever happens). So, for now that >>>> is >>>> in >>>> the resource API as well, for consistency, but I personally do >>>> not >>>> care much about it. If we limit that, then I would be in favor >>>> of >>>> the >>>> id version. >>>> >>>> >>>> Cheers, Andre. >>>> >>>> >>>>> Cheers! >>>>> Ole >>>> >>>> >>>> -- >>>> Nothing is really difficult... >>> >
-- Nothing is really difficult...
-- Nothing is really difficult...
-- There are only two hard things in Computer Science: cache invalidation and naming things.
-- Phil Karlton
On 16 January 2013 09:14, Andre Merzky <andre@merzky.net> wrote:
Hi Steve,
Either way, thanks for the comments. This will likely mean that we'll trim down the resource API to simply perform
- describe resource - acquire resource - release resurce - get job/data service for resource
Best, Andre.
Andre, I am happy with the first three operations - the last is less nice as not all resources will be able to support, for example, getJobService(). Better perhaps would be getService() where the Service is the same Service class that SD provides. This would support services of any type. Steve
Hi Steve, On Wed, Jan 16, 2013 at 12:06 PM, Steve Fisher <dr.s.m.fisher@gmail.com> wrote:
On 16 January 2013 09:14, Andre Merzky <andre@merzky.net> wrote:
Hi Steve,
Either way, thanks for the comments. This will likely mean that we'll trim down the resource API to simply perform
- describe resource - acquire resource - release resurce - get job/data service for resource
Best, Andre.
Andre,
I am happy with the first three operations - the last is less nice as not all resources will be able to support, for example, getJobService(). Better perhaps would be getService() where the Service is the same Service class that SD provides. This would support services of any type.
Steve
Well, we do have different resource types (*), and so far I don't see a reason nor argument to change that. So, for a compute resource, I would expect to always be able to retrieve a job service endpoint, for a StorageResource, I would expect to get a filesystem or replica endpoint. Also please consider that the main use case for the resource package is basically: - create a vm - run a couple of jobs on the VM Originally, that was supposed to look like: vm = resource_manager.aquire_resource (resource_description) vm.create_job (job_description) As we now want to decouple the resource and job parts, this will likely look like: vm = resource_manager.aquire_resource (resource_description) js = vm.get_job_service () js.create_job (job_description) Yes, adding a generic service inspection / retrieval to the resource makes the package more generic, obviously -- but it will make the main use cases more complex, too (error checks left out): vm = resource_manager.aquire_resource (resource_description) sds = vm.get_service_descriptions () js = None for sd in sds : if sd.type == 'saga.job.service' : js = saga.job.service (sd.url) break js.create_job (job_description) or, if you allow the filters on the sd retrieval: vm = resource_manager.aquire_resource (resource_description) sds = vm.get_service_descriptions (service_filter='Capabilities="org.ogf.saga.service.job"') js = saga.job.service (sds[0].url) js.create_job (job_description) I rather see the role for SD/ISN to be used for inspection purposes, to understand / inform the application about the properties of the resources (number or CPUs, load, lifetime, ...) -- we do have dynamic scheduling use cases which would need that kind of information about the resources, and I'd rather expose those data via the existing SD/ISN packages that getting them out-of-band... My $0.02, Andre. (*) although in the reduced resource package scope, we don't have any use for network resources anymore. -- There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton
Why not? I should be able to use a "network resource" to manage my bandwidth reservations / allocations, right? Thanks, Ole On Jan 16, 2013, at 11:51 , Andre Merzky <andre@merzky.net> wrote:
(*) although in the reduced resource package scope, we don't have any use for network resources anymore.
On Wed, Jan 16, 2013 at 1:01 PM, Ole Weidner <ole.weidner@rutgers.edu> wrote:
Why not? I should be able to use a "network resource" to manage my bandwidth reservations / allocations, right?
A network resource is only useful if you can specify what resources it connects -- that is one of the uses we had for the resource pool, which I think won't exist anymore in the next version. So, how would you specify connectivity? Cheers, Andre.
Thanks, Ole
On Jan 16, 2013, at 11:51 , Andre Merzky <andre@merzky.net> wrote:
(*) although in the reduced resource package scope, we don't have any use for network resources anymore.
-- There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton
participants (3)
-
Andre Merzky -
Ole Weidner -
Steve Fisher