May 2006 - ogsa-bes-wg - lists.ogf.org

Updates to document
by Andrew Grimshaw 28 May '06

28 May '06

All, I have done the updates to the document discussed in the last two phone calls, and in the EMS meeting in Japan. At a high level this includes modifications to adopt text/issues from the ESI document and the HPC profile working document. Note that this is NOT a final document. These changes revolve around: 1: Changing CreateActivityFromJSDL to a) take additional optional arguments, specifically notification arguments and an "idempotent" argument lifted from ESI. b) An extension to JSDL suggested in ESI to support libraries 2: Adding a resource information model section. Originally to be from 4.1 of the ESI document - but post discussion with Snelling, Stokes, et al a modified 4.1. 3: State model. This is one of the biggest jobs. At GGF in Japan the HPC profile group introduced a nice, simple, extensible state model that would allow supporting a variety of state machines while still allowing basic clients to understand what they were getting. After discussions with many people I have decided to adopt that simpler state model. CLEARLY THIS WILL NEED DISCUSSION BY INTERESTED PARTIES. I AM NOT TRYING TO PULL A FAST ONE. I have taken the text verbatim from the HPC profile paper. Note that I do not yet have permission from their lawyers - so we may need to yank this and retype the same info. I hope not. 4: Refer to managedjob "type" of EPR, just as we refer to "WS-Name" "type" of EPR. I have not even tried to modify the WSDL and renderings. Once we decide on the rest of the content we can do that. So don't read past page 20! I am not sure of the status of the weekly call this week. I will synch with Darren and Steven and send mail tomorrow. I'm also including some slides from the discussion of BES in the EMS session. A Andrew Grimshaw Professor of Computer Science University of Virginia 434-982-2204 grimshaw(a)cs.virginia.edu

3 4

Additional Input into BES w.r.t ESI document
by Steven Newhouse 24 May '06

24 May '06

All, From the OMII-Europe project (which is engaging with GGF in implementing grid standards) here are some comments from one of the teams that will be looking to implement the BES specification. Please respond to the following comments. The attached document has comments on the proposed ESI state model and information/resource model. Steven About the Job Factory Interface: * The list of the defined operations does not cover administrative operations like “rejectJobSubmissions(Policy)” and “allowJobSubmission (Policy)” useful for disabling/enabling new job submissions based on the policy defined by the CE administrator (e.g. if the CE has to be shutdown for maintenance; disable new submissions if the number of active jobs is > 3000; etc). Do you plan to provide it? * The current proposed JobFactory interface allows users to create new jobs AND (optionally) subscribe for notifications. We believe that the job management service should be better decoupled from the notification service, as they provide different functionalities. We suggest that Figure 1 be extended with a new box (“SubscriptionFactory”) which exposes an interface for creating, modifying and removing notification requests. In this way, notification management can be decoupled from job management, allowing a greater degree of flexibility. For example, it would be possible for users to subscribe to notifications after a job has been created (with the current proposed interface, this would not be possible). Moreover, it would allow users to submit to the notification service requests for “cumulative” subscriptions (i.e., in order to receive notifications related to all jobs submitted by the same user, or by members of the same Virtual Organization). * We propose to include a “JobAssess” operation on the JobFactory Interface. This operation provides the user with an estimate of the start running time, e.g. taking into account the current state of the Computing Element, the number of running/queued jobs and other parameters. * Related with the previous issue, it would be interesting to include an additional operation for estimating the Quality of Service (QoS) for the service instance. The exact meaning of QoS in general depends on user requirements (users may assume different weights for different parameters). * This is probably a “cosmetic” adjustment, but we believe that the name of the “Release” operation makes sense only for reactivating jobs from “Held” states; the transition from “Start Pending” and “Staging In” could probably be called something like “Activate”. * The proposed interface does not provide mechanisms for handling capabilities. With this we refer to the possibility for a user to authorize other user(s) to perform certain operations on his/her jobs. For example, a user may want to allow another user to monitor his/her jobs, or to interrupt and abort jobs and so on. Perhaps this functionality is not strictly related to job management, but is rather a security issue (which, according to the draft, is still to be discussed). We may keep it for future discussions. About the staging of files: * While it is clear that users might explicitly “push” files from their storage space to the Grid while a job is in “Start Pending” state, it is unclear how users might explicitly “pull” by hand resulting files from the Grid after a job competed, and before everything gets cleaned up. About the Job Interface: * The description of the states of Fig. 2 should be expanded with more details, including details on state transitions (basically, we suggest to put a complete description of the states in section 3.2.1). * Section 5.1: The meaning of the “Log” property on Table 3 is not clear: what does it mean? * From Table 3 we see a JobState property which represents the current state of the job. We think that it would be useful to provide the user with the history of all job status changes with the associated timestamp. We also support the need for exitCode and failureReason attributes (see Section 8.2, issue 9) to describe the job return code and job failure reason respectively. * It may be useful to provide an additional property (we may call it “CommandList”) representing the list of all commands issued for a given job. About the Application Interface: * The status of this interface is unclear: is this section going to be discussed? Are you going to consider the problem of user interaction with running jobs? -- Moreno Marzolla INFN Sezione di Padova, via Marzolo 8, 35100 PADOVA, Italy EMail: moreno.marzolla(a)pd.infn.it Phone: +39 049 8277047 WWW : http://www.pd.infn.it/~marzolla Fax : +39 049 8756233

4 6

Re: [saga-rg] Re: [ogsa-bes-wg] Extensible state model for jobs
by Christopher Smith 04 May '06

04 May '06

I agree ... we can discuss this next week. -- Chris On 03/5/06 10:09, "Thilo Kielmann" <kielmann(a)cs.vu.nl> wrote: > Chris and Andre, > > I think we really should resolve this issue face-to-face next week at GGF17. > That will save a lot of email typing time... > > Andre, do we have a slot in one of our sessions? > > > Thilo > > On Wed, May 03, 2006 at 06:08:46PM +0200, Andre Merzky wrote: >> X-Original-To: kielmann@localhost >> Delivered-To: kielmann(a)localhost.cs.vu.nl >> Delivered-To: grdfm-saga-rg-outgoing(a)mailbouncer.mcs.anl.gov >> X-Original-To: grdfm-saga-rg(a)mailbouncer.mcs.anl.gov >> Delivered-To: grdfm-saga-rg(a)mailbouncer.mcs.anl.gov >> Date: Wed, 3 May 2006 18:08:46 +0200 >> From: Andre Merzky <andre(a)merzky.net> >> To: Christopher Smith <csmith(a)platform.com> >> Cc: Andre Merzky <andre(a)merzky.net>, >> "ogsa-wg(a)ggf.org" <ogsa-wg(a)ggf.org>, ogsa-bes-wg(a)ggf.org, >> SAGA RG <saga-rg(a)ggf.org> >> Subject: [saga-rg] Re: [ogsa-bes-wg] Extensible state model for jobs >> >> Quoting [Christopher Smith] (May 03 2006): >>> >>> On 03/5/06 05:36, "Andre Merzky" <andre(a)merzky.net> wrote: >>> >>>> Right now its absent because only those states which can be >>>> actually reached by SAGA method calls are present. Now, as >>>> you said earlier, it can be assumed a flaw that SAGA has no >>>> means to put a job into 'Pending' state. >>>> >>> >>>> So, if there are some use cases or some agreement that the >>>> API should allow to move a job into 'Pending' state, the >>>> method/attribute will be added, and the state will be added. >>>> >>> The state that a job enters after a create_job operation is 'pending', so >>> the create_job operation is the method that puts a job in this state. >> >> Right, I understand that from the queuing point I think. >> However, from the application level it does not matter, as >> it cannot actively change that state. We have no notion of >> queue states at all. >> >> Well, let me put it differently: what do we gain by moving >> the state to the higher level? It complicates the state >> diagram (not much, that is not the point), but does not >> allow to perform any new state transition from application >> level. >> >> As said, I would compare this to Suspended state: its >> clearly an important state in many systems, but as long as >> the API does not expose means to suspend/resume a job, the >> state is meaningless, and merily informative (e.g. the job >> gut suspended by a 3rd party, or by the backend). That >> information is exposed however, through the job state >> details - same holds for hols (ha!) I think. >> >> >>> This state is required for queuing systems ... I can't even imagine how it >>> got dropped from the state diagram. I think you're trying to map the task >>> state model too closely. ;-) >> >> Nono, that has nothing to do with the task states - they >> could easily leave that state out. Well, its very >> convenient to have the same model here, thats for sure ;-) >> >> >>>> Similar reasoning holds for 'Suspend' - as long as we don't >>>> have suspend/resume methods, we would not like to expose a >>>> 'Suspended' state on the higher level state diagram. >>> >>> This I agree with. >> >> As said, it might well be that SAGA is broken in the respect >> that we doesn't allow job.hold (), or submit into hold. >> Well, it wasn't in the use-cases ... :-P >> >> Cheers, Andre. >> >> >>> -- Chris >> >> >> >> -- >> "So much time, so little to do..." -- Garfield >> > >

2 2

Extensible state model for jobs
by Christopher Smith 03 May '06

03 May '06

Hi all, As Marvin mentioned in his email on extensibility and composition of protocols, I'd like to present this short document describing, in more detail, a model for extending the state model of jobs from a "base profile". The document is based on some discussions that Marvin and I have been having about scheduler interoperability, and is very close to the model that has been proposed within the SAGA working group. Please review Marvin's email about extensibility to get a feel for the "context" of the model. It is my intention to show how the ESI and BES state models can be represented using this approach. I'll try to get that done before Tokyo, but might just need to present some slides at GGF17, assuming I'm given a few minutes to do so. This is very much a work in progress, so I'm looking forward to feedback.... A note for the SAGA group .... I'd still like to see "pending" in the base job state diagram ... don't know why it disappeared. ;-) -- Chris

2 3

Meeting Minutes for Wednesday April 26, 2006
by darren＠pulsipher.org 03 May '06

03 May '06

Here are the minutes from today? Next meeting is the same time same place.

2 1