July 2005 - saga-rg - lists.ogf.org

Jobs and SandBoxing
by Andre Merzky 25 Aug '05

25 Aug '05

Hallihallo! [For the attentive reader: right, the subject of the 'other' mail should have been "Jobs and GetState", not "Tasks and GetState" - sorry] Another comment I got about jobs in SAGA is: how is sandboxing supported? Can I at least determine if my job runs in a sandbox? Or at least what it's 'cwd' is? Does a job have a unique job ID I can use to identify it? (That question is related to the session persistency discussed in another thread I think). Any opinions? Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

3 10

Short Strawman versions
by Andre Merzky 04 Aug '05

04 Aug '05

Hi, to disable the excuse 'too long' for not reading the strawman API spec, we created a short version of the spec. It does not contain use case relation, notes, examples, and the detailed deiscussion of the API objects and methods, but only - intro etc - rationale for each subsystem - SIDL for each subsystem You can find the short version as pdf and txt on the WIKI: http://wiki.cct.lsu.edu/saga/ check the last links for the v0.2 version of the doc on the front page. However, please check the complete doc on any questions before complaining on the list, in particular when in doubt about call semantics. Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

4 8

Job States
by Andre Merzky 04 Aug '05

04 Aug '05

Hiho. SAGA Jobs have currently following states: - Unknown, - HoldSystem, - HoldUser, - Hold, - Queued, - Running, - SuspendSystem, - SuspendUser, - Suspend, - DoneOk, - DoneFail I got the comment from colleques that PreStaging and PostStaging are missing. Indeed these stages seem not to fir into any of the above ones. Running would be a candidate, but since the remote resource is not neccessarily used anymore, that might be confusing. Should these stages be added? However, they do also not appear in the DRMAA specification AFAIK. Any thoughts? Another question: Assume I check a job status and find it 'DoneFail' - how can I determine the reason of failure? It would be useful to know the status the job was in before it failed (e.g. if it was prestating, I know then that staging failed, and the job never really started). Also it would be nice to be able to query for any error message. I think that the error query is distinct from the exception mechanism we will have: a job entering DoneFail should NOT throw an exception in my opinion - but that leads to above question: how can I query the error leading ot the DoneFail state? BTW: We should also check/sync with the error reporting mechanism for Tasks. Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

2 2

GGF meeting notes
by Andre Merzky 03 Aug '05

03 Aug '05

Hi all, due to personal unexcusable stupidity, I removed parts of the session notes we took at GGF14. However, here is finally the small amount of notes we could recover, and which have been sent to me by others. If anybody has material/notes to add, please do so. Cheers, Andre. Session 1: ========== - discussion group org issues - Keith resigned as co chair, only Tom left - found new co-chairs (Andre, Shantenu) - instruct users how to present a use case - how to get application programmers to use SAGA? - feedback: should implementors focus on getting the entire implementation done, or just part of it? - feedback: agenda should be more detailed, this way other orgs will be involved - discussion of strawman API - first 3 priorities: - jobs - files - streaming - design choice: oo model - jobs: - JobDefinition may changed to even better match JSDL - Q: but would this violate the name of the group? - A: it could be still simple with setters/getters - files, logical files and namespaces pb: how to make a difference in the return type of files ops between logical files and just files connecting processes (streams) - follows the model of java-like factories Q: asynchronicity - tasks again java-like factories polling vs publish/subscribe - future issue: have task dependencies to provide API sections for simple workflows - again the issue discussed at DRMAA-1: session & job ids related to sessions -> security contexts - so far, gat-like approach: allow different types of security primitives (certificates, user/passwd, kerberos tickets) - sessions would also be defined by security context they are run in. - Comment: should add MyProxy to the other types of security. - Initial implementation experience - constraints: - ability to add new subsystems (SAGA is evolving) (write new adaptors) - adaptors - flexible adaptor scheduling - late / lazy binding - capability provider interface for one "object": there are 2 objs in the engine: - one is seen by the application - one is seen by the adaptor - the engine is in charge of scheduling the methods of that object - there are reasons not to keep everything hidden (in the sense of making the distributed app look local) Session 2: ========== - continued discussion of strawman - consistency: - all want, but most frequent is not worry... (80:20) - advisory locking helps - do good error reporting! - see NFS paper circulated by Jon McLaren on the list! - all error codes we can possibly think of, but no gtuarantee beyond that people need to check errors. - Atomicity not required everywhere, but makes much sense on constructors/destructors. Evt. move etc? - when is implementation conform? implement all? - better to have non conformacy errors earlier (compiler!) - but full implementation is preferred by apps - try to provide fallbacks of complex to simple stuff - do NOT allow NOTIMPL exception if not absolutely inavoidable - files: - eread, readv, pread: consensus: put all 3 versions as discussed into the draft, and match against use cases - fix flags! - R/W/RW flags for open... - need to be able to repeat call w/o bad effects - atomicity not necessary? - tasks: - task status: wait for monitoring - asynch needed: task model Session 3: ========== - create outline of requirement document -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 1

Re: comment on SAGA strawman doc.
by Andre Merzky 30 Jul '05

30 Jul '05

Hi Peter, since we have not approached ACLs yet, and since I am not really knowladgable about security, I have no answer. So, playing ping pong, what would you propose as a sensible approach? Others on the list any comments? Cheers, Andre. Quoting [Peter Kunszt] (Jul 26 2005): > > hi andre > > ok! about consistency, you are right, that should be discussed separately. > however, i have another - related - issue with copy: security semantics. > > if you issue a copy command and the source is not owned by you but > you have read permission (say through ACLs). and it is a recursive copy; > how do you propagate permission information to the target? do you make > everything owned by the person whe issues the copy (which may be a service!) > or do you copy the ACLs and the permissions along with the file (ie metadata > copy)? how do you make sure that the same users exist then on the target > site? > > can of worms opened... > > cheers > peter > > > ----- Original Message ----- > From: "Andre Merzky" <andre(a)merzky.net> > To: "Peter Kunszt" <peter.kunszt(a)cern.ch> > Cc: "Andre Merzky" <andre(a)merzky.net>; "Simple API for Grid Applications > WG" <saga-rg(a)ggf.org>; "Kalman Kovari" <Kalman.Kovari(a)cern.ch> > Sent: Monday, July 25, 2005 2:25 PM > Subject: Re: comment on SAGA strawman doc. > > > >Quoting [Peter Kunszt] (Jul 21 2005): > >> > >>hello andre > >> > >>maybe the doc we were looking at was not the latest one.. > >>where is the one that is the current reference? we got our > >>copy from gridforge.. > > > >The latest version is at our wiki: http://wiki.cct.lsu.edu/saga/ > > > >On GridForge was a good version as well, the one we wanted > >to have reviews for. So the excuse of having given you an > >old copy does not work it seems ;-) > > > > > > > >>>>> - The examples are sometimes more misleading than > >>>>> useful... > >>> > >>>Hmm, we are not sure why that is. Are they too complex? > >>>Not focused enough? > >> > >>our (certainly subjective) impression was that they are > >>too simplistic and not enough explanations are given. > >>specifying most possible errors would be also quite useful > >>and a motivation what problem the example at hand is > >>supposed to solve. maybe again it was an early document. > > > >Point taken I think. Currently the examples are pretty much > >just pieces of C++ code which are supposed to illustrate the > >useage of parts of the API. Seems to work well for the > >authors, but not so well for the reader. Ok, thanks. > > > > > > > >>>>> - Interface NSDir: what are the semantics of the copy > >>>>> method? of getName and getURL? > >>> > >>> Hmm, we actually hoped that the semantics is well > >>> defined. If the copy call returns w/o error, a copy of a > >>> file will exist on the target location. > >> > >>in the doc we had, it was just listed, no extra text was > >>there stating what happens if something goes wrong... > > > >For example for copy, the current version says (the flags > >are defined elsewhere): > > > >------------------------------------------------------------- > > Methods for operation on namespace entities: > > - copy > > Purpose: copy the entry to another part of the namespace > > Format: void copy (in string source, > > in string target, > > in array<copyFlags,1> > > flags); > > Inputs: source: name to copy > > target: name to copy to > > flags: flags defining the operation > > modus > > Outputs: none > > Throws: BadParameter: name(s) and/or flags are > > invalid > > NoSuchFile: name(s) do(es) not exist > > NoSuccess: flags inhibited successful > > operation > > Notes: - if the target is a directory the source entry > > is copied into the directory > > - it is an error if the source is a directory > > and the 'Recursive' flag is not set. > > - if the target already exists, it will be > > overwritten if the 'Overwrite' flag is set, > > otherwise it is an error > > - default flags set is > > {NoOverwrite, NoRecursive} > > - Overwrite and NoOverwrite cannot be specified > > at the same time > > - Recursive and NoRecursive cannot be specified > > at the same time > > - similar to the 'cp' command in Unix shells, > > as defined by POSIX > >------------------------------------------------------------- > > > >The old version should have been similar I think (have to > >check). It does indeed not tell you much about consistency, > >only refers to POSIX shell cp. > > > >About consistency we had a lengthy discussion, with the > >aggreement that the consistency model is to be defined and > >documented by the implementation. The API spec itself does > >not assume any specific consistency model, as we feel that > >(a) POSIX consistency is not achievable within reasonable > >effort/performance, (b) if the user assumes the worst (no > >consistency), he will still be able to make good use of the > >API, and (c) reality will be somewhere in the middle. > > > >Would you see problems with that approach? > > > >Cheers, Andre. > > > > > > > > > >-- > >+-----------------------------------------------------------------+ > >| Andre Merzky | phon: +31 - 20 - 598 - 7759 | > >| Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | > >| Dept. of Computer Science | mail: merzky(a)cs.vu.nl | > >| De Boelelaan 1083a | www: http://www.merzky.net | > >| 1081 HV Amsterdam, Netherlands | | > >+-----------------------------------------------------------------+ > > -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

4 7

Re: [saga-rg] persistent session management?
by Thilo Kielmann 29 Jul '05

29 Jul '05

[I have put this back to the list. -- Thilo] On Fri, Jul 29, 2005 at 01:11:04PM +0200, Andre Merzky wrote: > > As I see it, we have 2 different things: > > > > 1. session handles > > they are objects (data structures) within an application. > > > > 2. tasks > > they have some state stored at the job submission service/mechanism, > > at least while they are active and running > > > > Andre's question is about associating a task to a session handle. > > IMHO, this should alway be possible (e.g., create a session handle and > > assign it a task). > > To clearify that point: every task WILL be associated to a > session handle. Always. > > Question is, if that association can survive the > lifetime/runtime of a application instance. And if that is > necessary/wanted. My guess would be that NO. There should be an OPTION (default) to associate a task with a session handle. You can see from your example that a mandatory/permanent association is not always possible. > > > An "interesting" problem might be to find back an old job within the > > job submission machanism. For this purpose, two options might be > > possible: > > > > 1. the application stores some task identifier "somewhere" > > 2. the job submission machanism allows to retrieve a list of all > > tasks, and the application (or an adaptor?) can match te right job > > based on the set of given properties... > > 3. the job submission machanism allows to retrieve a list of > all tasks which the 'user' has access rights to, even if > they have not been created by a SAGA application. > > I am afraid though that we can mandate neither of these > options: 1 needs persistent storage, which might not be > available/accessible. 2 and 3 need support for task > listing on server side, which is not always given AFAIK. What should be mandated is an operation to associate a task to a session handle, based on some properties. The implementation (options 1, 2, 3, or even other) should go into the engine/adaptor implementation. In some cases, this association may just fail, tough luck if there is no implementation available under certain circumstances... Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/

2 3

find in name spaces
by Andre Merzky 29 Jul '05

29 Jul '05

And hi again. The current strawman API does not contain any means to find entries in a name space (e.g. to find a specific logical file in a replica catalog). The only way to perform such a operation is currently to walk through the namespace 'by foot', and evaluating the output of 'list' and the various inspection methods (isFile, isDir...) and possible attributes attached to the entries. We left out 'find' until now for the main reason that a good and mostly complete query language seemed difficult to define. However, my opinion is that a _simple_ find can easily be defined, hence this proposal for a simple find on namespaces: 1) find on names common find operations on name spaces search for name patterns "ls *.tex" "ls data/*/summary.dat)" such operations should be covered by a find. 2) find on meta data another common find paradigma is to find entries according to some meta data "find files named summary.dat" "find files owned by Fritz" "find files older than 3 days" "find files larger of size 1024 bytes" "find files larger than 1 MB" The second set of find examples impy some knowledge about attribute semantic, and that is where the complexity for find operations comes in. e.g. "find files older than 3 days" needs a compare relation for dates and times, and a specification about metadata structure (how is date/time represented? Part one of the problem is, in my opinion, easily dealt with, in two ways: - allow shell wildcards for all 'name' arguments on the name space operations: dir.list ("*.tex"); dir.move ("data/*/summary.dat", "/data/summary/"); - add a simple find, which runs recursively (removing the burden of implementing that on application level over and over again) dir.find ("data/*.tex"); Part two of the problem is, as said, more difficult. However, I think it can be coherently dealt with, in a way which is consistent with the evolution of SAGA. The strawman API includes the AttributeSet interface. By simply adding that interface to the NameSpace, and hence adding attributes to name space entries, a find can run over these attributes. More complex find semantics (dates, owners etc.) can evolve with more complex attribute types, which are planned to get defined in the future anyway. BTW: That argument can also be made in reverse: attributes will be more useful if there is a find which can operate on them. BTW: And that same argument can be made for ACL which are to be defined in SAGA still. As attributes are currently just key/value pairs of strings, the name space would look like: interface SAGA::NSEntry : extends-all SAGA::Attributes { } interface SAGA::NSDir : extends-all SAGA::NSEntry { find (in array<string, 1> query, out array<NSEntry, 1> matches); } The query would be defined as a set of key/value pairs, such as: "name=data/*.tex" "tag=useful" "enabled=" (memaning key must be present with any value) Note that this does not allow search for "size>2MB" or "date>2days" etc. - again, such queries need to wait until the attribute interface becomes more sophisticated. I would be happy to hear feedback on the list. - does it make sense at all? - is it too simple to be of any use? - is it too complex to fit into SAGA? Cheers, Andre. PS.: That proposal does not imply that any name space instance has to implement ar bitrary persistent meta data! Attributes can be read only (file size), and the set of attributes supported can be limited by the class specification (we do the very same for the JobDefinition). -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

persistent session management?
by Andre Merzky 28 Jul '05

28 Jul '05

Hi all, during last GGF, an interesting discussion happened in the DRMA group session. Discussed was following use case: - a application starts - it creates a session handle - it starts a task (long running) with that session handle - it closes the session (destroys the handle - it stops - the application starts again - creates a session handle - it retrieves the old task handle - it waits for that task to finish, or just queries its status - it destroys the session handle again The point here is obviously: the session lifetime actually exceeds the runtime of the application. That scenario is most simple to the application programmer if the session created during the second run is actually the SAME session as in the first run. But that mechanism would obviously involves some state management, and creates a set of problems: - who can (re)create the session? - can it be re cretaed twice? At the same time? - how is the session to be re created identified? - where are persistend information stored? is it neccessary to store persistent information? Now, the question I would like to raise here: - do people think that mechanism is useful? - are there really situations which cannot be handled by creating a new session with the same properties (security context, ownership etc) as the initial session? Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

Re: comment on SAGA strawman doc.
by Andre Merzky 25 Jul '05

25 Jul '05

Quoting [Peter Kunszt] (Jul 21 2005): > > hello andre > > maybe the doc we were looking at was not the latest one.. > where is the one that is the current reference? we got our > copy from gridforge.. The latest version is at our wiki: http://wiki.cct.lsu.edu/saga/ On GridForge was a good version as well, the one we wanted to have reviews for. So the excuse of having given you an old copy does not work it seems ;-) > >>> - The examples are sometimes more misleading than > >>> useful... > > > >Hmm, we are not sure why that is. Are they too complex? > >Not focused enough? > > our (certainly subjective) impression was that they are > too simplistic and not enough explanations are given. > specifying most possible errors would be also quite useful > and a motivation what problem the example at hand is > supposed to solve. maybe again it was an early document. Point taken I think. Currently the examples are pretty much just pieces of C++ code which are supposed to illustrate the useage of parts of the API. Seems to work well for the authors, but not so well for the reader. Ok, thanks. > >>> - Interface NSDir: what are the semantics of the copy > >>> method? of getName and getURL? > > > > Hmm, we actually hoped that the semantics is well > > defined. If the copy call returns w/o error, a copy of a > > file will exist on the target location. > > in the doc we had, it was just listed, no extra text was > there stating what happens if something goes wrong... For example for copy, the current version says (the flags are defined elsewhere): ------------------------------------------------------------- Methods for operation on namespace entities: - copy Purpose: copy the entry to another part of the namespace Format: void copy (in string source, in string target, in array<copyFlags,1> flags); Inputs: source: name to copy target: name to copy to flags: flags defining the operation modus Outputs: none Throws: BadParameter: name(s) and/or flags are invalid NoSuchFile: name(s) do(es) not exist NoSuccess: flags inhibited successful operation Notes: - if the target is a directory the source entry is copied into the directory - it is an error if the source is a directory and the 'Recursive' flag is not set. - if the target already exists, it will be overwritten if the 'Overwrite' flag is set, otherwise it is an error - default flags set is {NoOverwrite, NoRecursive} - Overwrite and NoOverwrite cannot be specified at the same time - Recursive and NoRecursive cannot be specified at the same time - similar to the 'cp' command in Unix shells, as defined by POSIX ------------------------------------------------------------- The old version should have been similar I think (have to check). It does indeed not tell you much about consistency, only refers to POSIX shell cp. About consistency we had a lengthy discussion, with the aggreement that the consistency model is to be defined and documented by the implementation. The API spec itself does not assume any specific consistency model, as we feel that (a) POSIX consistency is not achievable within reasonable effort/performance, (b) if the user assumes the worst (no consistency), he will still be able to make good use of the API, and (c) reality will be somewhere in the middle. Would you see problems with that approach? Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

issue list update
by Andre Merzky 21 Jul '05

21 Jul '05

Hi list, the open issue list in CVS was updated with the results from todays phone call. Next call will handle item 24 to 36. The list is attached. Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0