January 2010 - saga-rg - lists.ogf.org

Some more comments on the SAGA specs.
by Ceriel Jacobs 17 Feb '10

17 Feb '10

Hi, I have some more remarks/questions about the new Saga specifications, this time about the task package. Here they are, in random order: Bugs: task.wait() in the details does not specify a default value for the timeout. PreCond on task.cancel(): sais that task is in 'Running' state, but the Notes say that it can also be in a final state, in which case the cancel has no effect. Why does task_container.remove() return the removed task? It is already specified as parameter. PostCond on task_container.run(): sais that all tasks in the container are in 'Running' state, but I guess the only thing you can be sure of is that the state is not 'New', like with task.run(). PostCond on task_container.cancel(): sais that if no timeout occurs all tasks in the container are in 'Canceled' state. But the tasks can be in another final state as well. Debatable issues: task_container: what is the difference between list_tasks() and get_tasks()? list_tasks() originally returned the cookies. Should it now return a list of object ids? Or should the method just be removed alltogether? I am not sure that it is a good idea to have get_result() call rethrow(). How do I now make a distinction between an exception coming from the get_result() call itself and an exception from the task? I guess that in the exception handler I can check the state of the task, but that does not give you a definite answer: if the state is "FAILED", the exception can still come from get_result() itself, can it not? Why not leave it to the user to call rethrow() if he/she wants, and as it was before? Cheers, Ceriel

2 2

small glitch in Saga specs
by Ceriel Jacobs 29 Jan '10

29 Jan '10

Hi, a small glitch in the JobService specs: The notes on the CONSTRUCTOR mention that BadParameter is thrown, but BadParameter is not in the list of exceptions thrown. Ceriel

2 2

Problem with -Output attribute of JSAGA
by Lokendra S 27 Jan '10

27 Jan '10

Hi everyone, I intend to use JSAGA using SSH adaptor to run jobs in a Grid. For testing purposes, I am using my own machine to submit and run jobs. for eg,the command 'jsaga-job-run.sh -Executable /bin/date -r local://localhost#SSH' runs successfully and executes 'date' command and displays the result. However when I use the command jsaga-job-run.sh -Output output.txt -Executable /bin/date -r local://localhost#SSH -b to store the output of 'date' command in a file output.txt. No such file is created after completion of the job. Please help Thanks in advance Lokendra Singh The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/

2 1

no call today
by Andre Merzky 27 Jan '10

27 Jan '10

Hi all, I'd like to skip this weeks call if that's ok with you all -- too many other things going on this week. Please let me know if you think we should have the call though! Best, Andre. -- Nothing is ever easy.

1 0

profiling
by Andre Merzky 24 Jan '10

24 Jan '10

Hia folx, Having too much time (*ahem-lack-of-focus-ahem*), I run a profiler against saga-job. Attached is the result. The command line was: ./saga-job run fork://localhost /bin/date Walltime is consistently about 2 seconds, while a simple test program just doing fork/exec takes no measurable time. Now, this is of course not fair comparision, but I wanted to know where the time is spent anyway. The profiler gave me the figure attached as saga_job.prof_1.pdf . It turns out that most time is in fact spent in boost::uuid, which gets called a whopping 125 times, eating abouyt 89% of the run time. To me it looks like it is trying to re-seed the random number generator over and over again? So, I took the liberty to implement a much more stupid uuid generator, which simply assigns an integer as uuid (see attached uuid.hpp, which replaces saga/impl/engine/uuid.hpp), and indeed the runtime for the same command line decreases to about 0.5 seconds, which are now mostly spent in ini file parsing and adaptor loading, as it should be (see saga_job.prof_2.pdf). Morover, that time should be constant per application instance, whereas the uuid time was likely proportional to the number of SAGA calls (or objects) used by the application. Now, I am not claiming that we should use this overly simple uuid generator (although it should create 4.294.967.295 unique IDs per application instance), but in any case, the uuid generator could do with some improvement. I did not really understand the boost code well enough to see why the seeding is triggered so often, but maybe Hartmut has an idea, or someone else comes up with a cheap alternative. Cheers, Andre. -- Nothing is ever easy.

1 2

tomorrows phone conf details
by Andre Merzky 13 Jan '10

13 Jan '10

Hi all, our next call is tomorrow (Wednesday) at 10pm CET (3pm Central). Dial-in details are: +1-225-578-4942 (international) 866-573-0359 (US, toll free) conference-id: 118682 The agenda items are: - discussion of python bindings (again) - please read manuels comparison of gthe two python implementations which went to the list a couple of days ago - discussion of cpr package draft (depending on attendance) Best, Andre. -- Nothing is ever easy.

1 1

Fwd: SAGA GridCPR API document review
by Derek Simmel 13 Jan '10

13 Jan '10

Folks, I'm forwarding Nathan Stone's comments regarding the SAGA_CPR document: - Derek Begin forwarded message: > From: Nathan Stone <stone(a)psc.edu> > Date: January 6, 2010 4:47:58 PM EST > To: Derek Simmel <dsimmel(a)psc.edu> > Subject: Re: SAGA GridCPR API document review > > Hi Derek, > > Regarding the SAGA_CPR document: > > - CPRFrequency (page 4) should probably have a default more like 1 hour than 1 day, from previous machine experience. > - CPRTimeToLive (page 5) sounds like a bad idea. Checkpoints should be kept until the job is complete, without regard for how long that is. Otherwise you may have long-running jobs that will thereby have their recoveries cut out from under them -- while the job innocently and unknowingly gets swapped out for a higher priority job. > - class "directory" (page 9) seems to have nothing to do with CPR. I hope that's all just a restatement for purposes of clarity... > - class "checkpoint" (page 10) seems to be the same thing as class "directory". If they really are the same, then why define an identical object? Why not just say that a checkpoint is a special instantiation of a directory object? Or, if you're really gung-ho on the object framework, derive a "checkpoint" class from the "directory" class... and add one more special function (or a special constructor status?). > > The other CPR_Architecture document appears to be the same as when we last left it, and as such certainly seems fine to me. > > Feel free to pass along my comments to the current document steward. > > Thanks, > Nathan. --- Derek Simmel Pittsburgh Supercomputing Center (412) 268-1035

2 1

Fwd (mamonski@man.poznan.pl): Re: Fwd (kielmann@cs.vu.nl): Re: Face-to-Face Meeting
by Andre Merzky 13 Jan '10

13 Jan '10

FYI, Andre. ----- Forwarded message from Mariusz Mamo??ski <mamonski(a)man.poznan.pl> ----- > Date: Thu, 7 Jan 2010 15:16:06 +0100 > Subject: Re: Fwd (kielmann(a)cs.vu.nl) Re: [SAGA-RG] Face-to-Face Meeting > From: Mariusz Mamo??ski <mamonski(a)man.poznan.pl> > To: Andre Merzky <andre(a)merzky.net>, Thilo Kielmann <kielmann(a)cs.vu.nl>, > Peter Tröger <peter(a)troeger.eu> > Cc: Ariel Oleksiak <ariel(a)man.poznan.pl> > > Hi, > > As the late follow up let me also share with our experience on > exposing advance reservations capabilities of the batch systems: > > http://fury.man.poznan.pl/~mmamonski/wiki/index.php/DRMAAv2/Advance_Reserva… > > The similarity with the DRMAAv2.0 look-and-feel of this interface is > intended ;-) > > The main differences with SAGA approach are: > - we have no reservation state model. The rationale behind this is > that, in opposite to jobs, advance reservations are quite predictable > (you always know the start and end time of the reservation) so we do > not see any sense in polling whether reservation is in RUNNING state. > The only unpredictable state is an error state (e.g. reserved host is > down) but this is not easily detectable in majority of the DRMS we > tested. > - we proposed some best-effort semantics : > * the requested CPU count (here called "slotsCount") can be > expressed as a range value and the system is obligated to reserve as > much resources as possible (constrained by the given range) > * in addition to the reservation start and end time one can specify > also the reservation duration (shorter than the endTime - startTime). > In this case system tries to book resources in the earliest possible > time (having in mind the other resource requirements, e.g.: > slotsCount) > > All the best, > > On Fri, Dec 11, 2009 at 3:17 PM, Andre Merzky <andre(a)merzky.net> wrote: > > FYI, Andre. > > > > -- > > Nothing is ever easy. > > > > > > ---------- Forwarded message ---------- > > From: Thilo Kielmann <kielmann(a)cs.vu.nl> > > To: Andre Merzky <andre(a)merzky.net> > > Date: Wed, 9 Dec 2009 17:01:30 +0100 > > Subject: Re: [SAGA-RG] Face-to-Face Meeting > > > > Hi all, > > > > unfortunately I cannot make it to the meeting tomorrow :-( > > > > Nevertheless, I would like to share the XtreemOS view on > > resource reservations with you: > > > > 1. we have basically split SAGA's job-description into two parts > > a) a resource-description (everything machine specific) > > b) an application-description (everything app specific) > > > > 2. we have added a notion of reservation, with reservation service > > and state model > > > > 3. jobs can now be submitted either as in SAGA's job package, > > or indirectly, by getting a reservation first, and then submitting to > > the reservation... > > > > 4. a student of mine has extended this idea to clouds: > > a) he added image management for virtual machines > > b) he extended the reservation manager such that VM's can be created > > "on the fly" of reserving them > > > > The gory details are in the attached document. The interesting part > > starts on page 24, which is Section 5.7. > > > > As needed, I can call in tomorrow/friday. Please let me know. > > > > > > Regards, > > > > > > Thilo > > > > > > > > > > > > On Dec 5, 2009, at 11:20 PM, Andre Merzky wrote: > > > >> Hi all, > >> > >> this is an reminder that we have the DRMAA/SAGA F2F next week, > >> Thursday/Friday, at gthe Zuse Institute Berlin. We have booked the > >> room all day, so I propose we start at Thursday 9am and finish 4pm > >> Friday. > >> > >> As far as agenda goes, we will use up most of the time to sync the > >> evolution of DRMAA with the evolution of the SAGA job package. > >> We'll thus cover an overview of both in the beginning, and then > >> discuss various separate items in depths, such as > >> > >> - object hierarchy > >> - job monitoring > >> - notification mechanism > >> etc. > >> > >> This list is at the moment open, and will likely defined by the set > >> of participants on the fly. > >> > >> Please let me know if you need further information or travel advice. > >> > >> Cheers, see you soon, Andre. > >> > >> > >> > >> Quoting [Andre Merzky] (Nov 09 2009): > >>> > >>> Hi again, > >>> > >>> it seems that December 10/11 is the preferred option for the F2F, so > >>> we'll go ahead and plan for that date. > >>> > >>> So far we have confirmations from five people to attend. We'd > >>> appreciate if you could all confirm your attendence by end of > >>> November. > >>> > >>> The meeting will be hosted by Zuse Institute Berlin (thanks!). > >>> We'll send the exact details around about a week before the meeting. > >>> To help travel planning: ZIB is about an hour away from Tegel > >>> Airport, about two hours from Schoenefeld Airport, and about 30 > >>> minutes from Berlin Downtown. More details, and proposed > >>> accomodations, are on http://www.zib.de/besucher/index.en.html > >>> Please let us know if you want us to help organize accomocations. > >>> > >>> To re-iterate: this will mostly be a DRMAA meeting, dedicated to > >>> make progress on the open DRMAA.v2 issues, but we will use part of > >>> that F2F time to sync with the SAGA approaches which seem to have > >>> covered some part of the problem space targeted by DRMAA.v2. > >>> > >>> Cheers, see you soon in Berlin, > >>> > >>> Andre. > >>> > >>> > >>> Quoting [Andre Merzky] (Oct 24 2009): > >>>> > >>>> [ sorry for cross-posting ] > >>>> > >>>> Hi DRMAA group, Hi SAGA group, > >>>> > >>>> at OGF27, we discussed a joined 2-day face-to-face meeting between > >>>> the SAGA and DRMAA groups, with focus on the upcoming DRMAA.v2 > >>>> specification. We expect a SAGA DRMAA extension to be specified in > >>>> lockstep -- thus the cross-group participation. > >>>> > >>>> At the moment, we plan to have that F2F meeting in Berlin (thanks to > >>>> Alexander for agreeing to host!). We created a doodle form to poll > >>>> for possible dates - so for all those who intent to participate in > >>>> the F2F, please add your availability at: > >>>> > >>>> http://www.doodle.com/m4z7fy888yndp7f7 > >>>> > >>>> The F2F is open of course, but we may have some space constraints - > >>>> so please *do* fill the doodle soon, so that we can have a > >>>> reasonable estimate about participation. > >>>> > >>>> Cheers, Andre > >> > >> -- > >> Nothing is ever easy. > >> -- > >> saga-rg mailing list > >> saga-rg(a)ogf.org > >> http://www.ogf.org/mailman/listinfo/saga-rg > > > > -- > > Thilo Kielmann > > http://www.cs.vu.nl/~kielmann/ -- Nothing is ever easy.

1 0

Python bindings naming issues
by Manuel Franceschini 12 Jan '10

12 Jan '10

Hi folks, 'NotImplemented' [0] and 'NotImplementedError' [1] are reserved keywords in Python. Thus, the SAGA 'NotImplemented' Exception has the potential to hide some of these keywords in the Python bindings. So we could either rename the exception or live with the fact that there might be some (small?) problems. What do you think? Similarly, 'id', which is used as parameter in many methods in the SAGA spec, is a built-in function in Python. This, however, shouldn't be to much of an issue to rename. Cheers, /Manuel [0] http://docs.python.org/library/constants.html#NotImplemented [1] http://docs.python.org/library/exceptions.html#exceptions.NotImplementedErr… [2] http://docs.python.org/library/functions.html#id

2 1