June 2006 - drmaa-wg - lists.ogf.org

RE: [drmaa-wg] DRMAA test suite
by Rajic, Hrabri 28 Jun '06

28 Jun '06

The new pages look good. I would not mind if we switch to them very soon. We will edit them as we go. Regards Hrabri p.s. I have fixed the broken link Peter. ________________________________ From: owner-drmaa-wg(a)ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of Peter Troeger Sent: Wednesday, June 28, 2006 5:58 AM To: drmaa-wg(a)gridforum.org Subject: Fwd: [drmaa-wg] DRMAA test suite Just a reminder: I created a Wiki page which could act as new drmaa.org startup page (see below). I need your comments / additions. Thanks, Peter. Anfang der weitergeleiteten E-Mail: Von: Peter Troeger <peter.troeger(a)hpi.uni-potsdam.de> Datum: 23. Juni 2006 11:23:44 MESZ An: Daniel Templeton <Dan.Templeton(a)Sun.COM> Kopie: drmaa-wg(a)gridforum.org Betreff: Re: [drmaa-wg] DRMAA test suite Hrabri, The 0.35 IDL spec is *still* mislinked from drmaa.org. The right link is: https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.drmaa -wg/docman.root.ggf_14/doc5555/2 As everybody agreed to have the (updated) main page as wiki page, I prepared a first version: http://www.drmaa.org/wiki/index.php/home I cleaned up the content a little bit, and updated all relevant links. Please make your changes (it's a Wiki !), and inform the list when you are done. When everybody is happy, we could switch to the new start page. Regards, Peter.

2 1

RE: [drmaa-wg] Simultaneous waits on the same job id?
by Rajic, Hrabri 28 Jun '06

28 Jun '06

Good question. There is no such provision in the spec. One thread would need to be the first ... Hrabri >-----Original Message----- >From: owner-drmaa-wg(a)ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of >Ed Baskerville >Sent: Friday, June 23, 2006 3:32 PM >To: DRMAA Working Group >Subject: [drmaa-wg] Simultaneous waits on the same job id? > >With all the discussion of wait in multithreaded contexts, I thought >I'd throw out another related question... > >Are multiple threads allowed to wait simultaneously on the same job >id and get back results, or is it required that one of them gets back >DRMAA_ERRNO_INVALID_JOB? That is, is it possible for the data to be >reaped simultaneously for multiple waiting threads, or must only one >of them be lucky enough to get the data back? > >For Xgrid, either way is straightforward to implement; obviously >having the option of returning data to multiple simultaneous calls >would be nice, but I want to get the semantics right. > >--Ed

5 7

Re: [drmaa-wg] buffer length/size
by Piotr Domagalski 28 Jun '06

28 Jun '06

On Fri, Jun 16, 2006 at 01:36:01PM +0200, Peter Troeger wrote: > The DRMAA specification is intended to be language independent, and > hopefully talks always about the length of a string. A string is a > collection of printable characters, therefore '\0' should not count. > Could it be that you talk here about specifics of the C-binding > ("error_diag_len", "job_id_len") ? Yes, I'm talking mainly about the C-binding, but the specifiaction also adds some words about _BUFFER constants and is sometimes inconsistent as you noted... The constants (and whether they include '\0' or not) is one issue, but I think the most important is what all the _len arguments mean in C-binding. The worst thing that could happen when a developer misunderstands the constants (max len vs. buffer size) is truncation, but a wrong argument in function calls could lead to a crash. From C-binding document: job_id_len - The size in characters of the job identifier buffer. This would clearly mean sizeof(buf), but the "characters" makes me think it could be the maximum number of printable characters that will be put into the buffer. In my opionion this word is not needed here, it should only say "the size of the job identifier buffer". And as for the constants, in my opinion, as they are supposed to be langunage independent, they should mean the max string length... but then, the name preffix _MAXLEN instead of _BUFFER would be better. > String length vs. buffer size would be optimal. indeed. > > I'm more convinced that the constants define the total buffer size, but > > some DRMAA example source codes got me confused, e.g. they define > > variables char foo[DRMAA_JOBNAME_BUFFER] but use DRMAA_JOBNAME_BUFFER - > > 1 as call arguments. > > I think it was intended to do it the first way. I propose that the DRMAA > spec should say: > > "The length of any output context-specific error string SHALL NOT exceed > DRMAA_ERROR_STRING_BUFFER - 1." But doesn't it make the specification language dependent? -- Piotr Domagalski

2 2

DRMAA test suite
by Piotr Domagalski 28 Jun '06

28 Jun '06

Hi! DRMAA test suite (from Condor-ext project) tests for this: - drmaa_control() with no jobs in session but with SESSION_ALL argument succeds, - drmaa_wait()/drmaa_synchroznie() with no jobs in session but with SESSION_ANY/SESSION_ALL argument return INVALID_JOB error, As I remember, there is nothing about it in the specification, and I see little discrepancy - SGE's implementation fails in the first test. -- Piotr Domagalski

4 7

How long to use a Session?
by Jeff White 27 Jun '06

27 Jun '06

I have a daemon process that periodically submits jobs to SGE. I have a separate thread that uses Session.wait(JOBS_IDS_SESSION_ANY) to gather the JobInfo objects, for example to add to a completion queue. The session may be idle for long periods of time. Should I use a single Session for the entire duration of the application, for each submitted job, or something in between? jeff

2 1

RE: [drmaa-wg] Simultaneous waits on the same job id?
by Rajic, Hrabri 26 Jun '06

26 Jun '06

There is an ability to discard the reaped info in synchronized call due to memory constraints. It is neither ignore nor reap, but dispose, so you lose it. The reasoning for multithreaded library use went along the lines of users doing their own thread synchronization. Their flexibility was on our minds at that time, so we wanted a simple semantics attached to the routines. The way we saw it at that time, reaped info would be global data and therefore only thread would get to it. If a thread wants to reap it own jobs that would work fine also. Reaping the same info by multiple threads is definitely useful, but was not a preferred semantics by the participants at the time. A bit of history ... I hope it helps Hrabri >-----Original Message----- >From: owner-drmaa-wg(a)ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of >Daniel Templeton >Sent: Monday, June 26, 2006 8:36 AM >To: Ed Baskerville >Cc: DRMAA Working Group >Subject: Re: [drmaa-wg] Simultaneous waits on the same job id? > >Ed, > >What Andreas is trying to get at is that since synchronize() doesn't >return job information, so multiple simultaneous synchronize() calls >will all succeed. Even if synchronize() reaps job info, since other >calls to synchronize() don't care whether they actually are able to find >the job info, it's all good. As long as synchronize() is running, >reaped jobs are viewed as simply having ended, so not even a call to >wait() will cause a call to synchronize() to fail. > >Is that really correct? Can anyone else confirm? I think we should >probably have a tracker to clarify that point in the (g2) spec. > >Daniel > >Andreas.Haas(a)Sun.COM wrote: >> On Fri, 23 Jun 2006, Ed Baskerville wrote: >> >>> OK, I'll do that. But one more question: how do these issues apply to >>> synchronize? Consider the following sequence: >>> >>> thread 1: wait(job id 5) >>> thread 2: synchronize(job ids 5,6,7) >>> [job id 5 finishes] >>> ¿thread 2 synchronize call fails? >>> [job id 7 finishes] >>> [job id 6 finishes] >>> ¿thread 2 synchronize call succeeds? >>> >>> Should synchronize fail with INVALID_JOB as soon as any of the ids >>> it's waiting on are reaped? Or should it eventually succeed? >> >> There is no reason for synchronize to fail, as long as none of the >> jobs was reaped when synchronize() gets issued. >> >> Andreas

1 0

Simultaneous waits on the same job id?
by Ed Baskerville 23 Jun '06

23 Jun '06

With all the discussion of wait in multithreaded contexts, I thought I'd throw out another related question... Are multiple threads allowed to wait simultaneously on the same job id and get back results, or is it required that one of them gets back DRMAA_ERRNO_INVALID_JOB? That is, is it possible for the data to be reaped simultaneously for multiple waiting threads, or must only one of them be lucky enough to get the data back? For Xgrid, either way is straightforward to implement; obviously having the option of returning data to multiple simultaneous calls would be nice, but I want to get the semantics right. --Ed

1 0

DRMAA test suite JAVA
by José Herrera 23 Jun '06

23 Jun '06

Dear All, We have developed an implementation of the DRMAA Test Suite using the JAVA Bindings. There are two test missing, that will be available in the next release: 1.- ST_EXIT_STATUS 2.- ST_SUBMIT_KILL_SIG The test suite can be downloaded from: 1. http://www.gridway.org/documentation/guides.php also you can access to the last version using the CVS at: 2. http://viewcvs.globus.org/viewcvs.cgi/gridway/examples/drmaaTestSuite/ This test suite has been tested on Linux and with the GridWay DRMAA library ;-) Comments and bug reports are very welcome. Regards. José Herrera Sanz

2 2

Added flow processor to DRMAA for Ruby samples
by Andreas Haas 20 Jun '06

20 Jun '06

Hi, I added yesterday a little flow processor to the DRMAA for Ruby samples. A short description can be found in README below. Regards, Andreas == Workflow interpreter and processor flow.rb is a utility for running job workflows in DRMAA-compliant DRM systems. Workflows are specified in flowfiles that allow expression of * concurrent and sequencial execution of sub-flows * multiple runs of sub-flows with varying parameter sets * acutal jobs are defined in terms of DRMAA attributes in addition any subflow or sets of subflows can be run as used with make(1) by specifying it as target. === Job defaults To minimize the extend attributes necessarily to be specified in flowfiles, flow.rb provides defaults for jobs command path, stdout/stdin path, current working directory and job name. === Workflow verification The -verify option can be used to print dependencies and job attributes for diagnosis purposes and to ensure each workflow job can be run, a number of verifications is performed before the first job gets submitted. === Job streaming Large workflows are automatically run in job streaming mode upon DRM saturation and a job maximum can be used to set in .flowrc.rb an upper limit of job concurrently be kept in the DRM for each workflow. === Pre-submission plug-in Enforcement of site-specific policies can easily be achieved through pre-submission procedures that allow any job attribute freely be modified.

1 0

RE: [drmaa-wg] buffer length/size
by Rajic, Hrabri 16 Jun '06

16 Jun '06

[ ... ] > >I think it was intended to do it the first way. I propose that the DRMAA >spec should say: > >"The length of any output context-specific error string SHALL NOT exceed >DRMAA_ERROR_STRING_BUFFER - 1." This is a good suggestion, but it is too risky to make such a change at this time juncture. A Tracker should help, so we do not forget to fix this inconsistency down the road in the C binding implementation(s). Hrabri > > Peter.

1 0