December 2004 - drmaa-wg

Jini+DRMAA integration plus Condor problem
by Zoltan Juhasz 07 Feb '05

07 Feb '05

To the members of the DRMAA list for information. We have developed a Jini wrapper service to interface/integrate legacy batch runtime environments into Jini service communities (specifically into our JGrid service grid - more info at http://pds.irt.vein.hu/jgrid) Our first implementation of this service uses the Sun Grid Engine and its Java DRMAA API in the backend. The service supports input and output file staging, so remote user can upload files, execute commands/programs and save the results more or less transparently. The service is under testing but basically it works very nicely. We are also creating a version with a backend interface to the Condor system. This is where we have a problem and perhaps some of you can help us out. Since Condor only provides a C implementation of the DRMAA spec we our creating a new Java/JNI JDRMAA version based on the SGE one. The first real problem we have is that Condor does not support the mandatory Working Directory attribute and at the moment I cannot see how one can programmatically submit jobs to Condor without this. Where should users put their executables? Will all go to a system-wide submit directory? There may be a simple answer to this, so excuse me if I just miss some information or don't understand how Condor uses the file system for this. Any help is appreciated. Zoltan Zoltan Juhasz Dept of Information Systems University of Veszprem, Hungary

2 2

DRMAA-WG Dec 14, 2004 call
by Rajic, Hrabri 14 Dec '04

14 Dec '04

The bi-weekly DRMAA call is scheduled for 16:00 UTC (9:00PDT /11:00CDT/ 18:00 Central Europe). All Participants should use the following information to reach the conference call: ------------------------------------ * Toll Free Dial In Number: (866)545-5198 * Int'l Access/Caller Paid Dial In Number: (865)521-8904 * ACCESS CODE: 6898552 ------------------------------------ Attachments to this email: - Nov 16 meeting minutes Meeting Agenda: A. Meeting secretary for this meeting? B. Acceptance of the Nov 16, 2004 meeting minutes C. Admin * plan for 2005 F. Open/general issues discussion * JDBC/ODBC differences * SAGA-RG F2F summary Hrabri

2 1

RE: [drmaa-wg] perl DRMAA, SGE and working directory
by Rajic, Hrabri 14 Dec '04

14 Dec '04

Hi, If we look at the string /home/msarachu/.showdb.04.12.09:17.37.42 and consider the following expression (we need to add this syntax into the spec for the working directory as well) [hostname]:file_path then it is not surprising for the runtime to look for directory 17.37.42. Unfortunately, the second error, not being able to find host /home/msarachu/.showdb.04.12.09 is not displayed. Hope this helps from the standard point of view. Regards, -Hrabri -----Original Message----- From: owner-drmaa-wg(a)ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of Martín Sarachu Sent: Tuesday, December 14, 2004 8:28 AM To: drmaa-wg(a)ggf.org Subject: [drmaa-wg] perl DRMAA, SGE and working directory Dear list, I'm using Schedule-DRMAAc-0.81 and SGE to be able to queue jobs from a web interface. Here's my problem: When launching a job with something like /home/msarachu as the $DRMAA_WD it runs ok, but when using a directory like /home/msarachu/.showdb.04.12.09:17.37.42 as $DRMAA_WD the script does not run and the error reported by SGE is "28 : changing into working directory". I also passed the directory "escaped" (\.showdb\.04\.12\.09\:17\.37\.42) and got the same error, although passing the string "/home/msarachu/.showdb.04.12.09:17.37.42/job.sh" to the $DRMAA_REMOTE_COMMAND argument works fine because the job is sent to the queue. Is there any way to mask this directory so it changes ok to the working directory? Below is an email from a failed job I tried to run with DRMAA_WD = /home/msarachu/wProjects/tope/.showdb.04.12.14:16.30.47 Look at the sheperd error, apparently is truncating the dir just before the : If I submit the job from /home/msarachu/wProjects/tope/.showdb.04.12.14:16.30.47 with command 'qsub -cwd job.sh' it works ok. ----- Job 123 caused action: Job 123 set to ERROR User = msarachu Queue = all.q(a)pentiumIV.embnet-ar.org Host = pentiumIV.embnet-ar.org Start Time = <unknown> End Time = <unknown> failed changing into working directory:can't read usage file for job 123.1 Shepherd trace: 12/13/2004 16:31:04 [502:24622]: shepherd called with uid = 0, euid = 502 12/13/2004 16:31:04 [502:24622]: starting up 6.0u1 12/13/2004 16:31:04 [502:24622]: setpgid(24622, 24622) returned 0 12/13/2004 16:31:04 [502:24622]: no prolog script to start 12/13/2004 16:31:04 [502:24623]: pid=24623 pgrp=24623 sid=24623 old pgrp=24622 getlogin()=<no login set> 12/13/2004 16:31:04 [502:24623]: setosjobid: uid = 0, euid = 502 12/13/2004 16:31:04 [502:24623]: RLIMIT_CPU setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_FSIZE setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_DATA setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_STACK setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_CORE setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_VMEM/RLIMIT_AS setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_RSS setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [500:24623]: closing all filedescriptors 12/13/2004 16:31:04 [500:24623]: further messages are in "error" and "trace" 12/13/2004 16:31:04 [502:24622]: forked "job" with pid 24623 12/13/2004 16:31:04 [502:24622]: child: job - pid: 24623 12/13/2004 16:31:04 [502:24622]: wait3 returned 24623 (status: 7168; WIFSIGNALED: 0, WIFEXITED: 1, WEXITSTATUS: 28) 12/13/2004 16:31:04 [502:24622]: job exited with exit status 28 12/13/2004 16:31:04 [502:24622]: reaped "job" with pid 24623 12/13/2004 16:31:04 [502:24622]: job exited not due to signal 12/13/2004 16:31:04 [502:24622]: job exited with status 28 12/13/2004 16:31:04 [502:24622]: now sending signal KILL to pid -24623 12/13/2004 16:31:04 [502:24622]: no tasker to notify 12/13/2004 16:31:04 [502:24622]: failed starting job 12/13/2004 16:31:04 [502:24622]: no epilog script to start Shepherd error: 12/13/2004 16:31:04 [500:24623]: error: can't chdir to :16.30.47: No such file or directory Shepherd pe_hostfile: pentiumIV.embnet-ar.org 1 all.q(a)pentiumIV.embnet-ar.org UNDEFINED ----- I sent this same email to Tim and also SGE users list. Tim also suggested to send it to this list. Thanks in advance. Best regards, Martin -- Martín Sarachu msarachu(a)biol.unlp.edu.ar EMBnet Argentina http://www.ar.embnet.org

3 2

perl DRMAA, SGE and working directory
by Martín Sarachu 14 Dec '04

14 Dec '04

Dear list, I'm using Schedule-DRMAAc-0.81 and SGE to be able to queue jobs from a web interface. Here's my problem: When launching a job with something like /home/msarachu as the $DRMAA_WD it runs ok, but when using a directory like /home/msarachu/.showdb.04.12.09:17.37.42 as $DRMAA_WD the script does not run and the error reported by SGE is "28 : changing into working directory". I also passed the directory "escaped" (\.showdb\.04\.12\.09\:17\.37\.42) and got the same error, although passing the string "/home/msarachu/.showdb.04.12.09:17.37.42/job.sh" to the $DRMAA_REMOTE_COMMAND argument works fine because the job is sent to the queue. Is there any way to mask this directory so it changes ok to the working directory? Below is an email from a failed job I tried to run with DRMAA_WD = /home/msarachu/wProjects/tope/.showdb.04.12.14:16.30.47 Look at the sheperd error, apparently is truncating the dir just before the : If I submit the job from /home/msarachu/wProjects/tope/.showdb.04.12.14:16.30.47 with command 'qsub -cwd job.sh' it works ok. ----- Job 123 caused action: Job 123 set to ERROR User = msarachu Queue = all.q(a)pentiumIV.embnet-ar.org Host = pentiumIV.embnet-ar.org Start Time = <unknown> End Time = <unknown> failed changing into working directory:can't read usage file for job 123.1 Shepherd trace: 12/13/2004 16:31:04 [502:24622]: shepherd called with uid = 0, euid = 502 12/13/2004 16:31:04 [502:24622]: starting up 6.0u1 12/13/2004 16:31:04 [502:24622]: setpgid(24622, 24622) returned 0 12/13/2004 16:31:04 [502:24622]: no prolog script to start 12/13/2004 16:31:04 [502:24623]: pid=24623 pgrp=24623 sid=24623 old pgrp=24622 getlogin()=<no login set> 12/13/2004 16:31:04 [502:24623]: setosjobid: uid = 0, euid = 502 12/13/2004 16:31:04 [502:24623]: RLIMIT_CPU setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_FSIZE setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_DATA setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_STACK setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_CORE setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_VMEM/RLIMIT_AS setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [502:24623]: RLIMIT_RSS setting: (soft 4294967295 hard 4294967295) resulting: (soft 4294967295 hard 4294967295) 12/13/2004 16:31:04 [500:24623]: closing all filedescriptors 12/13/2004 16:31:04 [500:24623]: further messages are in "error" and "trace" 12/13/2004 16:31:04 [502:24622]: forked "job" with pid 24623 12/13/2004 16:31:04 [502:24622]: child: job - pid: 24623 12/13/2004 16:31:04 [502:24622]: wait3 returned 24623 (status: 7168; WIFSIGNALED: 0, WIFEXITED: 1, WEXITSTATUS: 28) 12/13/2004 16:31:04 [502:24622]: job exited with exit status 28 12/13/2004 16:31:04 [502:24622]: reaped "job" with pid 24623 12/13/2004 16:31:04 [502:24622]: job exited not due to signal 12/13/2004 16:31:04 [502:24622]: job exited with status 28 12/13/2004 16:31:04 [502:24622]: now sending signal KILL to pid -24623 12/13/2004 16:31:04 [502:24622]: no tasker to notify 12/13/2004 16:31:04 [502:24622]: failed starting job 12/13/2004 16:31:04 [502:24622]: no epilog script to start Shepherd error: 12/13/2004 16:31:04 [500:24623]: error: can't chdir to :16.30.47: No such file or directory Shepherd pe_hostfile: pentiumIV.embnet-ar.org 1 all.q(a)pentiumIV.embnet-ar.org UNDEFINED ----- I sent this same email to Tim and also SGE users list. Tim also suggested to send it to this list. Thanks in advance. Best regards, Martin -- Martín Sarachu msarachu(a)biol.unlp.edu.ar EMBnet Argentina http://www.ar.embnet.org

1 0

Driver Architecture
by Daniel Templeton 13 Dec '04

13 Dec '04

OK. Now I have a little time to get into this discussion. As I remember, we left off with a description of JDBC and ODBC and an open question as to which would be most appropriate for DRMAA, e.g. which best supports multiple sessions. To get this going, I'm just going to restate my previous assessment: the JDBC model is an object-oriented version of the ODBC model. Since we're working on a set of OO guidelines, I vote for the JDBC model. (Anyone surprised?) I would, however, love to hear arguments to the contrary. I don't want to force every binding to do things the Java way. I just want to do it the best wway. It's just that as it turns out, that *is* the Java way... :) Daniel

1 0