RE: [drmaa-wg] making a custom file path for output
Hi Ali, Placeholders would be special variables like home directory or job id ($ sign prefixed below), which DRMAA could substitute in job attributes like the output path: drmaa_set_attribute( $jt, DRMAA_OUTPUT_PATH, $DRMAA_HOME_DIR/MY_APP/my_job_output.$DRMAA_JOB_ID.out" -Hrabri
-----Original Message----- From: Ali Anjomshoaa [mailto:ali@epcc.ed.ac.uk] Sent: Tuesday, April 05, 2005 10:59 AM To: Rajic, Hrabri Cc: Andreas Haas; Tim Harsch; DRMAA Working Group Subject: RE: [drmaa-wg] making a custom file path for output
Hi Guys,
If you could give me a quick synopsis of what you consider placeholders and in what context, perhaps I can help from the JSDL perspective.
Cheers,
Ali
On Tue, 5 Apr 2005, Rajic, Hrabri wrote:
Andreas,
I would suggest we approach the question as two issues:
1) what are the global placeholders in DRMAA 2) in which (string typed) job attributes the placeholders could be used
Here are few questions that go with the above:
How is JDSL handling placeholders - nice to know? Do all major DRM systems support placeholders well? Do we need a new error code indicating failure in resolving a placeholder? Any other errors needed?
-Hrabri
-----Original Message----- From: owner-drmaa-wg@ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of Andreas Haas Sent: Tuesday, April 05, 2005 7:37 AM To: Tim Harsch Cc: DRMAA Working Group Subject: Re: [drmaa-wg] making a custom file path for output
Hi Tim,
I fully agree there is a need for this. Please see
http://forge.gridforum.org/tracker/?aid=805
I raised it's priority to P2. I believe that issue must be solved with resubitted DRMAA spec.
Regards, Andreas
On Mon, 4 Apr 2005, Tim Harsch wrote:
I'm sorry I wasn't clear. I guess the heart of my question is: How can create an arbitrarty template for the file and path name? Specifically what constant tells the DRM to insert job id into that filename template.
Said in Perl psuedo form:
drmaa_set_attribute( $jt, $DRMAA_OUTPUT_PATH + ":/tmp/my_job_output.$DRMAA_JOB_ID.out" );
# with that code job outputs look like /tmp/my_job_output.1.out /tmp/my_job_output.2.out /tmp/my_job_output.3.out
# without that code job outputs look like /home/harsch/script_name.o1 /home/harsch/script_name.o2 /home/harsch/script_name.o3
Is there a DRMAA_JOB_ID constant?
----- Original Message ----- From: "Rajic, Hrabri"
To: "DRMAA Working Group" Sent: Monday, April 04, 2005 12:02 PM Subject: RE: [drmaa-wg] making a custom file path for output There is a simple solution, provided it works for you: to rely on the shell to supply the home directory path instead of DRMAA.
-Hrabri
-----Original Message----- From: owner-drmaa-wg@ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of Tim Harsch Sent: Monday, April 04, 2005 1:19 PM To: DRMAA Working Group Subject: [drmaa-wg] making a custom file path for output
Hi all, I have a requirement for the app I'm writing ( using Schedule::DRMAAc ) which I'm trying to implement.
Normally the output files from jobs go to home directory and are of the form <jobname>.o<jobid>
I want to move the jobs to a subdir in /tmp, but retain the <jobid> in the filename. I tried this:
( $job_error, $diagnosis ) = drmaa_set_attribute( $jt, $DRMAA_OUTPUT_PATH, ":" . $tmpdir . '/' . __PACKAGE__ . '.' . $DRMAA_PLACEHOLDER_INCR . ".out" );
... but PLACEHOLDER is only for parametric jobs, I've been scanning the docs and have not found a placeholder for job id. Any ideas on how I could accomplish this task?
A similar task would be to move the job output path without changing the filename format at all. How could I do that?
--
---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali@epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------
Funny... I didn't realize I had posted this problem to the list once
before, after Andreas dug up the tracker item I went back to the discussion
in the list and realized I raised the same issue it at that time ( about a
year ago ). :-)
Part of my question I answered for myself, if you use DRMAA_OUTPUT_PATH to
provide just the directory where output files go, the SGE DRMAA
implementation will use that path and the output file will get named
<jobname>.o<jobid>. Which for me, in this case, will be good enough though
not optimal. I could code up a my application a bit better if I could
change the filename to a custom template.
Which brings up another question, which might be important for the spec
(unless its already there, please inform me if so)....
If I used DRMAA_OUTPUT_PATH to change the output directory to /tmp/mydir
say, and the DRM started creating output files in that directory of the form
/tmp/mydir/myjob.o1
/tmp/mydir/myjob.o2
/tmp/mydir/myjob.o3
etc.
Is there a way I can DRMAA to figure out the full path of the output file
given the ID: for instance ( in pseudo perl):
$jobid = 3;
$outpath = drmaa_get_output_file( $jt, $jobid );
# $outpath is now equal to '/tmp/mydir/myjob.o3'
----- Original Message -----
From: "Rajic, Hrabri"
-----Original Message----- From: Ali Anjomshoaa [mailto:ali@epcc.ed.ac.uk] Sent: Tuesday, April 05, 2005 10:59 AM To: Rajic, Hrabri Cc: Andreas Haas; Tim Harsch; DRMAA Working Group Subject: RE: [drmaa-wg] making a custom file path for output
Hi Guys,
If you could give me a quick synopsis of what you consider placeholders and in what context, perhaps I can help from the JSDL perspective.
Cheers,
Ali
On Tue, 5 Apr 2005, Rajic, Hrabri wrote:
Andreas,
I would suggest we approach the question as two issues:
1) what are the global placeholders in DRMAA 2) in which (string typed) job attributes the placeholders could be used
Here are few questions that go with the above:
How is JDSL handling placeholders - nice to know? Do all major DRM systems support placeholders well? Do we need a new error code indicating failure in resolving a placeholder? Any other errors needed?
-Hrabri
-----Original Message----- From: owner-drmaa-wg@ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of Andreas Haas Sent: Tuesday, April 05, 2005 7:37 AM To: Tim Harsch Cc: DRMAA Working Group Subject: Re: [drmaa-wg] making a custom file path for output
Hi Tim,
I fully agree there is a need for this. Please see
http://forge.gridforum.org/tracker/?aid=805
I raised it's priority to P2. I believe that issue must be solved with resubitted DRMAA spec.
Regards, Andreas
On Mon, 4 Apr 2005, Tim Harsch wrote:
I'm sorry I wasn't clear. I guess the heart of my question is: How can create an arbitrarty template for the file and path name? Specifically what constant tells the DRM to insert job id into that filename template.
Said in Perl psuedo form:
drmaa_set_attribute( $jt, $DRMAA_OUTPUT_PATH + ":/tmp/my_job_output.$DRMAA_JOB_ID.out" );
# with that code job outputs look like /tmp/my_job_output.1.out /tmp/my_job_output.2.out /tmp/my_job_output.3.out
# without that code job outputs look like /home/harsch/script_name.o1 /home/harsch/script_name.o2 /home/harsch/script_name.o3
Is there a DRMAA_JOB_ID constant?
----- Original Message ----- From: "Rajic, Hrabri"
To: "DRMAA Working Group" Sent: Monday, April 04, 2005 12:02 PM Subject: RE: [drmaa-wg] making a custom file path for output There is a simple solution, provided it works for you: to rely on the shell to supply the home directory path instead of DRMAA.
-Hrabri
-----Original Message----- From: owner-drmaa-wg@ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of Tim Harsch Sent: Monday, April 04, 2005 1:19 PM To: DRMAA Working Group Subject: [drmaa-wg] making a custom file path for output
Hi all, I have a requirement for the app I'm writing ( using Schedule::DRMAAc ) which I'm trying to implement.
Normally the output files from jobs go to home directory and are of the form <jobname>.o<jobid>
I want to move the jobs to a subdir in /tmp, but retain the <jobid> in the filename. I tried this:
( $job_error, $diagnosis ) = drmaa_set_attribute( $jt, $DRMAA_OUTPUT_PATH, ":" . $tmpdir . '/' . __PACKAGE__ . '.' . $DRMAA_PLACEHOLDER_INCR . ".out" );
... but PLACEHOLDER is only for parametric jobs, I've been scanning the docs and have not found a placeholder for job id. Any ideas on how I could accomplish this task?
A similar task would be to move the job output path without changing the filename format at all. How could I do that?
--
---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali@epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------
Hi Hrabri, we discussed this issue at some length at GGF13 and were unsure as to how to proceed. We would like to be able to reference labeled attributes, but, I think we're still discussing how to implement this issue in the xsd schema depending on current tooling! I think, unfortunately, our choice of how we implement this in xsd will influence how we specify how to do this in the spec. Clearly this is not ideal and should be the other way round! Watch the JSDL space and let us know how you do this in the end. Cheers and take care, Ali On Tue, 5 Apr 2005, Rajic, Hrabri wrote:
Hi Ali,
Placeholders would be special variables like home directory or job id ($ sign prefixed below), which DRMAA could substitute in job attributes like the output path:
drmaa_set_attribute( $jt, DRMAA_OUTPUT_PATH, $DRMAA_HOME_DIR/MY_APP/my_job_output.$DRMAA_JOB_ID.out"
-Hrabri
-----Original Message----- From: Ali Anjomshoaa [mailto:ali@epcc.ed.ac.uk] Sent: Tuesday, April 05, 2005 10:59 AM To: Rajic, Hrabri Cc: Andreas Haas; Tim Harsch; DRMAA Working Group Subject: RE: [drmaa-wg] making a custom file path for output
Hi Guys,
If you could give me a quick synopsis of what you consider placeholders and in what context, perhaps I can help from the JSDL perspective.
Cheers,
Ali
On Tue, 5 Apr 2005, Rajic, Hrabri wrote:
Andreas,
I would suggest we approach the question as two issues:
1) what are the global placeholders in DRMAA 2) in which (string typed) job attributes the placeholders could be used
Here are few questions that go with the above:
How is JDSL handling placeholders - nice to know? Do all major DRM systems support placeholders well? Do we need a new error code indicating failure in resolving a placeholder? Any other errors needed?
-Hrabri
-----Original Message----- From: owner-drmaa-wg@ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of Andreas Haas Sent: Tuesday, April 05, 2005 7:37 AM To: Tim Harsch Cc: DRMAA Working Group Subject: Re: [drmaa-wg] making a custom file path for output
Hi Tim,
I fully agree there is a need for this. Please see
http://forge.gridforum.org/tracker/?aid=805
I raised it's priority to P2. I believe that issue must be solved with resubitted DRMAA spec.
Regards, Andreas
On Mon, 4 Apr 2005, Tim Harsch wrote:
I'm sorry I wasn't clear. I guess the heart of my question is: How can create an arbitrarty template for the file and path name? Specifically what constant tells the DRM to insert job id into that filename template.
Said in Perl psuedo form:
drmaa_set_attribute( $jt, $DRMAA_OUTPUT_PATH + ":/tmp/my_job_output.$DRMAA_JOB_ID.out" );
# with that code job outputs look like /tmp/my_job_output.1.out /tmp/my_job_output.2.out /tmp/my_job_output.3.out
# without that code job outputs look like /home/harsch/script_name.o1 /home/harsch/script_name.o2 /home/harsch/script_name.o3
Is there a DRMAA_JOB_ID constant?
----- Original Message ----- From: "Rajic, Hrabri"
To: "DRMAA Working Group" Sent: Monday, April 04, 2005 12:02 PM Subject: RE: [drmaa-wg] making a custom file path for output There is a simple solution, provided it works for you: to rely on the shell to supply the home directory path instead of DRMAA.
-Hrabri
-----Original Message----- From: owner-drmaa-wg@ggf.org [mailto:owner-drmaa-wg@ggf.org] On Behalf Of Tim Harsch Sent: Monday, April 04, 2005 1:19 PM To: DRMAA Working Group Subject: [drmaa-wg] making a custom file path for output
Hi all, I have a requirement for the app I'm writing ( using Schedule::DRMAAc ) which I'm trying to implement.
Normally the output files from jobs go to home directory and are of the form <jobname>.o<jobid>
I want to move the jobs to a subdir in /tmp, but retain the <jobid> in the filename. I tried this:
( $job_error, $diagnosis ) = drmaa_set_attribute( $jt, $DRMAA_OUTPUT_PATH, ":" . $tmpdir . '/' . __PACKAGE__ . '.' . $DRMAA_PLACEHOLDER_INCR . ".out" );
... but PLACEHOLDER is only for parametric jobs, I've been scanning the docs and have not found a placeholder for job id. Any ideas on how I could accomplish this task?
A similar task would be to move the job output path without changing the filename format at all. How could I do that?
--
---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali@epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------
-- ---------------------------------------------------- |epcc| - Ali Anjomshoaa EPCC, University of Edinburgh James Clerk Maxwell Building Mayfield Road E-mail: ali@epcc.ed.ac.uk Edinburgh EH9 3JZ Phone: + 44 (0) 131 651 3388 United Kingdom Fax: + 44 (0) 131 650 6555 -------------------------------------------------------------
participants (3)
-
Ali Anjomshoaa
-
Rajic, Hrabri
-
Tim Harsch