Thanks Andre, That could be a good starting point for us. File transfer resolution is a complex issue, that would require a different approach today than 5 years ago, and possible yet another approach 5 years from now. Is it part of the current spec or another one could be also a good question. One litmus test for making it a standard would be when implementations start adding it as optional attributes. If this issue is not already recorded as a tracker we need to add it. For now this issue does not impact the current DRMAA-R candidate document. Regards Hrabri
-----Original Message----- From: drmaa-wg-bounces@ogf.org [mailto:drmaa-wg-bounces@ogf.org] On Behalf Of Andre Merzky Sent: Saturday, May 05, 2007 9:40 AM To: Peter Troeger Cc: DRMAA Working Group Subject: Re: [DRMAA-WG] Torque/PBS DRMAA - file transfer routines
For what its worth, the SAGA group also stumbled over this problem. We finally opted to adopt a LSF like syntax, following a suggestion from Chris Smith (no surpise, ey? ;-).
That lsyntax ism in our opinion, generic enough to easily map to the various backends. The syntax basically is (quoting the SAGA spec):
----------------------------------------------------------------- File Transfer Specifications ----------------------------
The syntax of a file transfer directive for the job description is modeled on the LSF syntax {(LSF stands for Load Sharing Facility, a commercial job scheduler by Platform Computing), and has the general syntax:
local_file operator remote_file
Both the local_file and the remote_file can be URLs. If they are not URLs, but full or relative pathnames, then the local_file is relative to the host where the submission is executed, and the remote_file is evaluated on the execution host of the job.
The operator is one of the following four:
'>' copies the local file to the remote file before the job starts. Overwrites the remote file if it exists.
'>>' copies the local file to the remote file before the job starts. Appends to the remote file if it exists.
'<' copies the remote file to the local file after the job finishes. Overwrites the local file if it exists.
'<<' copies the remote file to the local file after the job finishes. Appends to the local file if it exists. -----------------------------------------------------------------
That mechanism is also used for stdin/stdout/stderr. I don't want to say that DRMAA should use that - there are certainly limitations - but might be worth to look at at some point...
Cheers, Andre.
Quoting [Peter Troeger] (May 04 2007):
From: Peter Troeger <peter.troeger@hpi.uni-potsdam.de> To: DRMAA Working Group <drmaa-wg@ogf.org> Subject: Re: [DRMAA-WG] Torque/PBS DRMAA - file transfer routines
Issue:
File transfer routines are desired by many users and currently they are only limited to standard input, output and error streams. ---------
Omitting file transfer capability was done on purpose in version one to put all the major DRM systems at the same footing.
On of the ambitious DRMAA goals was to influence DRM system vendors or providers to standardize (one more time) the DRM systems and add missing capabilities as deemed necessary by the user community, so this is an ambitious goal - in contrast to the request.
Maybe this is out of scope for DRMAA. Even the three available file transfer attributes are implemented in a different manner in Condor, SGE and GridWay. Specifying a common file transfer API is also already handled by other working groups.
Peter.
-Hrabri -- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg
-- "XML is like violence: if it does not help, use more."
-- drmaa-wg mailing list drmaa-wg@ogf.org http://www.ogf.org/mailman/listinfo/drmaa-wg