Hi all,
Also, I just recently moved institutions, so while my previous place was using gridengine, now my new place uses lsf (which unfortunately does not have a python binding to DRMAA yet). What do you think is the best strategy to implement a python binding? Is using swig on the C binding a good option?
Enrico's implementation will again use a given DRMAA C library, but in a more dynamic fashion without SWIG. This makes it a generic solution for every DRM providing a DRMAA C library.
If you have enough detailed knowledge of LSF, you could also access LSF directly in a DRMAA Python implementation, either by command-line tools or some proprietary API. From a performance perspective, this might be a better solution than tunneling everything through DRMAA C. And it gives us a second independent implementation, which OGF would love to see.
I don't have detailed knowledge of LSF, or for that matter any implementation of cluster systems. I know that FedStage has released a C DRMAA implementation which works on our system here. https://www.fedstage.com/wiki/FedStage_DRMAA_Guide I'll take a look at Enrico's python DRMAA implementation (when it is out) and see whether it will work with FedStage's C library. The suggestion about doing a direct implementation is interesting. I'm not sure I know enough about LSF or our customizations here to be able to do that. But it raises an issue that I've recently had. I imagine that many sites would have some sort of custom default resource requests (we definitely do here). My naive pythonic solution would be to have a base class which has the "basic" drmaa requests, and then a derived class which mirrors our local set of submit scripts so that jobs submitted via drmaa have identical rights (and default settings) to jobs submitted by command line. Would this be a good thing to do? I guess my question is more about "How does one customize drmaa for a particular site?". Or is it bad to have the class implementing "basic" drmaa requests, because the cluster system admins probably would like such jobs. Cheng