Hello Cheng,
Thanks for getting an official python binding to drmaa! That's really great.
At some point I was using Enrico's python binding quite a lot, and wrote a wrapper on top of that to simplify some of the things that I did very often (attached). Basically I wanted a "generalized for loop", that would allow me to write more or less the same python code whether I was using the cluster or not.
The module uses the file system to transfer data between the submitter and the compute nodes, and it can also do local "parallel" computation using threading (however, I don't think threading is working particularly well at the moment).
Thanks for informing us about your efforts. In fact, we need every kind of feedback about implementations in the field. Adding a 'loopback mode' based on threads is of course a great idea for testing and debugging. Enrico somehow already agreed to give his updated implementation to the public. We thought about storing the sources on OGF resources. If this happens, you could simply jump in and add your extended features.
Also, I just recently moved institutions, so while my previous place was using gridengine, now my new place uses lsf (which unfortunately does not have a python binding to DRMAA yet). What do you think is the best strategy to implement a python binding? Is using swig on the C binding a good option?
Enrico's implementation will again use a given DRMAA C library, but in a more dynamic fashion without SWIG. This makes it a generic solution for every DRM providing a DRMAA C library. If you have enough detailed knowledge of LSF, you could also access LSF directly in a DRMAA Python implementation, either by command-line tools or some proprietary API. From a performance perspective, this might be a better solution than tunneling everything through DRMAA C. And it gives us a second independent implementation, which OGF would love to see.
Anyway, keep up the great work!
Thanks. I put the DRMAA list on CC, since this a community effort ;-) Best regards, Peter.