[DRMAA-WG] drmaa python

[Peter Tröger]

Hello Cheng,

> Thanks for getting an official python binding to drmaa! That's really 
> great.
> 
> At some point I was using Enrico's python binding quite a lot, and wrote 
> a wrapper on top of that to simplify some of the things that I did very 
> often (attached). Basically I wanted a "generalized for loop", that 
> would allow me to write more or less the same python code whether I was 
> using the cluster or not.
> 
> The module uses the file system to transfer data between the submitter 
> and the compute nodes, and it can also do local "parallel" computation 
> using threading (however, I don't think threading is working 
> particularly well at the moment).

Thanks for informing us about your efforts. In fact, we need every kind 
of feedback about implementations in the field.

Adding a 'loopback mode' based on threads is of course a great idea for 
testing and debugging. Enrico somehow already agreed to give his updated 
implementation to the public. We thought about storing the sources on 
OGF resources. If this happens, you  could simply jump in and add your 
extended features.

> Also, I just recently moved institutions, so while my previous place was 
> using gridengine, now my new place uses lsf (which unfortunately does 
> not have a python binding to DRMAA yet). What do you think is the best 
> strategy to implement a python binding? Is using swig on the C binding a 
> good option?

Enrico's implementation will again use a given DRMAA C library, but in a 
more dynamic fashion without SWIG. This makes it a generic solution for 
every DRM providing a DRMAA C library.

If you have enough detailed knowledge of LSF, you could also access LSF 
directly in a DRMAA Python implementation, either by command-line tools 
or some proprietary API. From a performance perspective, this might be a 
better solution than tunneling everything through DRMAA C. And it gives 
us a second independent implementation, which OGF would love to see.

> Anyway, keep up the great work!

Thanks. I put the DRMAA list on CC, since this a community effort ;-)

Best regards,
Peter.