[DRMAA-WG] Meeting Minutes OGF-42, Day 1

[Bruno P. Kinoshita]

Hello Khalid

Very interesting links, thanks! I haven't used JNI or JNA for quite some time. Supporting Maven is a requirement for us, so if the DRMAA v2 isn't released to Maven Central we will probably fork and release it under a different groupId.

Due to Jenkins master/slave architecture, we have two models in mind. One where the user will start a slave remotely in one computer with access to submit jobs and manage the queue (here we can use JNA or JNI). And the other model, where installing a slave (some Java code communicating with the master via SSH) won't be necessary, and we will use overthere [1] to send commands and transfer files via SSH. 

Our main requirements in this Jenkins plug-in are to submit Jobs but be able to mix local processing and cluster processing, and manage the queue remotely, so that researchers won't have to log in to check the status of the submitted jobs.

In any case, we will probably have to use the DRMAA v2 Java binding, but for the latter case write custom remote Java APIs (one for PBS, one for SGE, and so it goes) that don't use JNI or JNA, but sends commands via SSH and parses the output. 

It is not clear which is the best way, so we will work with the researchers interested in testing the plug-in to see which model works best.

Thanks!
Bruno

[1] https://github.com/xebialabs/overthere


>________________________________
> From: Khalid Shakir <kshakir at broadinstitute.org>
>To: Bruno P. Kinoshita <brunodepaulak at yahoo.com.br> 
>Cc: drmaa-wg <drmaa-wg at ogf.org> 
>Sent: Wednesday, September 10, 2014 3:15 PM
>Subject: Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
> 
>
>
>Hi Bruno,
>
>
>I've seen your blog posts on possibly using SWIG or JNI for DRMAA. Back with DRMAAv1, with both a C and a Java specs available, we implemented a generic DRMAAv1 Java-to-C binding using JNA's dynamic invocation. While I never had time to support our wrapper as a proper mavenized library, I'd love to see someone publish a similar DRMAAv2 JNA wrapper as an artifact, of course after both specs are out.
>
>
>Anecdotally, after a user contributed patch mapped other parts of our software to PBS specific variables, the above wrapper seemed to work for at least a another without further modification to our DRMAAv1 JNA code. While I haven't measured the actual speed of GridEngine-via-JNA vs. GridEngine-via-JNI, this dynamic wrapper "feels" fast enough for most users, as the JNA probably adds <0.5s per call, ever. This known cost has been worth it since we now only include a single DRMAAv1-JNA wrapper, versus multiple JNI wrappers for DRMAAv1-PBS, DRMAAv1-GridEngine, etc.
>
>
>Best wishes,
>-k
>
>
>
>
>On Wed, Sep 10, 2014 at 9:10 PM, Bruno P. Kinoshita <brunodepaulak at yahoo.com.br> wrote:
>
>Hello Peter
>>
>>
>>> We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step. 
>>
>>
>>That's our plan. And we also believe that using an industry-proven software, with formed community and documentation will make it simpler for contributors and new developers.
>>
>>
>>> Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
>>
>>
>>That sounds great. I'll try to hack an ansible + vagrant/docker script to give it a shot. 
>>
>>
>>> Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
>>
>>
>>Thanks! Are you going to use GitHub? If so I can probably bother you there with issues and maybe PR's :)
>>
>>
>>All the best,
>>Bruno
>>
>>
>>
>>>________________________________
>>> From: Peter Tröger <peter at troeger.eu>
>>>To: Bruno P. Kinoshita <brunodepaulak at yahoo.com.br> 
>>>Cc: drmaa-wg <drmaa-wg at ogf.org> 
>>>Sent: Wednesday, September 10, 2014 4:14 AM
>>>Subject: Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
>>> 
>>>
>>>
>>>Dear Bruno,
>>>
>>>
>>>thanks for your feedback, we always need encouraged people to join the DRMAA working group.
>>>
>>>
>>>I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.
>>>>
>>>>
>>>>Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.
>>>
>>>
>>>We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step. 
>>>
>>>
>>>Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
>>>
>>>
>>>Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
>>>
>>>
>>>Best regards,
>>>Peter.
>>>
>>>
>>>
>>>
>>>
>>>
>>--
>>  drmaa-wg mailing list
>>  drmaa-wg at ogf.org
>>  https://www.ogf.org/mailman/listinfo/drmaa-wg
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/drmaa-wg/attachments/20140911/d7e9285c/attachment-0001.html>